On Writing Quality Peer Reviews, with Noah A. Smith

Guest: Noah Smith
Hosts: Matt Gardner, Waleed Ammar

It's not uncommon for authors to be frustrated with the quality of peer reviews they receive in (NLP) conferences. In this episode, Noah A. Smith shares his advice on how to write good peer reviews. The structure Noah recommends for writing a peer review starts with a dispassionate summary of what a paper has to offer, followed by the strongest reasons the paper may be accepted, followed by the strongest reasons it may be rejected, and concludes with a list of minor, easy-to-fix problems (e.g., typos) which can be easily addressed in the camera ready. Noah stresses on the importance of thinking about how the reviews we write could demoralize (junior) researchers, and how to be precise and detailed when discussing the weaknesses of a paper to help the authors see the path forward. Other questions we discuss in this episode include: How to read a paper for reviewing purposes? How long it takes to review a paper and how many papers to review? What types of mistakes to be on the lookout for while reviewing? How to review pre-published work?

Matt Gardner
00:00

Hello and welcome to the NLP highlights podcast where we talk about interesting work in natural language processing.

Waleed Ammar
00:06

This is Matt gardener and Waleed Ammar , we are research scientists at the Allen Institute for Artificial Intelligence.

Matt Gardner
00:11

All right. Today our guest is Noah Smith who is a professor of computer science at the University of Washington and a senior research manager at the Allen Institute for Artificial Intelligence AI-2. He’s my boss. I have had the pleasure of working with Noah for quite a while. Both, I guess I took classes and T-aide for classes when Noah was at CMU. Waleed also has a long history with Noah. Noah, it’s good to have you on the program.

Noah Smith
00:32

Delighted to be back. Thanks for having me.

Matt Gardner
00:34

Yeah, I guess we also had one of our very first episodes. Yeah, it’s good to have you back. Today, we wanted to talk about something that you actually suggested and we thought was a really good idea with the NAACL deadline coming up. I guess by the time this is posted, this the NAACL deadline will probably already have passed and it will be reviewing time and you thought it would be a good idea to talk about advice for people who are doing reviews so that maybe collectively we can improve the quality of all of our reviews.

Noah Smith
01:00

Yeah, that’s right.

Matt Gardner
01:01

Noah, do you want to tell us what advice you have?

Noah Smith
01:03

Yeah, sure. So I’ve been, you know, writing papers for a long time now and I have been increasingly, it feels like I’ve been increasingly having this experience that I get back reviews with my coauthors who typically are students and we’re a little frustrated by what the reviewers have said about our paper. And I feel like this is probably a common experience given what people often say on social media and the complaints that come back to, you know, organizers of conferences after we do surveys and so on. We, we often hear this worry, it comes up at business meetings, at conferences, everybody’s sort of frustrated with the quality of reviewing. I remember sitting in one of the business meetings and realizing that we don’t actually do a whole lot collectively as a community to train people on how to review papers.

Noah Smith
01:48

Hopefully many of us who advise PhD students are doing this in our labs, giving people advice as they come up through the ranks and get invited to be usually workshop reviewers. Then conference reviewers, they’re going to come to us and say, Hey, I was asked to do this. Can I do it? And you say, yes, you should. You know, it’s, it’s your duty to review papers because you submit papers and here’s some tips. So what I thought would be useful would be if senior researchers in the community shared their thoughts on how to review papers. And so that’s kind of what I want to do today and encourage my colleagues around the world who have been reviewing for some time and feel they’ve gained some expertise, maybe even been recognized with the best reviewer award. Share your wisdom and help help younger members of the community learn how to do this better.

Noah Smith
02:30

What I usually do when I sit down with my students for their early training in how to review. I usually invite them to help me with a review I’ve been assigned and so you know the process I follow is give them the paper, give them some time. I don’t like doing these things too much at the last minute and say, all right, read the paper, draft what you think would be a reasonable review and then we’ll sit down and compare notes and write a review together. And so we read it independently and usually after that happens there’s we have a discussion like what? What’s in, you know, what’s in this paper and what do we think informally before we start writing the review and the structure I usually recommend for the review that’s going to go back to the authors I like as an area chair and and as one time a program chair to see a review that starts out with a completely dispassionate, ideally very brief summary of what the paper offers.

Noah Smith
03:14

No judgment, no goods, no bads, no pros no cons. Just what is in this paper stated as simply as possible for as wide an audience as possible. If you can, if you can do it without jargon, even better. Usually the people who are reading your review know something about the research area that the paper is contributing to, but they don’t know exactly all of the prior work. They don’t know exactly what problem. They may not be that familiar with this particular data set or task. They may not know all the details. You have to think about this as someone who’s super busy. The first reader of your review is the area chair. They’re super busy. They’ve got a million of these things to read. You want to make it as easy as possible for them first to understand what’s in the paper. So the first thing is, can you succinctly state what the contributions of the paper were without judgment?

Noah Smith
03:57

Again, not, you’re not judging. Is it enough? You’re not judging, is it? Is it amazing? Is it is it life changing? Is it incremental? That’s not for you to say right yet. Just first say what it is and you know often with papers that are not well written, this is the hardest part of the review because sometimes I, you know, I find even when I spend a lot of time with the paper, I can’t really tell what the contributions are. So this is a tip both for authors and for reviewers. Make it clear when you’re writing your paper so that reviewers can easily find it what new in this paper. And if you really can’t articulate that, then you probably have more work to do and it’s not time to write yet. But anyway, when you’re reviewing, start out with a clear summary.

Noah Smith
04:36

Ideally just a few sentences, keep it brief, don’t get into the details. Focus on someone who has just a few minutes to read your review and wants to know what’s in the paper. After you’ve done that, I usually structure the review into two major chunks. The most important one is really the major pros and cons of the paper in your judgment. And so this is what did they do well, what are the reasons that the strongest reasons one could make for publishing the paper. And so, you know, here I like to kind of play, I don’t know. First I guess it’s God’s advocate and then the devil’s advocate. So first fight for the paper and say the strongest things you can. And you know, sometimes this is a bit of a stretch because you’re angry about something in the paper, you’re irritated, you feel like it’s weak, you’re annoyed cause they didn’t cite you.

Noah Smith
05:16

There’s all kinds of, there’s all kinds of emotions that can get in the way here. But you know, you are a scientist and so you need to be clear headed. And so you need to do your, do your best at discussing what the strengths of the papers are. Often they may be right in front of your face. It may be like they’re asking really good questions. This is a really great research area. This is a hard problem that they’re trying to tackle or the central idea of the paper does seem quite intuitive. And it is exciting because nobody has tried this before. There are many things that you can say that are positive about a paper that has deep flaws. Why do this? Why put the strongest reasons you can? Well, you know, it’s ultimately somebody else is going to be making a decision about whether to accept or reject the paper.

Noah Smith
05:52

And if the strongest reasons to accept the paper are when compared to other papers not that strong, then you know, you’ve, you’ve managed to convey the information you need to convey, but you’ve managed to do it in a way that doesn’t completely demoralize those researchers who did the work. So, you know, one of the, one of the important things that happens in reviewing that we don’t often talk about because we’re so focused on the outcomes of the reviews, whether the papers are accepted or not, is what are the effects that we have on other researchers when we write these reviews. I tend now in this career stage when I’m writing a review to imagine that the person who wrote this paper is very early in their career and hasn’t learned how to do things quite right yet and they’re still struggling to figure it all out.

Noah Smith
06:31

And the last thing I want to do is to discourage that person from continuing to try. This part of the review where you focus on the positives I think can be really helpful in minimizing the amount of time that that person is depressed and out of commission and unable to keep going. So this is kind of a responsibility we have toward younger members of the community. Now, I could be completely wrong. Maybe the person who wrote this paper was very experienced and just in a hurry and was sloppy or taking it for granted that everyone would see the brilliance of their idea. Well, no harm lost if they feel, if they feel positively about about the positive aspects of their work. Again, we’re trying to give an honest appraisal. We’re not going to make things up. We’re not going to say things that aren’t true about the paper,

Noah Smith
07:09

We are going to say the best possible things that we can. I think that’s just a nice way to soften the blow for what’s to come next, which is usually the weaknesses of the paper. And so you know the, the weaknesses part, this is where you have to be diplomatic and careful because I think it’s very easy to sort of sigh and write the whole thing off and say, okay, this paper’s completely broken and there’s too many problems. I can’t possibly help them fix it. And so you say, you say things like the paper wasn’t clear or the results did not support the argument or the conclusions were too sweeping or they didn’t cite these five papers. Well actually that’s a, that’s, that’s not a terrible, that’s not a terrible comment because that’s fairly precise. The point I’m trying to get at is, is that you want to try to make your feedback on the weaknesses of the paper actionable.

Noah Smith
07:48

So again, the the authors can see the way forward and can see how to make their work better for the next round. So again, details are good. Exactly what’s, what is the failure of the argument? Exactly. What was confusing? Was it a, was it that the definition seemed off? Was it that they never defined certain terms? Again, the more crisp and clear your critique can be, the easier it is for the person to fix it and to understand how to do better in the future. Remember, you’re not just helping the authors fix this paper. You’re helping the authors learn how to write better papers so that in the future when you see their papers, it’s an easier job for you. Now notice everything I’ve said up until now is really about substance. It’s about, you know, what are the fundamental ideas in the paper? What are the major strengths and what are the major weaknesses?

Noah Smith
08:32

If the review is going on for pages and pages, at this point you might be nitpicking more than you need to. It’s good to remember that people have kind of limited capacity for handling lots of information. So if you go on and you write pages and pages here, it might be, it might be quite useful to the authors, but what I try to do is really focus on the most important things because ultimately no paper is going to be perfect and it’s not your job to move the paper from where it is now to the perfect ideal paper we all want to read, so usually because I’m a little bit OCD about writing papers, I find a bunch of things that are annoying that are more stylistic or typographical and I just save those for the end. There’s usually like a checklist after the strengths and weaknesses of little things that are easy for the authors to fix that.

Noah Smith
09:12

If these were the only things wrong with the paper, you’d say accept it but please fix your typos and that would be the recommendation. Sometimes there’s also a box for, you know, private comments to the area chair or other things you want to communicate. I use that very sparingly. There are relatively few circumstances where the situation is a little weird and more needs to be said. I tend to think that focusing your comments toward the authors to help encourage the good things that they’ve done and help them do better next time or do better with the camera ready. Right. Even if even if you think the paper’s amazing and should be accepted, you may still have some weaknesses that you would like them to address. Again, if you do that diplomatically, they’re more likely to do it and take it seriously, appreciate your comments and ultimately continue the conversation that we’re all having when we write papers. So that long spiel I just gave is usually more or less what I tell my grad students when they’re getting started.

Matt Gardner
09:58

So I guess you given us a good description of what the content of a review should look like. And if I could summarize you, I think you gave basically four main sections. One is a short one that says what’s in the paper. Then strengths than weaknesses, then typographical copy editing kinds of stuff. What about the process of actually doing the review? When I’m reading the paper, what should I be thinking about? What should I be doing? How long should it take?

Noah Smith
10:20

How long it takes is deeply personal, right? Different people read at different rates and the time that you put in may vary quite a lot depending on how familiar you already are with the research area. Back when I was a student and I was reading papers that were very close to the problems I was working on, it was like no time at all for me to get through because you know it was, it was stuff I largely already knew and I could almost see what was coming as I was reading. If I’m reading a paper that’s more distant, obviously takes me considerably more time. I may have to pause, go look things up, go look, go reference an earlier paper. I may have to stop and think for a bit. So the amount of time it takes, that’s kind of a know thyself kind of question.

Noah Smith
10:55

I think different people do this differently. The way I do it and I think a lot of others do is to start with a very quick read, a kind of scan, just so I know, you know, read the abstract, skim through the intro, look for the major figures that are going to give me an idea just to kind of situate where is this work going to be, where does it live in my internal knowledge graph or whatever my internal knowledge structure, my knowledge matrix tensor, I don’t know how we store knowledge, we just kind of localize it. That usually includes the final tables and conclusion in my case, and I usually scan the bibliography too, which is again kind of a way of knowing what this work is related to and what part of my brain I want to reactivate for thinking about what’s here.

Noah Smith
11:32

And usually by the time I’ve done that, I know kind of what kind of paper it is. Is it a theoretical paper? Is it a an engineering paper where they’ve made an advance on how to solve a problem? Is it a position piece and so on. So by now you kind of know what you’re expecting. Then I do a more careful read from top to bottom, usually with a red pen so I can leave little notes to myself in the margins and focus on reading, not formulating my review. After I do that, I like to get a little distance and go do something else for a bit and then come back and review my notes so that I can compose the review with a little bit of distance. I have to admit that doesn’t always happen. I often find myself reviewing papers on a plane and I’d rather finish the review for one paper before I start reading the next, or I’m going to start confusing things. So I will go top to bottom on one paper and write the review before I go to the next one.

Matt Gardner
12:17

To what extent is it the reviewer’s job to find errors?

Noah Smith
12:20

There are different kinds of errors, so I think reviewers can be forgiven for missing notational errors and typos and stuff like that. I think if a reviewer sees a flaw in an argument, that’s definitely something you want to watch for, right? So they claim that this experiment or result means X or that this error analysis implies the model is doing Y and you believe that there’s an alternative explanation or that they didn’t take something into account or they missed something they should’ve controlled for. They didn’t do the proper ablations experimentally or even in the philosophical part of the paper where they reason about why such a method should make sense or what linguistic phenomena make the case for doing some special kind of technique. All of those things I think are fair game and it is appropriate for a reviewer to be skeptical and question back. I’ve often had reviewers point out weaknesses in our, in our arguments that don’t fundamentally change the structure of the paper or the main findings, but do force us to rethink our claims a little bit and maybe say things in a, in a more cautious way. And that’s, you know, that’s not because we’re sloppy or we were trying to overclaim. It’s just that more eyes on a paper can help make sure that what it purports as truth is actually true.

Matt Gardner
13:26

Do you try or do you recommend students try to assure that proofs are correct or that the numbers that are reported for prior experiments matched what was reported in prior experiments?

Noah Smith
13:37

That’s a great question. So if you’re familiar with the experimental setup and you want to check these things, I don’t think anybody would fault you. I have not heard many reports of people misquoting numerical results from earlier papers, but it could certainly happen, you know, that would be appreciated. I don’t personally do that. Perhaps I come from a simpler time when there was more naive trust in other researchers for the amount of time I have just spend on a review that doesn’t feel like the best use of that time, although maybe I should reconsider it.

Matt Gardner
14:07

I guess it’s also for mistakes if they copied the wrong number or there were two different versions of an experiment in the original paper and they used the wrong one to compare against.

Noah Smith
14:17

Yeah, there are, there are some things like this that I try to be on the lookout for. Like if the paper’s not explicit about all the conditions. Say there was some kind of gold standard pre- processing that was assumed in the current paper that I want to know; did the baselines also have that same pre-processing? Right. If this isn’t made crystal clear in the details section, then I might ask, you know, could you clarify this just for peace of mind? Cause I can’t, you know, obviously nobody’s going to remember exactly what pre-processing was done in every single experiment by every single author. Especially, you know, nowadays we have so many people competing on the same datasets. So I think raising these questions, if you know about these experimental details, you know, sometimes you find yourself reviewing a paper. The dataset is new, the task is new to you and you’re just not really familiar with what the whole pipeline is.

Noah Smith
15:01

It would be laudable if you’ve made the investment to go find out and double check everything they’ve done. But the vast majority of reviewers won’t do it. We tend to trust each other. You asked about proofs, you know, we don’t see a lot of these in papers in natural language processing. We do see them in more in machine learning and they tend to be relegated to supplementary details. If you feel comfortable checking the proof, that’s a great service to do it. If it’s in the main paper, you should read it and you should check it because if it’s in the main paper, that implies that it should be understandable by members of the community. Now reading proofs and working on proofs is not a skill set that most people in the natural language processing community have cultivated. And so it’s very easy for me to imagine a graduate student reading a proof, getting frustrated because some of the notation is unfamiliar or some of the underlying math is unfamiliar and it feels like, you know, they’re skipping steps and and you get angry and then you project that anger onto the authors that may or may not be fair.

Noah Smith
15:53

So I think in this situation, the thing to do is maybe say in the review that you tried to follow the proof, you found it somewhat difficult. Did this in a self deprecating way maybe, but point out that if you feel your math skills are representative of those in the community, then you could urge the authors to take more time and make the proof a little bit more detailed. You know, there are choices that people have when they write proofs in both their notation and how much English they include and how much detail they offer and how much they walk through things step by step and encouraging authors who, you know, sometimes it’s a case of authors coming from a slightly different research community with different assumptions about the readership. You’re not going to make yourself look bad by encouraging authors to make their proof more clear.

Noah Smith
16:31

Obviously in this situation you are, you’re basically giving up on being able to find errors. If you can’t follow the proof, you’re not gonna detect errors in it. But if you can do it, it is a great service. And, and you know, ideally a theoretical paper would have some reviewers assigned who the area chairs can assume will follow and check the proofs. That is certainly something I would try to do as an area chair when assigning reviewers to a paper.

Matt Gardner
16:52

All right, so we’ve talked about like the general format of review and the process that you might go through when you’re doing a review. NAACL and EMNLP recently have experimented with changing review forms. Do you have any thoughts about this? Is it a good way to encourage better content for reviews?

Noah Smith
17:06

I think the the advice I would give for people who are in a position to change review forms is to be very cautious and incremental. People may remember that at NAACL this past year, they changed the scale, the ordinal scale for the overall rating to go from one to six instead of one to five as it had been in the past. And I think there was a tremendous amount of confusion and people didn’t know how to interpret their scores anymore. And maybe this doesn’t matter. I think everybody has opinions about ways we could improve the reviewing and it’s good to experiment a little bit and kind of see how things go. And it would be ideal if we had ways of checking people’s reactions after it was over, which I don’t think we do very carefully. We don’t really have a lot of data to go on, but generally the same rule should apply when you’re dealing with review forums that applies in research change one thing at time. Don’t go making massive changes, they’re just going to confuse people and you’re not going to really get a clear signal about what works.

Noah Smith
17:55

Personally, I thought the one to six scale was not great. I do think, you know, breaking out the review form into a few sections or sort of offering a pre-filled in template that encourages people to list strengths and weaknesses. Usually, by the way, those are not, those are not bullets. Those are usually like little paragraphs in a well-written review. It’s not like, you know, a little bulleted list of one-liners. It’s like several, several brief paragraphs each explaining a strengths. I think that’s fine. I think, I think encouraging people to be more thoughtful is good, but ultimately you can only do so much with the forms. People are gonna do what they’re gonna do. I really think the, the important thing is to create a sense of investment in the review system and you know, articulate our shared values around why, why do we do this whole peer review thing in the first place? That’s something I think people don’t completely always understand.

Matt Gardner
18:41

Yeah. Great point. I have a couple of higher level questions. How does archive and pre-prints change reviewing for instance, the BERT paper that made huge waves just like a month or two ago. It hasn’t been reviewed yet. Presumably someone’s gonna review it at NAACL. There are lots of similar papers like this that we could list that a lot of people are familiar with but haven’t been through review. What recommendation do you have about reviewing this kind of paper?

Noah Smith
19:03

So I guess in some ways this is new to have so many of our papers already publicly available with author information known and affiliation information known before the deadline. It’s not really new to have ideas out in public view or semi-public view that have been propounded by different researchers and shared in one form or another. You know, for years many of us have given talks about work we haven’t published yet. That’s often the part of the talk I’m most excited and passionate about when I’m giving a presentation is what we’re doing right now that we’re hoping to publish soon. Because we’re really thinking hard about it. Still. It’s not, it’s not a done deal. And you know, that’s not gonna go away. Archive maybe makes it a little bit more widespread, a little bit easier to see if a large number of our authors have posted their papers on archive before submitting to the conference.

Noah Smith
19:48

Then you know, you could, you could ask, well, what’s happened to peer review to blind peer review to double blind peer review. So if you know now, now we’re moving into a mode where it’s much more common not just to have some intuition or maybe some side knowledge that suggests you know, who the authors are, but you actually just know who the authors of the paper are. And that’s tricky because why do we like double-blind peer review? We like double-blind peer review because it separates the identity of the researcher from the research results and it forces us to evaluate work on its actual merits, evaluate arguments intrinsically rather than based on the fame or past results of the people who publish them, which can, which can cut both ways. We don’t want to penalize newcomers. In fact, we want to encourage them because the community will ultimately do better as a whole if we have increasing diversity of researchers who come in from other places.

Noah Smith
20:33

We also don’t want to give a bonus points to people just because they’ve had strong results in the past. And so this is quite a conundrum and I think the community’s really been struggling with it. So, so you see different policies being tested out by different conferences like no pre-prints, or you must publish your pre-print at least a certain window of time before the conference deadline and various other experiments. There’s no simple answer to this. So if you’re given a paper that you already know because it’s been published on the archive or in some other form, you’ve seen the work. The first question to ask yourself is, do you have a conflict of interest with this paper? Is there anything about your relationship with the authors or to the work that’s going to make it hard for you to be objective about it and give it a fair reading in either direction?

Noah Smith
21:14

Again, maybe you’re going to be super excited about the work and you’re worried that you’re going to be extra excited because you’re good friends with the authors or you know you you’re doing something that’s in competition and you’ve got a competing idea or a competing method and so you’re, you’re worried that you wouldn’t be able to give it a fair reading. This is obviously a little unfair to ask people to police themselves, but by and large, in my experience, when people think about this question, they tend to be more cautious than you would expect and so basically if your gut tells you it’s going to be really impossible for you to judge this fairly, then you should probably go to the area chair and say, I think I have a conflict of interest and talk it out and they might convince you that you don’t often an another pair of eyes on the situation will bring a little more clarity. I would suggest doing that right away so that you know that the area chair has has time to find somebody else if they need to.

Noah Smith
21:58

Once you’ve done that, suppose it passes and you’re like, well, you know, I’ve read this paper and I have some interest in the results, but I, but I don’t really have a, a strong reason to be biased toward or against the authors or the work. So I think I’m okay to do the review. Then your next responsibility is to read the paper as it’s been submitted. It’s easy to sort of assume you know what’s going on and then kind of skim through and do a quick job because you already know the work and life has already moved on since, you know, last summer whenever the paper was posted online. But in fact the submitted version might have new results and things, the story may have changed.

Noah Smith
22:30

They may have, they, you know, one of the reasons that people post papers on the archive in advance of a deadline is to get feedback so that they can improve the paper before it goes in for peer review. This is how, you know, some other disciplines actually function as a norm. The work is posted on the archive or some other pre-print server. It’s discussed maybe for months or years. And then given all that feedback and all of that input that others have offered on the paper, the authors continue to revise so that what goes in for peer review is actually much stronger. So give them the benefit of the doubt and assume that they, that they were acting in good faith and posting their paper on the archive in order to get feedback and improve the work. Read the version that they actually submitted. Don’t worry about the earlier version.

Noah Smith
23:07

We all have early drafts. Those are not what are reviewed, what’s reviewed is what’s submitted and give the paper as fair reading as it deserves. Like every other paper.

Matt Gardner
23:14

So you mentioned conflict of interest and bias. I want to push on this a little bit. Is it biased if it’s work in my area and I already have an opinion about the work of the paper.

Noah Smith
23:25

No, you can’t avoid that, right? You’re entitled to your opinion. And you know, we want papers should be reviewed by researchers who are deeply familiar with the related work and the past work and have even made contributions there. We want expert reviewers. That’s exactly what we need. A conflict of interest would be something like me reviewing your paper because we work together or me reviewing Waleed’s paper because we work and recently he was my PhD student.

Noah Smith
23:47

A conflict of interest would be reviewing a paper by someone who’s in a financial relationship, like you know, a sponsor of your research. That kind of relationship feels like it would muddy the water. Anything where it’s possible that the outcome of the review process you might stand to gain from one outcome over another. It looks to me like a conflict of interest now. Now that said, I’m giving you my kind of intuitive, subjective definition of this. Some research communities have very formalized definitions of exactly what counts as a conflict of interest. Reviewing agencies have extremely formalized notions of this. It’s not universally agreed exactly what is or isn’t and there are certainly some gray areas. You know, I finished my PhD many years ago. Do I still have a conflict of interest with my PhD advisor? I would say probably not at this point, but there might be reasons to dispute that.

Noah Smith
24:33

There’s definitely going to be some differences of opinion and you know, to be honest, I don’t think conferences often lay these things out as carefully as they should. The main thing that we’re able to catch more or less automatically is institutional conflicts of interest. So I should not review a paper or by anybody from AI-2 or the University of Washington where I’m employed because I, you know, I have some small amount to gain. If my institutions have more papers accepted, I guess, you know, it’s going to be harder for me to be objective about the situation. So we do pretty good I think at catching those, but there are others and no automated system is ever going to completely catch them. This is actually one of the things that area chairs are supposed to try to do is watch for conflicts of interest, but they, you know, nobody knows all of the possible conflicts of interest.

Noah Smith
25:14

We just don’t have a centralized knowledge base that captures all of this in any clear way. So my suspicion is that we do assign papers to reviewers with some frequency that do have a COI and they may not even be aware of it, if they don’t know that they have it, then maybe it’s fine. But there are probably people who are much better experts on this. Maybe from a legal background who can, who can say a lot more, I guess. You know, at the end of the day we’re never going to be perfect at detecting these things. And I, and I think it’s more important to focus on the quality of the scientific communication that happens both in the papers and the reviews than on making sure we never ever have a conflict of interest. It’s just impossible to be absolutely sure it will never happen. But the archive thing, as you said when we started this conversation, really does raise the question and, and bring it into relief. And we really, I think we really should take a little bit more time to declare what we think as a community. We think are conflicts of interest and what are not and and what should somebody do if they’re assigned a paper that falls under that umbrella.

Waleed Ammar
26:08

Yeah, that makes sense. I think one of the other kinds of bias that people question often is whether the person reviewing is trying to demote this paper because we have another submission in the same conference or another conference that will kind of have a similar idea. Like you said, it’s hard to catch all of these cases.

Noah Smith
26:24

So this is sort of like a conflict of interest because if this paper under under scrutiny is going to be published, then it might detract attention or make it harder for you to publish your own work. I think in general, when people feel that way, it’s out of a lack of creativity or a lack of appreciation for the larger process. What I would tell a student who felt this way, like, Oh, you know, imagine we reviewed a paper together, me and a student. And they said, well, I like this paper. It’s really strong. But it makes me sad because once this paper is published, it’s going to make it a lot harder for me to publish my project. My immediate answer to that would be, well, it’s true that they, you know, that they’ve raised the bar for you a little bit. In fact, I’ve had this happen, right?

Noah Smith
27:00

You know, they’ve raised the bar for you a little bit. They’re going to make it a little, it’s gonna make it a little bit harder on the numbers or you know, they started to creep up towards your idea. So we’re going to have to do a little bit more work to make sure that we have something that really is bold and new. But you know, the truth is they’ve done the work. There’s nothing you can do about that now. It’s better it be out in the open so that you can address it. And you know, quite honestly, this was a solid paper. So there are things we can learn from it and they’ve saved us some time. Some of the tricks that they’ve used, we can now incorporate into our method and yeah, okay it’s going to take a little bit more time, but in the end we’re going to have an even stronger result than we would have had otherwise.

Noah Smith
27:32

So you know, kind of like the old quote about standing on the shoulders of giants. I think when you’re in this situation you have to train yourself to make lemons out of lemon-aid, sorry, make lemonade out of lemons and stand on the shoulders of those people who got there, you know, got, got somewhere close to where you were headed. But you know, it’s gonna be a long road. You’re not a, your whole career isn’t going to be based on getting to one little bit of territory first. And what you should be thinking about is in the, in the very long run, are we making progress on bigger problems? And it’s inevitable that that part of getting to solutions to the big problems our field is facing is going to come from many contributions from many people. So, you know, in some sense we’re all on the same team and our problems are so hard that there’s really plenty for everybody to do. I guess my general advice when you’re in that situation is to lay it aside, do what you would like them to do for you if the situation were reversed and continue to continue to do the best work you can building on the new results. You’re lucky to now have some advanced notice of, because wasn’t it great that you got to be assigned a reviewer of this paper?

Waleed Ammar
28:33

So another thought that I had is doing all this care for reviewing takes time and there’s a lot of paper requests, paper reviewing requests that we get, especially if you’ve been around if you’ve been publishing in this domain for for awhile, I wonder what would be a fair amount of reviewing.

Noah Smith
28:50

You could put it into a purely transactional terms and say, well, I wrote N papers this year, but really it’s fractional because I had co-authors, so you know, on a two author paper I wrote half or whatever. Or you could even work out how much you wrote on each paper. I don’t know. I don’t know if people can can do that. But you could add up how many of these things you did and then say, well there were three reviews for each of those submissions. And then say, well, I should, you know, I should contribute back that many reviews because I benefited that much. And so I should put back into the pool what I take out. And then maybe you should say, well, but you know, I’m a, I’m a more experienced researcher and it takes less time for me to do these things. Then someone who’s new and I have broader expertise or experience that.

Noah Smith
29:27

And so maybe I should, maybe I should do a little more. You know, you could do this calculation and people sometimes do, I can never remember, I’ve heard five different senior researchers offer six different opinions about how much this should be and what the multipliers are. And it only sounds crazy when you hear what the final number of reviews you should be doing this year is and it always sounds really high. And then you can ask, well what is it? What does it count? If I do some other service that’s not reviewing but is related to reviewing, like being an area chair, how much does that count? Because we need people to do that job. And we really, we also need people to be, you know, action editors for journals and do journal reviews, which are sometimes a considerably more work because the papers often have a lot more meat in them.

Noah Smith
30:04

So I would hesitate to put a number on it. I also kind of think different people have different strengths and should contribute in different ways. Some people are really amazing reviewers. If you’re an amazing reviewer and you’re getting a lot of encouragement, then consider this. Maybe you know a big portion of your service can be reviewing and that’s great. The community needs it. I’ll say it again. If you’re, if you’re also getting these awards for being a best reviewer, consider doing a podcast with Matt and Waleed and help other people learn how to do it too. I’m sure you guys could sustain a a couple more of these interviews. I’m not the only person with things to say about this. I had one other point about this, which is that reviewing isn’t entirely service, particularly as you get farther along in your career and you have less time to sit down and peruse papers.

Noah Smith
30:44

Reviewing is how you keep up. It’s part of how you keep up and it’s part of how you learn what’s going on. And so while you know some fraction of the papers you get, you’re not going to learn a whole lot from because there’s a lot more to be done and they’re not ready. There will be papers that you get assigned that give you really good ideas and you’re grateful for having had the opportunity to review them. That sounds really corny, but it actually is true that somebody does review the good papers and sometimes it will be you. And there’s a lot of value in sitting down and doing that deep reading largely, you know, on the surface for altruistic reasons. But there are selfish reasons to do it too. And so I justify the amount of time I put into this activity by saying, well, this is, you know, this is my opportunity to update myself this month on what’s going on in the field. I guess it’s a, it’s an alternative to reading the archive feed every morning.

Matt Gardner
31:28

Yeah. Yeah. Good points. My last question for you is more about at a high level, what’s the point of a review, what are we trying to get out of this? And in particular one aspect, so I’m remembering it was maybe a year ago Yoav Goldberg had a big rant about a natural language generation paper that was posted on archive about how it was done totally wrong. If you strongly disagree with some aspect of the paper that you’re reviewing, maybe it’s using bAbI to make claims about natural language or maybe your Yoav and you’re reviewing this NLG paper. What should you do?

Noah Smith
32:01

So the first thing to say is I think, I think the speech act that Yoav was producing when he, when he wrote that rant, was not really intended to be a review of that paper. I think he was using that paper as an example of a larger pattern that he was, that he was frustrated by. But I think if he had written a review of that paper, he would have done it in a very different way than the rant. I think the speech act that you’re trying to perform when you write a review, I guess there’s sort of two parts, right? So this is maybe a good recapitulation. The first part is to help the area chair come to a decision about what to do with the paper. That does not mean you’re trying to find a reason to get the paper rejected because you know that 80% of papers or never going to be rejected, it really means that you want to help them make the right decision to the best of your ability.

Noah Smith
32:43

And you know you recognize there are going to be other reviewers, but you want to pull your weight. So your job is to read the paper carefully, figure out what’s in it, discuss the strengths and weaknesses and give the area chair all the information they need to make a decision. The second act that you’re performing at the same time is again to nurture the community. Help the people who wrote this paper make the camera ready version of their paper even better, or make the next version of their papers something that you would be excited to read and more much more likely to accept. We often take a very adversarial view of this process and we always have, I’m not going to claim that it was better in the old days when there were fewer people. Things can get quite nasty, even back then, maybe even nastier than they are now because things were maybe more personal and everybody knew each other a little better.

Noah Smith
33:25

We tend to emphasize the adversarial nature of this process and you know, what are my odds? What’s the chances paper’s going to get in? We really fixate on the scores. If you have an average above four, you’re probably okay. What do we need to do to, to satisfy this reviewer, to win them over and convince them over to our side? And it all takes on this very persuasive different actors trying to maximize their, their gains. That’s really not the way we ought to think about it. We really all ought to take a step back and say what we really want is for the scientific field to move forward and for results that people need to know about to be published and for findings that are going to accelerate our pace to be published and for important new information that’s going to affect the way people do things to get out there.

Noah Smith
34:05

And so I think if we, if we focus more on that and we focus more on, you know, the information that people need to make these decisions and if we all act a little bit more dispassionately, then the process doesn’t have to be quite so fraught. We are in a new time, there are, there are a lot more papers being written. There’s a lot more new confusing situations that we find ourselves in new kinds of conflict of interest and so on. But ultimately the goal here is to get the claims out into the world that need to be heard. And so to the point of like, what if I fundamentally disagree with something about this paper? I guess the answer to fundamental disagreement in my view is never censorship. So I might fundamentally disagree with some of the things in a paper or the way something was done or the assumptions that were made.

Noah Smith
34:45

But again, your role as a reviewer is not to make a value judgment on everything about the paper. It’s to focus on the argument that’s being made and whether the findings are going to have value for the research community. So you know, you might fundamentally disagree with a question that’s being asked and you might say that in your review you might say, I don’t think this is the right question. I think it’s more important to be asking this or I think this data set is problematic as pointed out in and then give citations. But that doesn’t necessarily imply that the whole thing should be tossed out and maybe it does sink the paper and maybe there because there aren’t enough strengths on the positive side to justify it being accepted. Again, it’s your job to play both sides and seek those arguments in favor of the paper.

Noah Smith
35:26

But I think again that you can’t assume that your fundamental disagreement with the paper implies it should be rejected. And even even if you deeply disagree that doesn’t, the papers shouldn’t be published. No one of us has a monopoly on what ideas are to be heard and not heard. If you disagree with an idea, the best thing is to have it is to have it voiced so that you can give your counter point to it and fight back suppressing the idea is certainly not going to lead to the result that you want.

Matt Gardner
35:53

I want to push on that a little bit because it’s these reviews that determine best paper awards that determine what gets highlighted at conferences. You said we shouldn’t make value judgments in these reviews, but we are implicitly anyway because our review determines what’s going to happen. Right. And you also said it should be published so that I can respond to it. What’s the right place to respond to it?

Noah Smith
36:14

Well, the next paper.

Matt Gardner
36:15

Okay.

Noah Smith
36:15

If you think about the scientific discourse across a question or a problem over a long period of time, it often has the feeling of a conversation. There’s kind of a back and forth and it’s ideally a very rational conversation focused on the facts and the arguments. You’re right, I shouldn’t say that we don’t make value judgments. We do have to make value judgments, but the value judgment should ideally come from a position of an a reasonable dispassionate scientist who’s seeking to find out what’s true and we’re always going to have disagreements about certain assumptions. There’s room for that, and again, you can state that you disagree with some of the fundamental underlying ideas that are assumed by the paper, but you shouldn’t then necessarily fight to have it rejected. But yeah, I would say your response is going to be, well, you know, it could be, it could be the next paper, it could just be a blog post critiquing that paper in particular or a trend in general.

Noah Smith
37:02

I understand the frustration of seeing work that you fundamentally disagree with winning awards and being widely recognized. It is very frustrating. I was fortunate to have gotten advice early, very, very early in my career before I even started grad school, that being a researcher was going to have these frustrations that you will see people getting credit for things you don’t think they deserve credit for and your ideas will be attributed to others. And there are all these things that we take very personally and can go horribly wrong in scientific fields and they’re extremely frustrating. And so I guess you know, expecting otherwise is setting yourself up for pain and disappointment. That doesn’t really make the problem go away. But ultimately the one kind of reassurance that you should take is if the field is moving forward and you know that you’re contributing to it, then that should be in some ways enough. And you should take satisfaction in that. And these awards, you know, they often look quite silly in retrospect, the conversation never ends after the final session of the conference. There’s going to be another conference, they’re going to be more papers. There’s always more to say. So don’t take these things too seriously and focus instead on what’s true.

Matt Gardner
38:05

Great. Thanks. This has been a really interesting conversation. Thanks a lot for suggesting this topic. It was really nice talking with you.

Noah Smith
38:10

Thanks for having me. Looking forward to the next time.