NINDS's Building Up the Nerve

Episode 6: Scientific Review

• NINDS • Season 1 • Episode 6

Share episode

Learn what happens during study section and the role of a scientific review officer from SROs Drs. Ernie Lyons (NINDS Review Branch Chief), Shanta Rajaram, and Bill Benzing.

Building Up the Nerve is a podcast from the National Institute of Neurological Disorders and Stroke for neuroscience trainees that takes you through the life cycle of a grant from idea to award at NINDS with the people who make it happen. We know that applying for NIH funding can be daunting, but we’re here to help—it’s our job!

Transcript: http://ninds.buzzsprout.com/558574/2309078-episode-6-scientific-review

Lauren Ullrich:

Welcome to the National Institute of Neurological Disorders and Stroke's Building Up the Nerve, a podcast for neuroscience trainees that takes you through the life cycle of a grant from idea to award at NINDS with the people who make it happen. We know that applying for NIH funding can be daunting, but we're here to help. It's our job.

Marguerite M:

Hi, I'm Marguerite Matthews, a health program specialist at NINDS.

Lauren Ullrich:

and I'm Lauren Ullrich, a scientific program manager at NINDS and we're your hosts for this episode. So last time we discussed submitting your application and receipt and referral for assignment to an Institute. Today we're going to talk about what happens during review.

Marguerite M:

And as always we want to provide a disclaimer that everything we talk about in this podcast may only be relevant to NINDS. And if you're applying to a different NIH Institute or center, it's best to check with them about their policies.

Lauren Ullrich:

So I was really lucky in my graduate work--I went to Georgetown, so we are very close to NIH--and we had mock study sections at our first year with actual SROs. It really was central to opening my eyes to how the process works because I think there's nothing like experiencing it to really understand what goes through the mind of a reviewer. And Marguerite, did you have any experiences like that in grad school?

Marguerite M:

Unfortunately not as a trainee. The NIH review process was completely foreign to me and since I didn't submit an F31, I never got a chance to get any summary statement or reviews back. So I didn't learn about the NIH review process until I started working at NIH.

Lauren Ullrich:

Yeah. And I think unfortunately that's a lot of trainees' experience, so...

Marguerite M:

So that's why we're talking about it today!

Lauren Ullrich:

Exactly.

Marguerite M:

Have a lot in store.

Lauren Ullrich:

Joining us today are Dr Ernie Lyons, chief of NINDS's scientific review branch, Dr Shanta Rajaram, scientific review officer who runs the NINDS clinical trials NSD-K study section and Dr Bill Benzing, scientific review officer who runs the NST-1 study section and other training-related special emphasis panels. So let's start with introductions.

Ernie Lyons:

As you mentioned, I'm the chief of review here. So I manage the review branch and I manage all of the assignment of all the incoming applications to the study sections and other receipt and referral processes. I help manage developing our FOAs and setting up receipt dates. Just, you know, make sure that all our reviews follow all the rules and regulations and are conducted appropriately. And I've worked here since 2001 so quite a while. I was an SRO for 10 years before I became chief. And before I started here I was a research associate at a university and you know, academic research. I had small grants and private grants. I did write some NIH grants; I never got funded. My hobbies, I have several hobbies. I like to garden and I like to play music.

Shanta Rajaram:

I, as you said, I'm a scientific review officer here at NINDS. I have been in some ways a jack-of-all-trades. I've done all sorts of reviews through my 15 years at NINDS, but I primarily manage the reviews of our clinical programs. Uh, in addition to that, I'm involved in initiatives such as the BRAIN and HEAL and I've done a few odd training reviews here and there. What I did before I started here, before I started at NINDS, I was in the intramural research program at NIMH. That allowed me an opportunity to do a detail in, in a branch that I might be interested in, um, for as a future career. And that's how I ended up, uh, as a detail yet at NINDS and ended up getting a job as an SRO here 15 years ago. Passions outside of work is actually rescuing dogs. Um, I have two of them at home right now, but I'm very much involved in the volunteer organization that rescues those animals.

Lauren Ullrich:

Nice. I want to see pictures.

Marguerite M:

Yeah, same.

Shanta Rajaram:

Absolutely

Bill Benzing:

I'm a scientific review officer at the NINDS as well. And, uh, I've been at the NIH as long as Ernie actually almost 20 years. And I was at CSR, center for scientific review, first for a while, and then I came to NINDS about 13 years ago. Uh, what I did before here, I worked in academia and then in biotech. My research focus was age-related neurodegenerative disorders. As far as hobbies or passions, obviously my kids and gardening and music and I've rescued a few dogs for my own sake. Dogs are a large part of my life as well.

Marguerite M:

All right. So can one of you tell us about the relationship between the center for scientific review and reviews that are done at the IC level?

Ernie Lyons:

So the center for scientific review, as you mentioned, is one of the 27 institutes or centers at NIH. And they handle the majority of reviews at NIH, about 75% of the applications are reviewed there. And just in general you could say they review the more standard applications that are typically submitted in response to parent funding opportunity announcements, like the standard R01, just regarded as the gold standard. I think it's the most common NIH application and it's, as I said, sort of a gold standard at academic institutions to have your lab funded through an R01. They also review R21s, R15s, small business, so many other mechanisms. But institutes as well have their own review branch, which tend to, and again this is general, review applications that are submitted in response to more specialized funding announcements and often have um, institute-specific review criteria or eligibility requirements or definitions. For example, translational research may be defined differently at different institutes. So often, uh, we review that. We have a translational review program that reviews our translational research, clinical research, training, as you'll hear a lot about. And as well as many RFAs and other programs like our centers awards, centers grants, core grants, and those kinds of complex mechanisms.

Lauren Ullrich:

That's a lot of different kinds of mechanisms. I don't think when I was a trainee, any of those things were really on my radar. And so specifically for training at NINDS, which training mechanisms are reviewed at CSR and which ones are reviewed by us at NIDS?

Bill Benzing:

Um, the ones that we do in NINDS review branch, uh, specifically for training, while we do review other types of grant mechanisms, are the F32s the various K01 mechanisms, clinician scientists, independent junior faculty K02, and the clinician scientists mentored training programs, K08, K22, and K23. And we also have diversity versions of, of many of these mechanisms including a diversity F99/K00 and a diversity K99. And diversity K01 and diversity K22. Uh, we also review, um, the more departmental type training programs like T32s and summer research programs like R25s.

Marguerite M:

Once I've submitted my application, um, where can I check exactly where my application is being reviewed?

Shanta Rajaram:

So you can check the locus of review for your application in your commons account--in NIH ERA commons account once you've submitted it. An alternate place to look, although specific review panel may not be listed at that spot, a good place to check is also to look under the agency contacts listed under the funding opportunity announcement for your review locus.

Ernie Lyons:

And in section five of the FOA, section five number 2 lists the locus of review, so you can know ahead of time where it's going to go.

Lauren Ullrich:

Right. It's kind of buried in there a little bit. Hard to find if you don't know what you're looking for. And specifically if I want to know what study section my application will be reviewed in, where do I go to check that? And also who is going to be on the review panel?

Ernie Lyons:

I believe the easiest way to know your study section assignment is: it's posted in your ERA commons account once it's assigned to a study section and I think there may be even a link there to the roster to see who's on the roster. Rosters typically become available 30 days befor the study section meets. That would be the meeting roster. You can also preview the committee members for standing committees, you know, to see who the regular members are on the NIH website. There's a helpful tool in, in the NIH reporter called the matchmaker tab and you can paste in a keywords or an abstract and it will match up a potential study sections and you can actually suggest a study section when you submit your application either in the cover letter or on the assignment request form that's available during the submission process.

Marguerite M:

And how are reviewers chosen to review applications?

Bill Benzing:

Well, a number of factors go into how we choose reviewers. At the level of, I'm looking for reviewers for an individual application or a group of applications or for a whole study section, we typically look for individuals, definitely with a doctoral degree or equivalent. Most importantly, we look for demonstrated scientific expertise in the relevant area of either an individual application or a group of applications. One of the criteria that we look at is a track record of independent research support and demonstrated breadth of perspective. When we get to know reviewers, we really want reviewers that can provide an overall broad objective perspective, free of bias and we want reviewers that can exercise sound and mature judgment. And most importantly individual reviewers should also be free of personal professional or financial conflicts of interest to review the set applications. At the level of entire review panel, additional factors go into creating the makeup of an of a whole panel and these include diversity, geographic distribution of the reviewers, and some other factors that get factored into those things.

Marguerite M:

So you don't purposely pick investigators who are going to be the third reviewer to be out to get a given application? That's not how it works?

:

[Laughter]

Marguerite M:

They're not looking for people to be extremely harsh in their review? Cause I think as trainees especially, we see what our mentors and other faculty are going through and we just assume that reviewers are purposely picked to just be mean and give bad reviews.

Shanta Rajaram:

And actually we come to some of this in one of your questions in the sense that applicants are almost always wrong when they try to guess which reviewers on the panel was out to get them. The support for your application may come from the direction you least expect. So I always advise applicants don't guess and certainly don't assume the worst based on a roster.

Ernie Lyons:

And I would add just in my long experience: in general, reviewers really aren't out to get anyone. I think the important thing to remember is that study sections are made up of scientists, they're all humans and we basically ask them for their opinions. Maybe, I think overall the process works very well and it's probably the best we have, but that doesn't mean they're always perfect or they always give us the right answer or the right opinion or you might not agree with some of the things that they say, but...

Marguerite M:

Right.

Lauren Ullrich:

Earlier we mentioned standing study section. And so what is the difference between a standing study section and a special emphasis panel and how, um, how is it decided what applications go to which kind of review panel?

Shanta Rajaram:

So a standing study section has appointed members. Usually it's anywhere between 16 to 24 and they are appointed for terms that are typically between four or six years long. The standing study sections usually meet three times a year and as additional expertise not provided by the standing members is required based on applications that get assigned to the study section, ad hoc expertise gets recruited each round. A special emphasis panel or a SEP as NIH tends to call it--one of its millions of acronyms--

Lauren Ullrich:

Yeah we love our acronyms.

Shanta Rajaram:

Is assembled de novo each time. It is usually usea for reviewing applications in response to say to request for applications commonly known as RFAs. These are typically one time solicitations. So these are the kinds of initiatives that get reviewed in special emphasis panels or they may be used for special programs that might require unique review criteria, for example, or very common other use of special emphasis panels is to review applications from members of standing study sections. The so called member conflict SEPs. In terms of deciding which applications, I think I talked a little bit about how applications or which applications typically get assigned to special emphasis panels. In study sections they could be divided by a number of different ways. In NINDS we have different study sections for clinical, translational, and training applications. And each of these, even between the translational applications, the two study sections have divided based on areas of science that NINDS funds, that are divided between the two study sections.

Marguerite M:

And ultimately how does this influence the actual review of applications?

Ernie Lyons:

I think in general, the process between the two types of study sections, standing committees or special emphasis panels, is basically the same. As Shanta mentioned, SEPs are usually created in response to special, you know, typically one time announcements and they're tailored specifically for that review. Standing committees tend to review things that come in on a regular basis like many of our training awards that we have. They come in three times a year, and so we have two standing study sections that review our training. I think that the difference there is that a standing committee provides some continuity and consistency from round to round. Those study sections develop a culture, they become familiar with the training mechanisms and hopefully provide some consistency in their scoring behaviors across different rounds.

Lauren Ullrich:

So shifting gears a little bit, in terms of the application, are applicants allowed to send any updates on their application after it's submitted?

Bill Benzing:

Yes. Up to 30 days prior to the review meeting. You're only allowed to submit items that are allowed in the NIH policy for post submission materials, unless the funding opportunity announcement that you're applying to, the FOA, states otherwise that no post-mission material can be submitted or otherwise allows specific additional materials than in the standard policy. The NIH policy on post submission materials can be found at the NIH notice NOT-OD-19-083.

Lauren Ullrich:

And what kinds of materials generally are allowed by the policy?

Bill Benzing:

There's a handful of things. But the most common thing are accepted publications or recently published manuscript that have become published since application has been submitted.

Shanta Rajaram:

And what is most commonly not allowed and which we get the maximum questions for, is anything that is a fixing of an error that you made during submission. So check the final form of the application once the PDF is in.

Ernie Lyons:

And some of our announcements do allow late submission of um, of recent data. So if, if you have more preliminary data that would support your application, some of our FOAs allow late breaking data to be included.

Bill Benzing:

That would be listed specifically in the funding opportunity announcement.

Lauren Ullrich:

So read the FOA. Carefully.

Shanta Rajaram:

Absolutely.

Marguerite M:

So now that we've established that selecting reviewers is not quite the hunger games, what actually does go on during the review?

Shanta Rajaram:

The process actually starts a lot earlier than the actual reviews; about four to six weeks prior to the actual meeting. Each application is assigned to a number of reviewers, um, typically three, but the more complex the application, the more the number of reviewers assigned to it. The reviewers get access, as I said, to not just their assignments but also to review instructions, relevant guidelines through a secure web interface that NIH uses for this for this purpose. About a few days before the actual review the reviewers are expected to upload preliminary scores and critiques. Now if it's a special program or special initiative with special review criteria, as we said, there might even be a reviewer orientation call to make NIH ensures with involvement of both program and review staff on those calls that all reviewers are on the same page in terms of understanding the goals of that specific program. Once the reviewers have posted their own critiques and scores, they are also allowed to read the comments posted by others assigned to review the same application. So there is a little bit of an understanding of where the others are coming at when the discussions actually start. At the meeting, there is a chairperson who facilitates discussions. Discussions very typically focus on the strongest half of the applications that have been assigned to that study section or that panel. Reviewers take turns stating the preliminary scores and going over their main score driving issues. We try to get them to focus on both the strengths and the weaknesses in the application. All non-conflicted members are then invited to join the discussion once the assigned reviewers have had their say. Once it appears that the score driving issues have been hashed out completely in a thorough discussion, the chair will then specifically invite comments on other score driving criteria such as human subjects, protections, vertebrate animals, inclusion policies, biohazards, so on and so forth. Um, once this is done, the review chair person asks for final scores. The assigned reviewer state their final scores, which sets the score range,and reviewers are asked to vote their conscience based on that score range. Now our reviewers in the room may choose to vote outside the score range, but any intent to do so must be declared publicly at the meeting when the chair asks. Lastly, the chair will ask for comments or concerns on any non score influencing elements. These are administrative elements such as budget, if the organization requesting funding is a foreign entity, issues such as resource sharing plans, authentication plans for biological chemical resources, et cetera. And that typically ends the review discussion for an application.

Lauren Ullrich:

One of the things that we get questions about a lot is this difference between the different criterion scores that you might see on your summary statement and the overall score. So how do reviewers come up with that overall score? Do they just average the five criterion scores or how does that work?

Bill Benzing:

Well, um, let me give a little bit of background on just the scoring first. The NIH uses a nine point scoring system with one being the best score and nine being the worst. And most research grant applications reviewed at the NIH are evaluated on five core review criteria of which you're alluding to is the these individual criterion. Um, these are typically significance, investigator, innovation,, approach and environment. For training and career development applications. These individually scoring criteria are different and may include applicant, sponsors, which is also mentors, collaborators, consultants, research plan, training, plan, training, potential, and institutional environment and commitment. Each of these get an individual score, which is called a criterion score. The overall impact score, which is an overall opinion of of the uh, application by a reviewer is an assessment of the likelihood of the project to exert a sustained powerful influence on the research fields involved or the training of the individual applicant if it's a training grant, and its determined by reviewers by considering the five individually scored criteria, but it's weighted on the overall reviewer judgment. And the overall score includes human subjects issues, animal welfare concerns, and things like that as well.

Lauren Ullrich:

And those are things that they're not individually scored, but they can count towards the final overall score

Bill Benzing:

Right. If there's issues with human subjects, issues with the scientific justification for using the humans subjects or the animals, um, those can affect the overall score. So the overall score is an impact score of the whole application and individual criteria and scores are a judgment each of those different individual criteria.

Marguerite M:

So the final number that applicants see is the impact score. How is that calculated?

Bill Benzing:

The overall impact scores provided by all reviewers in the study section are averaged and then rounded mathematically to one decimal place. And then this is multiplied by 10 to give you the two digit impact score. So if something had an average of say half the panel or roughly half the panel gave 1s to an application and the other half gave 2s that yielded an average of say 1.5, it would yield an overall impact score of 15.

Lauren Ullrich:

And in addition to the impact score, many summary statements also provide a percentile. And what is that and how is it calculated?

Ernie Lyons:

So a percentile is essentially just a way of ranking the applications. It's a ranking mechanism and it's expressed as a percentile. And it's really the rank of your application's score divided by the total number of applications. So for example, if there are hundred applications and your score ranked 10th on the list, it was the 10th best, you would receive a 10th percentile. And that's a little bit of an oversimplification, but that's essentially what it is. But importantly, how we use it and what it allows us to do is to compare scoring or ranking across different study sections so we can fund the same percentage of applications, the top 15% for example, from each study section. So where study sections may have slightly different scoring behaviors, we're still funding the same percentage from each study section. And percentiles are also calculated across three council rounds, which helps to account for round to round variability. So if you get a particularly strong set of applications, one round and a weaker set the other round, that tends to normalize over the three rounds.

Shanta Rajaram:

I would like to add though that NINDS does not percentile any of the applications, either in our study sections or special emphasis panels. So really reading the summary statement and looking at your score and what those numbers mean in terms of descriptors is usually very helpful.

Lauren Ullrich:

Yes, that is a very common question that we get because most of our training grants are reviewed in house and so most of them don't get a percentile. And I think a lot of mentors, they're used to just zeroing in on that percentile and disregarding the impact score. And so when a trainee gets their review back and it only has the impact score, they're a little bit at a loss of how to, um, how to interpret that.

Marguerite M:

How do applicants know the outcome of their review?

Shanta Rajaram:

So the score and the summary statement are both posted in the ERA commons. And again, the applicants can access it by signing into their commons accounts. Typically applicants receive an automated email notification as soon as their scores are released, but we always recommend that you check periodically. The scores are released within a couple of days of the study section, of this review panel meeting. The summary statements can take longer, anywhere from four weeks from the review date to a month before that particular council review round. So we always ask that applicants check their commons accounts periodically because there's always technical glitches due to which that automated email may never actually make it to them.

Lauren Ullrich:

And if you have questions about the review, who is the best contact person?

Bill Benzing:

This is a question that often gets misunderstood. If its prior to the review and prior to the score coming out, the person to contact is the scientific review officer, which can be found in the review section related to the application in your commons account. Prior to the review, anything that's related to submission of additional materials or other questions should be directed to the scientific review officer. If it's post-meeting and the scores have been out and/or the summary statement is out, they should be contacting their program director or their assigned program officer.

Marguerite M:

Are there opportunities for early career scientists to participate in review?

Shanta Rajaram:

Yes, absolutely. In fact, most SROs welcome the perspective that junior investigators often bring to these panels. The center for scientific review, which you've heard of--us mention several times now, it actually has an early career reviewer program, so called the ECR program, that is designed to help emerging researchers advance their careers by exposing them to review early on. In fact, I always tell applicants that one of the best ways to improve your grant writing skills is to serve as a reviewer. So I encourage applicants to both review, uh, both register for the ECR program or if they are interested in an IC review panel, to just send their CVS, reach out to the chief of the scientific review branch and send us your bios.

Lauren Ullrich:

And is there anything, common misconceptions, things that you wish that applicants understood about this process?

Ernie Lyons:

So what I was planned to say here, I, I've already mentioned a little bit: to remember that the study sections are comprised of scientists who are humans. They have a lot of grants to read. They're not going to get everything right all the time. But for the most part, their intentions are good. They're trying to do their best to identify the best science and discriminate between large numbers of applications. It's very stressful for reviewers; they have to present critiques of multiple applications in front of their colleagues, in front of their peers. And for the most part, again, they're just doing their best to try to review these as accurately as they can. But it's, you know, it's certainly not a perfect process.

Marguerite M:

And I think especially for training opportunities, reviewers really want the best for the trainee. So they really look for opportunities where the trainee has, has highlighted some great things and often want to highlight the weaknesses more as an opportunity for growth for both the trainee and the mentor.

Shanta Rajaram:

I absolutely agree. I sometimes think that, especially for some of our training study sections, if the applicant could be a fly on the wall when the study sections actually go on, they would really get a flavor for how much the reviewers actually look out for these--for the training and for the applicant and for what is important for their career. And the reason why sometimes things don't score as well is because they just don't think that that meets what the applicant needs to further their career.

Lauren Ullrich:

Right. And resubmission is a chance to get those weaknesses addressed and commitments to you and your training in writing. And once they're in writing, in a grant, then we at NIH through the progress reports have some authority to make sure that those things happen. But if they're not in writing then we can't help you as well. So thank you all so much for coming on the podcast today. And can I ask each of you for one last piece of parting advice for applicants?

Shanta Rajaram:

Sure. Um, if I were to pick the top three that significantly influenced review outcome, my top three would be: make sure that the mentor and the trainee have read each others' portions of the applications and the application clearly states, mentor really states what the mentee can take away with him or her. Another issue is the description of the actual training plan. It must fill specific gaps in the applicant's research experience and the goals should be specific of the training plan should be specifically linked to the proposed research plan. The last point amongst my top three would be the mentoring team should be well suited to the proposed plan. If your primary mentor doesn't have experience mentoring candidates who have moved on to independent positions, make sure there is a co-mentor who's senior and much more experienced in this area. If you are a clinician moving towards a clinician-scientist career, make sure that if your primary mentor is an academic doctor, make sure that you have a clinician or a professional doctor as a co-mentor. Essentially, if you don't get funded the first time, always try again. You know, you miss 100% of the shots that you don't take. So carefully look through the critiques, seek advice from others who have both been successful grantees and as well as experienced recent reviewers on NIH panels and think carefully, revise and resubmit. Sometimes it's simply a matter of knowing what just needs to be explained better versus something that actually you have to go back to the drawing board and readdress.

Ernie Lyons:

And I agree with all of those. I would add, as Shanta said, it's very helpful to show your application to colleagues. Maybe even besides your mentor, someone who's had experience on review panels, start early and read the funding opportunity announcement carefully that you're applying. Too often we see applications come in where it appears that they've left things out or just maybe haven't read the opportunity announcement or don't understand it and have submitted to the sort of to the wrong thing and if you aren't clear about this or even in general, talk to the NIH, talk to your program directors, folks like Lauren and Marguerite. They're here to help you and guide you through the process.

Lauren Ullrich:

That's our tagline of this podcast.

Marguerite M:

Yes.

Ernie Lyons:

Make use of them.

Bill Benzing:

and going last, they've already stolen some of my thunder but, Ernie I know just indicated to have other people read your application before you submit and it's really important to get people that do not know what you're doing or not know enough about what you're doing because they can fill in the gaps where things are not written clearly. And for training applications, I wanted to add: specifically, very clearly didactically state how the research plan addresses the gaps in your career development or your training plan. Be very, I mean don't, don't assume that the reviewers will link those together because they want the research plan to support the gaps in your training that you're trying to fill and how this research will accomplish that.

Lauren Ullrich:

And Marguerite, do you have advice?

Marguerite M:

Yeah, I'd like to build on something Shanta mentioned earlier. Um, think about serving as a peer reviewer for your colleagues and other folks that you're with so you get to be in the reviewer seat and you get to see their application and really be thoughtful about what's in the criteria for the funding announcement and have they addressed all of that. And you can give critical feedback that may help them think about have they reached all the points that reviewers should know about their application? And again, you serving as a reviewer will help you be a better grant writer.

Lauren Ullrich:

That's exactly what I was going to say.

Marguerite M:

See now we're sharing a brain.

Lauren Ullrich:

Great minds think alike. But just to build on that, I think there, there are opportunities to serve as a reviewer even as like, a graduate student. So, for example, at Georgetown we started a very small scientific grant program that was completely run by graduate students. So you would submit, you know, a small proposal and graduate students would review, have a discussion--we modeled it exactly after NIH--and then you could get, you know, a couple thousand dollars. And so there's opportunities like that. Oftentimes like travel grants or through like post-doctoral offices and things like that, they're looking for reviewers. So even if you're not at the stage where you can review at NIH, you can still get the perspective of what reviewers are looking for and other opportunities. So that's all that we have time for today on Building Up the Nerve. Thank you again to our guests this week for sharing their expertise and thank you to program director Bob Riddle for our theme song and music and we'll see you next time when we tackle program review and recommendation. You can find past episodes of this podcast and many more grant application resources on the web at ninds.nih.gov.

Marguerite M:

And be sure to follow us on Twitter@NINDSdiversity and@NINDSfunding. Email us with questions at nindsnervepod@nih.gov and be sure to subscribe to the podcast on Apple Podcasts or your favorite podcast app of choice so that you don't miss an episode of this great content. We'll see you next time.