NINDS's Building Up the Nerve
Season 5 of the National Institute of Neurological Disorders and Stroke’s Building Up the Nerve podcast helps you strengthen your science communication skills with tools and advice to use throughout your career. We know that navigating your career can be daunting, but we're here to help—it's our job!
NINDS's Building Up the Nerve
Episode 7: Program Recommendation
Learn how program staff decide whether to recommend funding from Dr. Lyn Jakeman, Director of Neuroscience, and Program Directors Drs. Amelie Gubitz and Jim Gnadt.
Building Up the Nerve is a podcast from the National Institute of Neurological Disorders and Stroke for neuroscience trainees that takes you through the life cycle of a grant from idea to award at NINDS with the people who make it happen. We know that applying for NIH funding can be daunting, but we’re here to help—it’s our job!
Transcript: http://ninds.buzzsprout.com/558574/2400797-episode-7-program-recommendation
Welcome to the National Institute of Neurological Disorders and Stroke's Building Up the Nerve, a podcast for neuroscience trainees that takes you through the life cycle of a grant from idea to award at NINDS with the people who make it happen. We know that applying for NIH funding can be daunting, but we're here to help. It's our job.
Marguerite M:Hello, I'm Marguerite Matthews, a health program specialist at NINDS
Lauren Ullrich:and I'm Lauren Ullrich, a scientific program manager at NINDS, and we're your hosts today. In the last episode we discussed the scientific review process
Marguerite M:and today we're going to talk about the program review and recommendation for funding of grant applications. As always, we want to remind you that everything we talk about may only be relevant for NINDS, so if you're applying to a different Institute or center, it's always best to check with them about their policies.
Lauren Ullrich:So joining us today are Dr Lyn Jakeman, director of the division of neuroscience, Dr Jim Gnadt, a program director in the systems and cognitive neuroscience cluster and Dr Amelie Gubits, a program director in the neurogeneration cluster. Let's start with introductions.
Lyn Jakeman:Hi, how are you? I'm Lyn Jakeman and I'm the director of the division of neuroscience. This is the division that oversees most of the program directors that have portfolios in the neuroscience areas. Um, so I've been here about a little over five years. I came here as a program director. I managed the spinal cord injury portfolio for about five years. And before then I was at the Ohio State University where I was a faculty member in the department of physiology and cell biology there. I also worked a lot with the neuroscience department. Um, and I really enjoyed working at Ohio state; I enjoyed teaching, I enjoyed research and I enjoyed, um, service and I came here to kind of further my service career. At home I do a lot of, lot of things. Um, and I think one of the things I'm doing right now is learning how to quilt. So that's been a lot of fun.
Lauren Ullrich:Nice
Marguerite M:My mom's a big quilter and so is Lauren's mom.
Lauren Ullrich:I learned it from my mom. Yeah, I love quilting.
Lyn Jakeman:Yeah, it's all about putting little squares together and making bigger pictures. It's lots of fun.
Amelie Gubitz:and I'm Amelie Gubitz. I'm a program director in the neurodegeneration cluster and my program area focuses on adult onset neuromuscular diseases. So for instance, many of the grants that I manage focus on ALS, but I also cover some of the ultra rare neuromuscular diseases like Kennedy disease and hereditary spastic paraplegia. So yeah, I've been at NINDS for a little over 10 years now. And actually when I first started at NINDS, I was part of the neurogenetics cluster. I was an analyst supporting a group of five program directors there. And then a few years later I moved over to the neurodegeneration cluster as a program director. So that's my current position. And before I came to NINDS, I was a research assistant professor at the University of Pennsylvania and I did, um, the disease mechanistic research on spinal muscular atrophy, which is a juvenile and, um, pediatric neuromuscular disease.
Jim Gnadt:Hi. So I'm Jim Gnadt and I'm a program director in the systems and cognitive neuroscience cluster at NINDS and, uh, my portfolio's in systems neuroscience and computational neuroscience. Um, so I've been at NINDS for also a little over 10 years. Um, and before I came here I was a independent investigator since, uh, 1990 when I got my first faculty position. Uh, I've been at several universities but most of my time was at Stony Brook University in New York. And I worked on sort of neuroengineering approaches to understanding circuits in the brain.
Marguerite M:So one of the first things posted after the study section is the priority score and the percentile of an application. Can one of you tell us what the difference is between a priority score and a percentile and what do those numbers mean to the applicant?
Amelie Gubitz:The score is sort of an aggregate score. So the final score from every reviewer who sits on the study section. So each reviewer votes with one single score. And when you get the summary statement, you'll also see that there are criterion scores. And one question I often get is that people ask what's the algorithm to get from the criterion scores to the final scores? And the answer is there is no algorithm. The criterion scores just provide additional guidance to the applicants, what the strengths and the weaknesses were. But, um, the final score, that's what counts. And for certain grant activities, um, the scores then get converted into percentiles and the percentiles are basically your relative ranking and it helps normalize different study sections and also helps normalize a single study section over three review meetings.
Marguerite M:So what types of trainees or applicants might see a percentile versus others? Like which applications are not percentiled.
Lyn Jakeman:So the applications that are reviewed at NINDS are not percentiled. The ones that are reviewed in the center for scientific review, they're reviewed with a lot of other training applications of similar--from similar areas of research and those will be percentiled'cause they have a good history of how that study section scores over time. So these, the percentile, they kind of um, normalize the curve so to speak, so that, uh, 80th percentile is the same in one study section or another study section relative to the grants that that study section has seen over time.
Lauren Ullrich:Right. And I think at this point, only our F31s are percentiled and reviewed at CSR. I think all the rest of our training related mechanisms are reviewed in house. So where can applicants find these scores and what should they do once that score is posted?
Jim Gnadt:So, uh, the, the score will first show up in the ERA commons, uh, after the review panel has met. So that's the first place that you can see the raw score. Now, um, it will take a couple of weeks before the summary statement comes out. So in the time between when you can find your score in the ERA commons and when the summary statement comes out, um, you know, you can talk to your advisor about the score, what that might mean. Um, just to review how the scores are reported from the panels: what they do is everybody votes on a one to nine scale. Um, everybody in the room votes on a one to nine scale. And then the average of those is, is reported as the score that you get in your summary statement. So scores of one, two and three are really good scores. It means there are little to very few, uh, concerns of the reviewers about the application. Scores of four, five and six are good scores, but it means the reviewers had concerns about something in, in the review. And once you get the summary statement, it's, it's those concerns you want to look at, you want to discuss with your advisors and your mentors. And at that point, uh, it might be appropriate to contact your program officer by email and say, I'd like to discuss some of the issues in the summary statement. And then if the summary statement has scores of sevens, eights, and nines, that means the reviewers had some really serious concerns about your, your proposal. And those are things you should really talk seriously to your mentors about, uh, about how to deal with, what to do with such serious concerns about your proposal.
Lyn Jakeman:So I love the question about what to do when you get your score because it's just out enough information to do a lot with. So I usually say, you know, when you first get your score, look at it, and then do whatever you do for fun. Go for a walk, go out with your friends, go play soccer, go do whatever you want to do because the score isn't telling you much. It's telling you what range you're in. So if it's one of the, if it's a really high score, then you can go out and do what you do to celebrate. And if it's a low score, you can do whatever you do to commiserate. But, but for the most part you don't have a lot of information with just the score. So just keep doing what you do, take a little break, go back to your experiments and wait for that summary statement.
Amelie Gubitz:Yeah. And it's the scientific review officers who prepare the summary statements and they have a large number of summary statements to prepare right after study section so it takes them awhile. And I think they oftentimes going to score orders so that the um, summary statements with the better scores go out first. So sometimes patience is warranted. And there's really not much that a program director would be able to tell you before the summary statement is released. So it's good to wait until it's out, but then don't hesitate to send us an email, we are really happy to sort of talk with the applicants about the critiques. And um, you know, it's often times things may be fixable and some criticisms may be a little bit more challenging and that's sort of once the summary statement is out, that's the time to strategize and to decide whether the application could be, should be resubmitted or maybe a new application is warranted or a change of strategy. So that's the time when it's, it's good to contact NINDS and at the same time also brainstorm with your advisor how to move the project forward. If it's in a range that is likely not going to be funded. And we'll have a sense of that but final decisions get made much closer to the actual advisory council meeting.
Marguerite M:How does the program arm of NINDS decide how to recommend funding for council?
Lyn Jakeman:How about I start with that one?
Lauren Ullrich:Okay.
Lyn Jakeman:I'd love to start with that one. So what we have to do as leadership--which includes Dr Koroshetz, who's the Institute director, all of the division directors and all of the scientific leads from the Institute--we have to make sure that we're funding the best science and we have to make sure that we're funding the best training. So with your training grant, what we're looking at is the score and the summary statement from grants that were reviewed by peer reviewers. So as far as the score goes, if it's a really high score, that pretty much says that the entire panel felt that this was a strong application from all the perspectives. The project is great, the investigator is great, the environment is great, all those things the entire study section was pretty in agreement on. And so those, we don't spend a lot of time talking about them to tell you the truth. The ones that were scored very, very low, same kind of thing. There are several concerns that weighed down that study section. All of the reviewers kind of said maybe not so much. The ones in the middle and the middle can be anywhere from this score of about a 20 down to about a 40 ish score. That kind of range, we actually look at every single one of those applications and we look at them carefully with the program director, which is why it's a good idea to contact your program officer after you get your summary statement. And what we're looking for is, is this going to be a project that is really going to be good for the trainee, it's going to be good for this science and things are going to work out so that at the end of this project, the trainee is going to be one step farther toward becoming an independent investigator. Um, and so we look at, we look at a variety of things, but it's really those, those basics is, is the trainee really the strongest trainee for this project? Is the project a great project that's going to succeed and is the environment supportive? And in that sense we are looking at the training plan. We're looking to make sure that your training plan isn't something that's just generic that everybody just gets stomped into, but it actually is for you. Um, and so we look at all of those things.
Amelie Gubitz:Yeah. And that's sometimes when we actually then ask the applicant to send us an informal rebuttal. It's a document that won't be presented to our advisory council. It usually also doesn't go to senior leadership, but it just helps us as program director to see how you think you can address the identified weaknesses. And sometimes it may just be an oversight or grantsmanship issue. And sometimes, um, you know, it may require additional preliminary data on new experiments and a more substantial change to the project. It just then helps us evaluate whether we can push this forward and make sure that, um, you know, the research can start sooner or whether it may be a good idea to sort of take a step back and craft a stronger reapplication.
Jim Gnadt:Yeah. I think Amelie has a really important point there. What's most useful for us is to understand what you would do in response to the concerns of the reviewer. So a list of complaints about the reviewers is actually not very useful. Even an iterated list of just rebuttals, while they sometimes may be valid rebuttals, really what's most valuable is what are you going to do about the concerns that the reviewers had, uh, either in terms of giving us some ammunition to promote your project in this council round or to think about what changes you would make if you need to write a revision.
Marguerite M:So sort of to Lyn's point before, it may be best that after you get your summary statement, you write all the nasty things you didn't like about the review on a piece of paper and then you crumble it and throw it in the garbage and then get to thinking about what are some ways that you know, again, as to your point, Jim, of how you're actually going to address this and make your training plan better. I think one thing our audience should really appreciate is that this isn't just about getting funding. It's also about really considering what your training plan is going to be and how you want to proceed in your career because you're going to have to do this over and over again. How am I going to grow? How am I going to become better? And I think just even sometimes in our personal lives it's nice to have a little bit of plan of action and is this really going to get me to the next step?
Lauren Ullrich:Right, right. I think trainees might be surprised by how much of the conversation that we have is centered around what's best for the trainee.
Marguerite M:Yes.
Lauren Ullrich:and like whether the weaknesses that reviewers identified are going to be things that are really going to hurt the trainee: it's not a good mentorship situation; the project is not going to set them up well for their postdoc or their junior faculty position; like, a lot of the conversation in the training realm is centered around that. And so it's not just that we don't want to give you money,[laughter] it's that we don't want to enable a bad situation to continue. And if we have something in writing from you and your advisor about what's going to be changed, then that gives us a lot more leeway to say like,"Hey, you said you were going to do this for this trainee" and it lets us help you. Um, and so, you know, in that vein, if an applicant gets asked to respond to a critique from a review, are there certain do's or don'ts that they should consider when putting together this response?
Lyn Jakeman:Well, I think Amelie touched on one thing earlier, which is, um, these criterion scores really only provide you with information about what kinds of concerns drove those reviewers to come up with an impact score. So I wouldn't spend any time averaging or dividing or multiplying those scores out. Um, because you can go crazy a little bit and you won't come up with the magic number at the other end. Um, and I think Jim touched on, try and look at this as a critique or concerns of the document that you sent them. These are not critiques of you or your advisor or the day of the week, but they're just the document in front of you in comparison to the others that the, that the reviewers have seen, um, have some areas that need to be addressed. And so if you can, if you can stay objective in terms of evaluating the critique of the document as such, I think that it helps calm down some of those emotive responses that we all have.
Amelie Gubitz:Yeah, I think the reviewers are really well-intended and really what they are trying to do is help you strengthen the project. And I think especially for the training grant, the criteria and scores, they are helpful to tell you whether there may be some weaknesses in the research plan or maybe some weaknesses, more sort of in the training career development aspects of the application. So that will help you if you decide to work on a resubmission, you know, to sort of figure out which parts of the proposal need a little bit more work and need to be strengthened or if it doesn't need a resubmission, you still should sort of take that criticism to heart and see, even if I get the grant now, this is something that I really should be working towards and sort of be very aware of. Especially with for instance, career development activities, which are very addressable.
Jim Gnadt:Yeah, it's all about constructive critique.
Lyn Jakeman:I think another"do," cause we did some"don'ts," but I think another"do" is, is do talk to your program officer. Um, sometimes the written word isn't everything and sometimes the program officer can help you interpret in the context of other reviews that, that program officer has read about what this might really mean, um, for your application.
Amelie Gubitz:Yeah, I think phone calls are great and that's really part of our job. In a way we almost expect it that we get a lot of calls after study sections and sometimes we even reach out to the applicants to,"Hey can we set up a call?" But be proactive to send us a line. Then we'll set up a phone call and that's probably the best and fastest way to sort of make sense of the summary statement, but also always have your advisor in the loop so they sort of understand how we see things and how they can help you.
Marguerite M:And this meeting is generally more of a processing session and not really a,"you're going to get answers from the program officer of whether or not you're going to get funded" or"you should and shouldn't do these changes." It really should just be an opportunity to sort of debrief and get more clarity and not expect to get any hard and fast answers.
Lauren Ullrich:Right.
Amelie Gubitz:Well we don't know either, especially for the applications that are sort of in the gray zone. We don't know what our training office, what their pay plan will look like and how many funds are available. So it's a little bit of a process and it certainly takes a couple of weeks.
Lyn Jakeman:Yeah. And there's other things to be thinking about, especially with the training grant and, and that is thinking about in the process of considering whether or not to resubmit a proposal that's in the gray zone and thinking about how that fits into your training plan as Jim was saying, in terms of the timeline as well. There are some cutoffs in time, um, for the F31s, for the F32s, for the K99s, and if you're watching all of those timelines, it may make more sense just to move on to another project; it may make sense to be really seriously considering a resubmission even if you don't know yet if your grant's going to get funded.
Amelie Gubitz:Yeah. But sometimes it does mean that you may actually have to switch the funding opportunity announcements, especially for the F32s, you have to apply very early during your postdoctoral training or even beforehand. So by the time you've gone through review, you sort of may have moved out of the eligibility window. But it doesn't mean that there are no routes for funding for you because we have other grant activities that'll allow you then to apply to. So you may switch to a K01 or other mechanisms. So there's always a solution, but that's when it's good to talk to us and also colleagues in our training office.
Lauren Ullrich:Right. We can always help you out. Um, and so if an applicant gets asked to respond to the critiques, does that mean that they will get funded?
Jim Gnadt:No.[laughter] The decision to get funded is made at the time that advice is given by council to the Institute directors to make a final decision on who gets paid and who doesn't. But talking to your program officer and talking to your mentor and advisors is useful to give us ammunition about how we will deliberate about your project and how you're thinking and responding to the reviewers' critiques. Um, but the, the final decisions, uh, are made by council and the Institute directors and the director of the training program. And that will come later at council time.
Lauren Ullrich:Yeah, it all takes time.
Marguerite M:Right.
Lyn Jakeman:This is true not just for fellowships, but throughout your career. Um, anytime that a program director reaches out and asks you for a response to the review, all it means is: we'd like a little bit more information. That's really all it is. And we may be specific about what we're looking for or we may be very vague about what we're looking for, but, but it just means we need more information.
Marguerite M:Yup. So speaking of Council, we will be talking a little more in depth about the council review process. And in a previous episode we talked about the scientific peer review process. But can you all explain to our listeners the difference between the scientific peer review and the advisory council review?
Amelie Gubitz:I mean, the peer reviewers will look at the actual full application and the experts in the field, the will probably have quite a bit of expertise in reviewing grants over the year and they meet at study section. The advisory council it's sort of a secondary review panel, it's higher level. And one thing to note is that not all grant activities actually go to our advisory council meeting. So, for instance, fellowship applications, the F31s, the funding decisions get made in house before the actual advisory council meeting. But um, the career development grant activities, they typically go to, um, council meeting. It doesn't mean that every application gets discussed, but certainly they go to council considerations. But the council members will only have access to the summary statement. So it's a second tier review and sort of higher level, more strategic. They are not going to sort of look at individual applications. They won't have access to them and they make decisions based on the input from the initial peer reviewers.
Jim Gnadt:Yeah, I look at it as the peer review are the experts in your scientific domain and in training and the specifics of your training program. I know what I'm looking for when I talk to, uh, applicants afterwards is, um, how are you responding to the summary statement and the advice and the constructive critique you got from the peer review? And, and you know, it's, it's very nice to see people who have very productive responses to that. But I don't make decisions about the expertise. That's what the peer review is about.
Lauren Ullrich:Is there anything else that you wish applicants understood about this process or any common misconceptions that you want to clear up?
Amelie Gubitz:I mean, I think one thing to note is that it is a very competitive process and I think it's sort of my key advice is always don't get discouraged. And I think perseverance often helps and I think it's important as an applicant to embrace going through the peer review process and always view it as a learning opportunity. Even if the outcome isn't exactly what you were looking for, you're still going to learn a lot from going through the process, getting the critique from the peer reviewers and then strategizing how to move forward. I think both for um, a really good summary statement or for some that sort of picked on some weaknesses, you always learn from the process and never take it personal.
Lauren Ullrich:All right, well thank you all for sharing your wisdom today. Um, so can we ask each of you for one last piece of parting advice for our future applicants?
Jim Gnadt:Be persistent. It's, this is a very competitive field. You've chosen one of the most competitive and difficult fields there is to, to join. Um, but those who are persistent, those who are good, you will succeed. Just keep at it.
Lyn Jakeman:If I were to give some advice to trainees and to young investigators, persistence definitely plays a role. I would also say that you're, you're running a marathon and not a sprint. Um, and what I mean by that is that if you look at any grant deadline and put it on your calendar and start worrying about it as if it was an undergraduate exam and kind of cram the night before that your results probably aren't going to be what you hoped for. But one of the best things to do is to consider that you've taken on a career in writing, really. And the science is almost secondary to the writing. So develop a style, a pattern where you write every day, a little bit every day. And then when you're doing a grant application, you're building on the things that you do every day. And it's much easier than if it's a big mountain that you have to overcome and the stress of procrastination.
Amelie Gubitz:Yeah, I think Lyn, that's a really good point. And certainly putting the applications together, it's not an easy task and it takes time and it has to mature and you have to sort of get a lot of support from your advisors and other colleagues. But even in before you submit the grant, don't hesitate to reach out to program directors at NINDS. We are, you know, happy to sort of brainstorm about a project idea, to help you identify the right um, funding opportunity announcement. And we oftentimes take a peek at draft specific aims page and we can give you a little bit feedback. And sometimes there are also overlapping areas of research, so it may not always be that, you know, NINDS is the only Institute at NIH that may support that type of research. So I think it's a good idea to actually sort of find out who the program director at NINDS would be for your program area. And you can contact us and we can give you a little bit of advice and we are certainly happy to look at a draft specific aims page.
Lyn Jakeman:and don't be afraid of your program directors. We are so lucky in this country that we have a system at the NIH where program directors can reach out and be your advocate and your friend. Um, and we've seen a of applications. So we come with a great deal of experience. Um, and we're really happy to help you navigate that whole confusing process. We're here to help.
Lauren Ullrich:That's our tagline.
Marguerite M:Yes. We are here to help.
Lauren Ullrich:And Marguerite, do you have any advice?
Marguerite M:Yeah, I think that getting this feedback is a great opportunity to have an attitude check and have, um, an outlook check and really think about what it is that you want from this whole process that isn't just about dollar signs. Um, you know, we certainly would like to see your work get funded. Um, but we also want you to be good scientists and, um, have a plan. And so sometimes your plan wasn't that well put together or maybe your best, um, just wasn't quite good enough. Um, but that doesn't mean that there aren't opportunities for growth. And so hopefully you will take all of this advice and be able to, you know, pull yourself up out of despair if that's the case for you or if you're already on cloud nine, stay up there, um, and hopefully help other people as they tackle this. But it's, it doesn't have to be the end of the world. And, um, it's not necessarily about building thick skin, it's just about thinking about how you're going to um, try again. What about you Lauren?
Lauren Ullrich:I mean, I think to follow up on that, the summary statement is just an additional perspective from, you know, three to five additional people that have read your application. And like I think we touched on earlier and have spoken about in earlier episodes, this is not an indictment of you. This is merely an evaluation of the application that you have put together. And really think critically about the reviews, discuss it with your mentor, discuss it with your peers. Sometimes the reviewers get it wrong and that's something to have a conversation with your program director about, but they didn't get everything wrong, right? So before you have the conversation with your program director, you want to have that debrief with other people at your institution to say like, okay, these critiques are valid, this one for real--I don't agree with. So let's make a plan for how we're going to address this in the future application. And then that's something you can discuss with the program director. But just think about it as, as getting some outside perspective, which is always valuable. So that's all we have time for today on Building Up the Nerve. So thank you again to our guests this week for sharing their expertise and thank you to program director Bob Riddle for our theme song and music. We'll see you next time when we talk about the Council process. You can find past episodes of this podcast and many more grant application resources on the web at ninds.nih.gov.
Marguerite M:And be sure to follow us on Twitter@NINDSdiversity and@NINDSfunding. Email us with any questions nindsnervepod@nih.gov. Make sure you subscribe to the podcast on Apple podcasts or your favorite podcast app so you don't miss an episode. We'll see you next time.