What we can learn from NASA's Dual-Anonymous Peer Review

Research Development recently launched a faculty cohort approach to space-related funding (check out the panel discussion here). Working with them, we committed to notification regarding NASA's Research Opportunities in Space and Earth Sciences (aka ROSES).

ROSES, for those not in the know, is a 676 page document that describes all you need to know to apply to a series of opportunities in the Science Mission Directorate. (It is recommended that you read it.) Similar to other grant production/submission guides, we do recommend that you read it as it outlines everything you could ever want/need to know about the directorate prior to writing your proposal.

NASA releases appendices under ROSES. Each appendix has a specific topic of interest and describes duration of funding, funding amount, and specific requirements for the submission. The list of topics can be found by appendix and by due date.

Changing gears a bit, NASA kicked off a few diversity enhancement activities which included a presentation and a series of workshops that sought to encourage diversity in new applicants. I blogged on this topic here. Concurrently, NASA was applying dual-anonymous peer review to proposals for telescope time.

When reviewing ROSES changes for 2020, it was noted that nine programs in ROSES will be participating in dual-anonymous peer review. This is important in a myriad of ways, one of which significantly impacts proposal writers: proposer text must be written so that it is anonymous.

Proposers to these programs must provide an anonymized version of the proposal for peer review, and a separate non-anonymized document that contains elements of the proposal that would reveal the identities and affiliations of participating researchers, such as expertise, facilities and resources. (ROSES-20 SoS-17)

All of this was done to address the issue of unconscious bias.

SMD is strongly committed to ensuring that the review of proposals is performed in an equitable and fair manner that reduces the impacts of any unconscious biases. (ROSES-29 SoS-42)

Finally, they will be instituting a review for high-risk/high-impact for proposals not selected for funding.

Starting in ROSES-2020 another novel variant of the peer review process will be focused on high-(intellectual)-risk high-impact proposals. SMD will collect information from proposers and reviewers to assess (intellectual) risk and impact of ROSES proposals and the Associate Administrator will assemble a second panel of senior researchers to examine a few high-risk high-impact proposals that were not selected for funding through the normal review process, but were nominated by selection officials for an independent evaluation of intellectual risk and impact.

Blind peer review is a hot topic among funding agencies, with mixed results and lack of systematic change.

Which leads me to the question: What can we learn from NASA's dual-anonymous peer review?

The answer is that we won't know until the awards are noted this coming year. But, I will be watching the outcome. I hope you do too. I also hope that other funding agencies are watching.