Distributed Peer Review (DPR): Study Shows Potential for Faster and More Democratic Review

Traditionally, panels of external experts evaluate funding applications in time-consuming and resource-intensive procedures. DPR takes a different approach: applicants in a call for proposals evaluate each other. This approach was originally developed to allocate research time on astronomical telescopes more fairly and efficiently, but is now also being tested in other funding areas.

The Volkswagen Foundation is testing the procedure in its funding initiative 'Open up – New Research Spaces for the Humanities and Cultural Studies' – and is thus using the procedure in a social science setting for the first time. The Research on Research Institute (RoRI) is conducting the accompanying scientific research for the experiment. The recently published study compares the DPR method with the ‘conventional’ panel review used by the foundation in parallel in the same application round. The results are exciting.

Key findings of the DPR panel review comparison

• Less review effort: On average, the time spent by a single person on reviews fell by around 60 per cent. If participation in the panel meeting is also taken into account, the combined individual time was reduced by as much as 87 per cent.
• More attention per application: Under DPR, at least three times as much time was spent reviewing the content of each application as is typically the case in the conventional procedure.
• Comparable quality: The new reviewers from among the applicants applied the evaluation criteria with the same consistency as experienced panel members.
• High level of agreement: Eight out of ten applications recommended for funding by DPR reached the final panel stage in the conventional track.
• Positive response: 83 per cent of funded applicants and 60 per cent of non-funded applicants expressed a positive opinion about participating in future calls for proposals with DPR. Submissions increased by 18 per cent between the first and second DPR rounds; a total of around 140 applications were received – an increase compared to previous years.

Assessment of results

The study goes beyond mere feasibility tests: by comparing both procedures in parallel on the same applications, it enables a direct, reliable comparison. RoRI emphasises that DPR can shorten the time required for assessment and funding decisions while providing helpful, constructive feedback for all parties involved – regardless of the outcome of the procedure. Overall, the experiment underscores the potential of DPR as a further development of the established peer review principle in a more ‘democratic’ rather than ‘gatekeeping’ mode.

Looking further afield

Other funding organisations are also testing or scaling up DPR, including UK Research and Innovation, the US National Science Foundation and the National Institute of Food and Agriculture, as well as the Netherlands Research Council. Much of the experience gained has been summarised in the new guide ‘Applicants as Reviewers: A Guide to Distributed Peer Review’, which is available on the RoRI website.