Illustration Lotterie Kugelmaschine

Everyone is equal in the lottery drum

Author: Mareike Knoke

Reaching funding decisions by drawing lots? Although that might sound a little dubious, it is not. This is proven by a scientifically monitored experiment carried out by the Volkswagen Foundation, which concludes that a partially randomized selection procedure following the scientific reviews brings more diversity into the pool of grantees. The researchers asked by the Foundation to monitor the procedure arrive at some interesting insights in their overall assessment.

Through its "Experiment!" funding initiative, the Volkswagen Foundation provides important impetus for testing unusual and innovative research ideas. Between 2013 and 2021, the Foundation funded a total of 183 "risk-laden" projects with uncertain outcomes. More than 5,000 researchers from the natural, technical and life sciences – from established professors to postdocs at the very beginning of their scientific careers – applied for funding of up to 120,000 euros on the sole basis of anonymously reviewed draft proposals. Up to 40 projects were selected for funding in each of the eight funding rounds. 

The Foundation used this unprecedented initiative to test a selection procedure that is still something of a rarity on the German funding landscape, i.e. a certain number of grant approvals were determined by random selection. However, this was in all cases preceded by scientific review, which is why the Foundation refers to the procedure as "partial randomization". And it took care to have this partially randomized selection scientifically monitored: Dagmar Simon and Martina Röbbecke from Evaconsult interviewed "Experiment!" grantees and examined the approvals both before and after the partial randomization procedure was introduced.  

The researchers' results are now available in a concluding report, which evidences some significant positive effects. The partial randomization procedure proves to be particularly well suited to funding programs aimed at supporting research proposals designed to break new scientific ground. It can increase diversity among the grantees and reduce the risk of possible bias as present in conventional review procedures.

How exactly does partially randomized selection work? 

As of 2017, the Volkswagen Foundation introduced the procedure exclusively in the "Experiment!" funding initiative. In a first step, the Foundation's funding team selected the applications from all those received that met the program criteria, thereby forming a shortlist of 80 to 120 applications in each round. Two members of an external, multidisciplinary jury with the requisite expertise for the respective field then assessed the eligibility for funding from the scientific point of view. In the next step, the jury met to determine the best 15 to 20 research proposals. All the eligible applications were then placed in a lottery drum (including those already selected), from which the same number of applications were drawn again under the supervision of the Foundation's legal advisor.  

Why did you decide to trial this procedure?

On the one hand, the Foundation has always had a focus on increasing diversity among its grantees. In the context of "Experiment!", this means that a larger proportion of women and early career researchers should be given the chance to put forward their ideas. 

Secondly, the final decision arrived at by implementing partially randomized selection was intended to reduce the risk of bias – a distortion of the vote due to bias, prejudice or prejudice that can never be completely ruled out. "For example, the ultimate selection by drawing lots is not influenced by any subliminal dynamics," explains Ulrike Bischler, who supervised the funding initiative at the Foundation together with her colleague Pavel Dutow. "Such dynamics can be seen, for example, when the less experienced members of the jury might prefer to follow the vote of the more experienced reviewers in the final discussion about approval or rejection rather than continue to argue for their personal choice."

Grafik zum randomisierten Auswahlverfahren

Moreover, reviewers in the conventional peer review process rarely decide "out of the box", but tend to be consensus-oriented or to view funding applications through the lens of their own discipline. "This is another reason why it can make sense to partially randomize the final selection," says Bischler.

The positive effects of partial randomization on diversity were clearly demonstrated by the example of female scientists: prior to 2017, every sixth approved proposal was submitted by a woman; with the partially randomized procedure, it was every fourth. This was in line with the overall participation of women in the calls for proposals. Another positive observation: the participation of women increased in the last rounds of the "Experiment!" initiative. The increase in the proportion of women can be explained by a demographic effect – and is therefore not to be seen as probabilistic. In other words: the percentage of women is lower among professors, while it is significantly higher among early career researchers due to demographic factors. Since early career researchers are more likely to benefit from the partially randomized procedure, the proportion of women among the grantees increases.

What do the grantees themselves say about this innovative procedure?

In their concluding report, Röbbecke and Simon say the majority of respondents generally rate partially randomized selection as positive. For example, researchers value the fact that partially randomized procedures promote equal opportunities for individuals (92%), encourage applications involving risk-laden research (84%), and offer better opportunities for high-risk research (80%). 

For most of the respondents, however, in terms of scientific quality assurance it is crucial that review by a jury has taken place before their application reaches the randomized stage.

Eine Lostrommel, Loskugeln und ein Zettel mit einer langen Liste sind zu sehen.

With the help of a real lottery drum, the same number of projects were drawn in addition to those selected by the experts.

This is where the long tradition of the peer review process comes into play. "Peer review is indispensable, for example, for selecting manuscripts for publication in journals or anthologies, for appointment procedures, for funding decisions on projects with high funding amounts, and for awarding scholarships or academic prizes," says Martina Röbbecke. However, it does have its limits; for example, when it comes to projects located between different disciplines.

According to the survey, this is also the view of many grantees – although they still attach great importance to scientific peer review. The study shows "that a large number of respondents agree with various statements that indirectly represent a criticism of peer review procedures," write the authors. According to 88 percent of respondents, randomized selection clearly avoids conflicts of interest and unconscious bias. "Moreover, they are also of the opinion that partially randomized procedures offer opportunities for subjects that are poorly represented on the jury and for more thematic and methodological diversity." 

What's more, the idea of possibly "only" making it to the end thanks to a random selection does not seem to have a negative connotation for the majority of respondents – or for their institutions.  

One researcher from the last funding round, Prof. Dr. Gerhard Fischerauer from the University of Bayreuth, said at a conference organized by the Volkswagen Foundation in December 2022 that it was of no major significance at his university whether the third-party funding was a grant that had ultimately been decided by random selection: "I have never heard the argument 'colleague A gets money from BMW, colleague B only from Dacia', but 'colleague C only had the luck of the draw at the Volkswagen Foundation'."

However, as Dagmar Simon points out, the fact that no distinction is made is primarily due to the high reputation of the Volkswagen Foundation. "If a smaller or less well-known funding institution had launched such a call for proposals and implemented a partially randomized procedure, the view taken would probably have been different," she emphasizes.

A model worth emulating

In light of the positive feedback and experiences from "Experiment!", Röbbecke and Simon believe it would be desirable for similar application procedures with partially randomized selection to be tested by other renowned funding bodies.  

This is a view shared by the German Council of Science and Humanities, which advocated random selection in a 2017 position paper on evaluations in the science system, stating that in the case of heavily oversubscribed funding offers it may become difficult to justify a decision. Furthermore, the Council advocates testing innovative selection procedures and systematically evaluating their introduction, as the Volkswagen Foundation has done.

Dr. Ulrike Bischler ist zu sehen

Dr Ulrike Bischler is responsible for the "Experiment!" funding initiative.

The German Agency for Transfer and Innovation (DATI), recently set up by the Federal Ministry of Education and Research, has introduced a partially randomized procedure for the "Innovation Sprints" funding line. Here too, the aim is to get innovative ideas off the ground more quickly.

Otherwise, though, the introduction of partially randomized procedures in Germany remains a rare exception. However, a look across the border to Austria and Switzerland shows that major state funding institutions in other countries are also venturing in this direction: the Swiss National Science Foundation has already introduced a partially randomized  procedure for some funding lines, as has the Austrian Science Fund (FWF).

Partially Randomized Procedure - Lottery and Peer Review

Since 2017, the Volkswagen Foundation is testing a new selection procedure for project applications: In the funding initiative "Experiment!", part of the funded projects are selected by an independent jury. Additionally, further projects are drawn from those applications that are suitable for the program and eligible for funding. Background and reactions to a new and unusual selection procedure.

Learn more

Nevertheless, many in the scientific community who have been researching peer review for some time remain skeptical about adopting such procedures. At the Volkswagen Foundation’s conference on the topic in December 2022, they voiced some sharp criticism: in their view, the most important criterion for granting third-party funding must always be scientific quality; there is no alternative to peer review and decision-making, otherwise there is the risk of becoming arbitrary.

Ulrike Bischler from the Volkswagen Foundation is quick to refute this criticism. She emphasizes that the concluding reports show there is no difference in the number of patent applications and scientific articles between direct selection by jury decision and partially randomized selection: "In both cases, the proportion of those producing publications was 60 percent. This shows that we didn't get more diversity and fairness at the price of a reduction in quality."

More on the topic