grantscience.com

Codings, summaries and estimates on this site are AI-assisted and provisional, pending verification against primary sources. About the data

grantscience.com · evidence synthesis

The shape of the evidence on grant peer review

How reliable is the expert review that decides who gets funded? This site presents a systematic synthesis of the research literature on peer-review reliability, grant review first, with journal review and neighbouring judgment contexts as the comparison: what has been studied, where the gaps sit, and what the measured agreement between reviewers actually looks like.

9,783 records identified → 9,242 screened (dual AI screen, kappa 0.866; recovered arm 0.890) → 1,627 included → 1,123 in the corpus after de-duplication and retrieval

Scoping funnel: PRISMA-ScR scoping search, kappe-scoping screening records. Provisional.

1,123
papers in the scoping corpus
8,422
in-corpus citation links
16
inductive topics (BERTopic)
1,960
reliability coefficients from 175 studies
17,533
outside-corpus works linked

01The field at a glance

What the corpus is made of. The two facets, review context and research construct, re-express the screening themes around the decomposition of judgment error into bias and noise; study design is classified from titles and abstracts by the coding panel.

85431937: 11951: 11960: 21966: 21968: 11969: 11970: 21971: 31972: 21973: 21974: 31975: 31976: 31977: 71978: 61979: 71980: 31981: 41982: 151983: 41984: 21985: 101986: 61987: 41988: 51989: 41990: 131991: 71992: 21993: 41994: 71996: 31997: 31998: 121999: 102000: 42001: 52002: 92003: 52004: 32005: 112006: 172007: 122008: 142009: 182010: 172011: 212012: 222013: 92014: 152015: 632016: 492017: 682018: 532019: 602020: 662021: 632022: 692023: 712024: 822025: 852026: 47194019501960197019801990200020102020
Publications per year, 1937 to 2026. The field is old but small until roughly 2015, then accelerates sharply.
Grant422
Journal515
General124
Analogue62
Review context. Papers are classified to one primary context; the site defaults to the grant subset throughout.
observational-quantitative505
conceptual-commentary278
simulation-methodological118
qualitative75
experimental-rct64
systematic-review-meta-analysis37
book-monograph34
guideline-policy12
Study design, panel-classified from titles and abstracts. Observational designs dominate; controlled experiments are the minority.

02Does the field practise what it studies?

Every paper is coded for the transparency of its own reporting: preregistration, data and code availability, statement by statement. Preregistration is counted among papers that could have been preregistered (empirical studies), open data among papers reporting their own data. The literature on peer review adopts these practices late, but the recent shift is clear.

38.8%19.4%before 2013: 2 of 240 papers (0.8%) preregistered0.8%before 2013: 2 of 228 papers (0.9%) open data0.9%before 20132013 to 2018: 3 of 188 papers (1.6%) preregistered1.6%2013 to 2018: 40 of 186 papers (21.5%) open data21.5%2013 to 20182019 onwards: 46 of 367 papers (12.5%) preregistered12.5%2019 onwards: 145 of 374 papers (38.8%) open data38.8%2019 onwards

PreregisteredOpen data

Preregistration rises from 0.8% of applicable papers before 2013 to 12.5% 2019 onwards; openly deposited data from 0.9% to 38.8% of papers with own data. Statement-level coding from full text; provisional.
51
preregistered or registered-report papers
146
data deposits linked
97
code repositories linked

03Explore the evidence

Four views over one dataset. Every count, label and colour derives from the generated data files, and every view opens on the grant subset with the other contexts one click away.

Evidence gap map

Process stage crossed with construct: bubbles sized by number of studies, coloured by the share that are controlled experiments. 422 papers in the Grant subset, mapped over 7 stages of the review process.

Reference network

All 1,123 papers as an interactive map: 5 layouts, 8,422 in-corpus citation links, recolourable by any facet, with per-paper summaries and references out into the wider literature.

Reliability, coded

Every reliability coefficient located in the corpus, coded to its exact form so like compares with like. Median single-reviewer ICC 0.28 against a panel-average 0.55.

Browse the corpus

Search and filter all 1,123 papers by context, construct, study design and process stage, each with an AI-generated summary and a link to the source.

04Read this with the caveats

Not yet source-verified. Every number on this site is provisional until checked against the primary source's full text.

Machine-classified. Context, construct and study design are the majority vote of a five-model coding panel reading titles and abstracts, with ties adjudicated by hand; process stages come from a separate AI pass. The full human validation pass is still pending, so labels and counts may change.

A modest corpus for topic modelling. Inductive topics and time-sliced trends rest on a corpus that is small by topic-modelling standards; early time slices are thin.

Original summaries. Paper summaries are AI-generated original text, never the publisher's abstract, and are flagged for verification before citation.