Verifying a Paper's Credibility
Understanding a paper is only half the job — the other half is deciding whether to trust it
By the end of this topic you'll be able to
Understanding a paper is only half the job — the other half is judging whether to trust it, and that starts with a crucial distinction: arXiv is a preprint server with no peer review at all, just an automated check that a submission is on-topic and not spam. A paper being on arXiv means only that its authors chose to post it, not that anyone independent has checked it.
So the first thing worth checking is where else, if anywhere, the work has appeared: has it been accepted at a peer-reviewed venue — a journal like Physical Review Letters or Nature, or a vetted conference like QIP? Most arXiv listing pages show this directly as a 'journal reference,' and its absence, especially for an older paper, is itself informative.
Next, check who wrote it and from where: an author's institutional affiliation, their track record of prior publications in the specific subfield, and whether the result was produced by a single lab or independently corroborated by others all bear directly on how much weight to give a claim, particularly an extraordinary one.
Check who else cites the paper, and how: tools like Google Scholar, Semantic Scholar, or (for physics specifically) INSPIRE-HEP show citation counts and let you read the citing papers — do later papers build on the result, or do they report failing to reproduce it? A high citation count paired with citations that merely mention a paper in passing is a weaker signal than a handful of citations that build directly on its central claim.
For anything claiming a computational result — a new algorithm, a hardware demonstration, a benchmark — check whether code and data are actually available, typically linked from the paper or its arXiv listing (a GitHub repository, a Zenodo archive), and whether the comparison against prior approaches (classical or quantum) is apples-to-apples: many quantum advantage claims have been walked back after independent groups found a classical algorithm the original paper hadn't compared against.
Finally, watch for a few concrete red flags: claims framed as overturning well-established results (like the postulates or no-cloning theorem covered earlier in this curriculum) without correspondingly strong evidence; papers posted only to arXiv for years with no journal follow-up and no citations building on them; and hardware claims that turn out, on close reading of the methods section, to be classical simulations of the proposed hardware rather than an experiment run on it.
None of this is a reason to distrust arXiv as a whole — it is how essentially all of modern physics circulates results quickly, this curriculum's own citations included. It's a reason to treat 'posted on arXiv' and 'independently verified' as two entirely different claims, and to keep asking which one a given paper has actually earned.
Try It Yourself
You find an arXiv preprint, posted 6 years ago, claiming a classical algorithm that efficiently simulates any quantum circuit (which would undercut the basis for quantum advantage). It has no journal reference listed, and a citation search shows 2 citations total, neither of which builds on or confirms the result. Walk through the credibility checklist and decide how much weight to give this claim.
- 1Venue: no journal reference after 6 years is a meaningful negative signal — a genuinely correct, field-shaking result would typically have been vetted and published somewhere by now.
- 2Citations: 2 citations in 6 years, neither building on the result, is very low for a claim this significant — if it were true and reproducible, other researchers would very likely be building directly on it or at least engaging with it substantively.
- 3Extraordinary claim: this result would overturn a load-bearing assumption behind the entire field (that no known classical algorithm efficiently simulates general quantum circuits) — exactly the kind of claim that needs unusually strong, independently corroborated evidence, not just a preprint.
- 4Weighing all three together: an old, uncited, unpublished preprint making a claim that would upend a well-established distinction is a textbook case for the checklist's red flags, not a reason to update your beliefs about quantum advantage.
Treat the claim with strong skepticism — the missing journal reference, the near-absence of citations after six years, and the extraordinary nature of the claim all point the same way. This doesn't mean the paper is definitely wrong, but it does mean it hasn't earned the same trust as a peer-reviewed, independently-cited result, and shouldn't be treated as settled science until it has.
Reference
| Preprint vs. published | arXiv only checks a submission is on-topic, not that its claims are correct | |
| Credibility checklist | What to check before trusting a paper's result |
Quick Check
A paper posted only to arXiv, with no journal publication and no other papers citing it years later, making a claim that contradicts a well-established result. What should that combination make you do?
Besides checking the venue, what's another concrete way to judge a paper claiming a new computational result?