How Dual Review Works
The rules behind a dual review: roles, blind review, AI evidence, how two decisions become one, Cohen's Kappa, and what happens when a decision, the criteria, or a reviewer changes mid-review.
This page is the reference for dual review. For the step-by-step path from creating the review to the report, see Getting Started with Dual Review.
What dual review is
A standard Paperguide systematic review is AI Review: Paperguide AI screens every paper and one assigned verifier confirms or overrides each decision. That is fast, and for many reviews it is enough.
Reviews that must survive peer review, a Cochrane editorial process, or a regulatory audit usually need something stricter. The standard is two people screening every paper independently, with a documented process for settling disagreements. Dual review is that method, built into Paperguide.
In a dual review:
- Two reviewers screen every paper independently. Each has their own full queue. Papers are not split between them.
- The final decision comes from agreement. When both reviewers reach the same decision, it is final. When they differ, the paper goes to a conflict resolver.
- A conflict resolver settles disagreements. Their decision is final.
- Paperguide AI supports the reviewers without deciding for them. With AI evidence on, it shows the reasoning and cited text for each criterion, so reviewers find the relevant passage quickly. It never records an include or exclude decision.
- Everything is recorded. Every vote, every change of vote, and every resolver decision is kept on the paper's history.
Dual review applies to the two screening stages, title & abstract and full text. Data extraction and the report work the same way in both methods: AI extracts, one verifier checks, and the report is generated from confirmed data.
Choosing the review method
You choose the method once, when you create the review, under Review method. It cannot be changed afterwards. If you need to switch, create a new review.
The method applies to the whole review. A dual review is dual at title & abstract screening and at full text screening. The screening stage toggles work as usual, so you can still run a dual review with only one of the two screening stages. Dual Review needs an Evidence plan or higher.
Roles in a dual review
Each screening stage has three roles, assigned separately in the review's Settings & Roles tab. The people can differ between stages.
| Role | What they do |
|---|---|
| Reviewer 1 | Screens every paper in the stage, independently |
| Reviewer 2 | Screens every paper in the stage, independently |
| Conflict Resolver | Decides the papers the two reviewers disagreed on. Their decision is final. |
A few rules apply:
- Every role is assigned to a named person. There is no "anyone on the team" option in a dual review.
- Reviewer 1 and Reviewer 2 must be different people.
- The conflict resolver can also be one of the reviewers. On a small team this is common: two people screen, and one of them settles conflicts.
- Both reviewers must be assigned before screening on that stage can start. The resolver can be assigned later. Leaving the resolver empty only holds up conflicts, not screening.
- Data extraction keeps its single Extraction Verifier, as in an AI Review.
Only members with Full Access or Review access can hold a role. See Members, Access & Roles for how access works.
Review settings
Two settings shape how a dual review runs. Both are in the review's Settings & Roles tab. Only Full Access members can change them.
Blind review
On by default, for the whole review.
Blind review keeps one reviewer from being influenced by the other. It hides exactly one thing: the other reviewer's decision on a paper. The paper, the criteria, the AI evidence, and any comments stay visible to both.
Blinding is lifted paper by paper. As soon as both reviewers have decided a paper, each can see how the other voted on it. Papers only one reviewer has decided stay hidden until the second decision lands.
On the review overview, each reviewer can always see the other's progress: how many papers they have decided and how many are left. The other reviewer's split between Include, Maybe, and Exclude stays hidden until you have decided every paper they have, so it cannot hint at a decision you have not yet made.
The conflict resolver is never blinded. If the resolver is also a reviewer, they are blinded while acting as a reviewer and see everything once a paper reaches them as a conflict.
AI evidence
On by default, set per screening stage.
With AI evidence on, Paperguide AI evaluates each paper against your criteria. For every criterion it shows the reviewers its reasoning and the text it relied on, in the same panel they use to decide. The point is speed. A reviewer can go straight to the sentence that matters instead of reading the whole abstract or paper to find it.
AI evidence is reasoning only. In a dual review, Paperguide AI never makes an include or exclude decision. The decisions are the reviewers'.
With AI evidence off, no AI runs for that stage and no AI credits are spent on it. Reviewers see the paper and the criteria, nothing more.
To generate evidence, a Full Access member selects Start AI Evidence Run on the stage card. Reviewers can start voting while the run is still in progress. If you add papers later, Continue AI Evidence Run covers the papers that have no evidence yet. After a criteria change, Restart AI Evidence Run regenerates it for the whole stage.
How reviewers decide
Each reviewer's decision on a paper is called a vote. Both reviewers work from the To Review tab of the screening workspace. They read each paper against the criteria, with the AI evidence beside it, and choose under Make Your Decision:
| Stage | Decision options |
|---|---|
| Title & Abstract | Include, Maybe, or Exclude |
| Full Text | Include or Exclude |
Excluding a paper requires choosing one criterion as the reason. You may feel a paper fails several criteria, but only the one you select is recorded and reported. This is the reason that appears in your PRISMA exclusion breakdown, so choose the one that matters most for your review. Include and Maybe need no reason.
The stage pages walk through the workspace, the tabs each role sees, and the decision step by step: Title & Abstract Screening and Full Text Screening.
Comments
Any member with Review or Full Access can leave a comment on a paper. Comments are visible to everyone on the review, including under blind review, so use them for questions about the paper rather than for your decision. Your decision stays hidden from the other reviewer until they have decided too.
Agreement and conflicts
A paper's final decision comes from comparing the two reviewers' decisions.
| Reviewer 1 | Reviewer 2 | Outcome |
|---|---|---|
| Include | Include | Final: Include. The resolver is not involved. |
| Maybe | Maybe | Final: Maybe. The paper carries forward to full text. |
| Exclude, criterion A | Exclude, criterion A | Final: Exclude, reason A |
| Exclude, criterion A | Exclude, criterion B | Conflict: same decision, different reason |
| Include | Maybe | Conflict |
| Maybe | Exclude | Conflict |
| Include | Exclude | Conflict |
Two exclusions with different reasons count as a conflict on purpose. The exclusion reason is part of what your review reports, so a disagreement about the reason has to be settled by a person, not merged silently.
Conflicts appear in the Conflicts tab of the screening workspace and are counted on the stage card in the overview.
Cohen's Kappa
Once both reviewers have decided some papers, the stage card shows Cohen's Kappa, the standard measure of agreement between two raters, with the number of papers it is calculated over. It is labelled from Poor to Almost Perfect Agreement.
Use it as a health check on your screening. A low score early in a stage usually means the two reviewers are reading the criteria differently. That is cheaper to talk through after twenty papers than after two hundred. Many journals ask for the kappa value in the methods section, and this is the number to report.
Resolving conflicts
The conflict resolver works only on the papers in the Conflicts tab. For each one they see everything: both reviewers' decisions, their exclusion reasons, the AI evidence, and the comments. Nothing is hidden from the resolver.
The resolver can:
- side with Reviewer 1,
- side with Reviewer 2, or
- make a different decision of their own.
At title & abstract, Maybe is a valid resolution. The paper moves on to full text screening, where it gets a firm decision. At full text, the resolver chooses Include or Exclude. A resolving Exclude needs a criterion as its reason, like any other exclusion.
Whatever the resolver chooses becomes the paper's final decision. The paper's history records who resolved it and how.
How papers move between stages
This is the biggest practical difference from an AI Review.
In an AI Review, a stage has to be fully confirmed before the next one opens. In a dual review, papers move individually, the moment their decision is final:
- A paper with a final Include or Maybe at title & abstract is available for full text screening straight away. Full text reviewers can work while title & abstract screening is still in progress.
- Data extraction opens once full text screening is complete, since the extraction table needs the final set of included papers. Papers still waiting for a PDF do not hold it up.
Screening never waits for a whole stage to finish. On a large review, this means the full text reviewers are not idle for weeks while abstract screening completes.
When a stage is complete
A screening stage is complete when both of these hold:
- Every paper in the stage has a decision from Reviewer 1 and from Reviewer 2.
- Every conflict has been resolved.
Stage status always reflects the current decisions, so it can move backwards. Adding papers, a changed vote, or a reopened conflict takes a completed stage back to In progress. Settling the last open item completes it again.
Papers without a PDF
At full text, a paper without a PDF cannot be voted on. It is held as not retrieved and does not block the stage from completing. Add the PDF, by uploading it or fetching it with the Paperguide browser extension, and the paper enters both reviewers' full text queues.
Adding papers later
You can add papers at any point. New papers enter both reviewers' queues at title & abstract, and the stage returns to In progress until they are decided. If AI Evidence is on, run Continue AI Evidence Run to generate evidence for them.
Changing a decision later
Reviewers and resolvers can change their decisions at any time, including after a paper has moved on to later stages. The rules are designed so that one person acting alone can never undo a final decision or destroy work downstream.
Editing the protocol during screening
Decisions were made against the criteria as they stood at the time. If you change the screening criteria while a dual stage is underway, some of those decisions no longer stand.
When you save a criteria change, Paperguide asks how to handle the existing decisions:
- Apply changes and reset the screening decisions. Every decision in the stage is cleared and both reviewers screen everything again.
- Apply changes and reset only affected decisions. Only the decisions the change actually touches are cleared.
Which decisions count as affected:
| Change to the criteria | Decisions reset | Decisions kept |
|---|---|---|
| Criterion added | All Include and Maybe decisions, since they never checked the new rule | Every Exclude, since its reason still holds |
| Criterion deleted | Excludes that used the deleted criterion as their reason | Includes, Maybes, and Excludes on other criteria |
| Criterion edited | All Include and Maybe decisions, plus Excludes that used the edited criterion | Excludes on other criteria |
A reset decision goes back into the queue of the reviewer who made it, blind again as normal. With the affected-only choice, a paper may return to one reviewer and not the other. If a reset decision fed a conflict resolution, that resolution and the final decision are cleared too, and anything done downstream for that paper is removed. A generated report is deleted as well, since it was built on decisions that no longer stand.
This is why the protocol is worth settling before screening starts. See Protocol.
Replacing a reviewer mid-review
You can give a reviewer or resolver role to someone else at any time from the review's Settings & Roles tab.
Changing the resolver asks nothing. Changing a reviewer who has already made decisions asks how the new reviewer should start:
- Continue from previous progress. The previous reviewer's decisions stay. The new reviewer sees them labelled as the previous reviewer's, can confirm each one as their own, or change it. Confirming a decision unchanged does not reopen anything.
- Reset & start fresh. The previous reviewer's decisions are removed. The other reviewer's decisions are untouched. Papers the departed reviewer had decided lose their final decision and return to the queues, along with anything done downstream for them, and any generated report is deleted. Before you confirm, the dialog shows how many decisions and papers are affected.
Continue is the right choice in almost every case: the record stays complete, and the new reviewer can work through the inherited decisions at their own pace.
The paper's history
Every paper in a dual review carries a full record. It holds each reviewer's vote and when it was cast, any change of vote, the resolver's decision and reason, and any reset with the reason for it. Open Logs on a paper to read it.
While blind review is on, a decision the other reviewer has made on a paper you have not yet decided shows in the log as "voted" without the direction. The direction appears once you have decided too.
This record, together with the review-level Activity log, is what lets you document your screening process in a methods section or answer a reviewer's question about a specific study.
What dual review does not cover yet
- Dual data extraction. Extraction is done by Paperguide AI and checked by one verifier in both review methods.
- AI as one of the two reviewers. Both reviewer roles are held by people.