The temptation, and why it's the wrong default
As submission volumes surge, reviewer pools have become harder to expand, and the share of reviews showing signs of LLM authorship continues to rise. Faced with this, conferences are increasingly reaching for the obvious solution: let the model write the review.
For computer scientists, automation is an especially tempting response. But I would argue it is the wrong default—not because today’s tools are too weak (they will improve), but because automating the act of reviewing undermines the very thing peer review is quietly for.
Peer review is not just a quality filter for papers. It is one of the few structured apprenticeships scientists get in judgment: reading a claim carefully, locating the evidence behind it, and deciding whether that evidence actually supports the conclusion. When we automate the review itself, we may save reviewer-hours, but we also stop training reviewers. In a world where AI-generated scientific output may vastly outpace what humans can produce, the human capacity to evaluate—to assess design, novelty, reasoning, and impact—becomes more valuable, not less. Handing that skill over to a model optimizes away the one competency we can least afford to lose.
There is also a second, subtler failure. When a model hands a reviewer a ready-made draft, the reviewer is likely to anchor to it. A large randomized study of an LLM feedback tool at ICLR found that a substantial fraction of reviewers who received AI feedback later revised their reviews. That can sometimes be helpful—but it also means the model is steering the discourse rather than merely supporting it. Scale that dynamic up, and you get what recent work has called a hivemind effect: AI-generated reviews begin to converge on one another, feedback diversity collapses, and—because gaming a predictable reviewer is easy—the ecosystem drifts toward an intellectual monoculture in which papers are optimized to satisfy the model rather than to be correct.
So the danger is not simply that AI-written reviews might be wrong. It is that they are fluent, generic, and contagious—and in the process, they quietly relocate the responsibility for judgment away from the scientist.
The automation reframe : evaluate the reasoning, don't produce it
Here is the move I am proposing.
This keeps the scientist firmly behind the driving wheel of reviewing. The scientist still does the reviewing — still forms the claims, still decides what matters. The assistant's job is narrower and more honest: did you make the argument you think you made, and does it hold up?
The two designs pull in opposite directions. A draft-writing assistant anchors the reviewer to the model's text; a role-checking assistant pushes the reviewer back to the paper.
The reframe rests on two hypotheses
First: scientific writing is argumentative in structure, and so are reviews. A scientific paper is not a flat list of facts. It is a web of background claims, evidence, and the authors’ own contributions, tied together by relations of support and opposition. Argument mining—the subfield devoted to recovering this structure automatically—has spent the past decade learning to identify these roles. Much of my own work in summarization and coverage analysis builds on exactly this premise (e.g., ArgLegalSumm, ARC).
Second: a review has its own distinct argumentative structure. The foundational study here—Hua et al., “Argument Mining for Understanding Peer Reviews” (NAACL 2019)—annotated more than ten thousand propositions in peer reviews into a small set of discourse roles. At a high level, a review is built from three core components:
- Facts — statements describing what the paper claims, proposes, or does.
- Evaluations — the reviewer’s critiques, assessments, and qualitative judgments of those facts.
- Requests — suggestions, questions, or demands for improvement.
Put the two together, and the design almost writes itself. If both the paper and the review can be decomposed into argumentative roles, then an assistant can check a review against the paper, role by role—a much more tractable and intellectually honest task than asking it to “write a good review.”
The key idea: the role types the feedback
The contribution I care about is not "generate evidence-based feedback" and not "rewrite the review." It is this: the argument role of a span determines which kind of feedback is even applicable to it. The role acts like a type system for feedback — it rules out the checks that don't apply and routes each span to the one that does.
You don't run a constructiveness check on a Fact — a Fact is supposed to be true, not constructive. You don't run a fidelity-to-the-paper check on a Request — a Request isn't a claim about the paper, it's a proposed action, so the right question is whether it is redundant or out of scope. Each role admits a different, predicated check:
Two properties distinguish this from scoring a review against a global rubric. First, the unit is the span, not the whole review: the same review can hold a well-grounded evaluation beside a hallucinated fact beside a redundant request, and a document-level score flattens exactly the distinctions a reviewer needs. Second, the check is selected by the role, not applied uniformly — the role decides which single question is worth asking of each span.
What argumentative discourse actually looks like
Here is a realistic review snippet, decomposed and checked span by span. Notice that none of these checks requires the assistant to have an opinion about the paper — only about the review.
"The paper introduces a contrastive pre-training objective and reports a 4-point F1 gain over the BERT baseline on GLUE. The gains are impressive and clearly demonstrate the method's superiority. However, the authors should evaluate on low-resource languages, and they never report variance across seeds."
| Review span | Role | Predicated check → feedback |
|---|---|---|
| "introduces a contrastive pre-training objective and reports a 4-point F1 gain over the BERT baseline on GLUE" | Fact | Verified (0.9). The gain and baseline appear in Table 2. Caveat: reported on 6 of 8 GLUE tasks — consider qualifying "on GLUE." |
| "The gains are impressive and clearly demonstrate the method's superiority" | Evaluation | Weakly grounded (0.3). Asserted, not argued. "Superiority" rests on a 4-point gain with no significance test. Tie the judgment to specific evidence, or soften it. |
| "the authors should evaluate on low-resource languages" | Request | Valid, out of scope (0.6). Not addressed, so not redundant — but the paper's stated scope is English GLUE. Flag whether this is fair or a new-paper-sized ask. |
| "they never report variance across seeds" | Request | Valid and material (0.9). Confirmed: no seed variance anywhere. This directly undercuts the "superiority" evaluation above — the two spans are in tension, worth surfacing. |
The payoff is that last row. Because the assistant reasons over roles and their relations, it catches that an unsupported evaluation ("clearly demonstrates superiority") is contradicted by the reviewer's own request (no variance reported). A draft-writing tool would have smoothed both into fluent prose. A role-checking tool hands the tension back to the human to resolve.
The proposed system
- Argument-role classifier as the trigger. The agent runs the human review through a role classifier (Fact / Evaluation / Request, extending the AMPERE scheme). This segmentation is the entry point for everything downstream.
- Each role maps to a predicated feedback type. Facts → factual verification against the paper's full text; Evaluations → validity/groundedness against paper and cited literature; Requests → necessity checks (already answered? in scope? novel?). The Fact check connects to long-context claim verification, which I studied in SynClaimEval.
- A quality score plus an explanation, per span. Not "3/5" but a calibrated score and a rationale the reviewer can act on — the explanation is what keeps the human doing the thinking.
Crucially, this inverts the anchoring problem. A draft-writing assistant pulls the reviewer toward the model's text; a role-checking assistant pushes the reviewer back toward the paper and their own reasoning — "you claimed the method is novel; here's the prior work you didn't mention; you decide." It provokes thought rather than supplying a conclusion.
How this differs from prior work
I want to be precise about what is and isn't new here, because several nearby ideas already exist.
In Hua et al., argument roles served only as auxiliary features for training a review-quality prediction model — the roles informed a prediction, but they never selected what kind of feedback a span should receive.
The position that LLMs should give feedback on human reviews rather than replace them is now well argued — recent position papers make exactly this case and even propose global quality rubrics (fidelity, clarity, fairness, proportionality, constructiveness). Large deployments like the ICLR Review Feedback Agent already check human reviews for vagueness and misunderstanding. And decomposition itself is not new: TreeReview breaks review into sub-questions, and DISAPERE (NAACL 2022) models discourse structure in review discussions.
What I add is the coupling: argument-role classification as the routing function that assigns each span a single predicated check, at span granularity rather than document granularity. Prior rubrics treat their criteria as parallel virtues applied to the whole review; here the role is a typing discipline that selects the one check a span can meaningfully receive. The role vocabulary itself (Fact / Evaluation / Request) is Hua et al.'s; the routing of role-to-check is the piece I claim.
Bottlenecks
1. Evaluating the judges
The assistant scores review spans — but who validates the assistant? LLM judges correlate weakly with human judgment, inflate scores, and reward fluency. If the role evaluator inherits those pathologies, we've just moved the monoculture up one level. This needs genuine human evaluation of the evaluator — ideally adversarial and calibrated against expert meta-reviewers — before any deployment.
2. The explanations only pay off under human–AI collaboration
A per-span rationale is worthless if the reviewer rubber-stamps it, and worthless if they ignore it. Its utility is entirely in the loop: the reviewer reads "this request may already be answered in Section 4," checks, and either revises or holds firm. That has to be studied as a collaboration — measuring whether the tool improves the reviewer's reasoning, not just changes their text.
Both bottlenecks are, I'd argue, features. They force the human to stay the final judge — which is the entire point.
Why this is the right first step
If the goal were only to reduce reviewer burden, writing the review is the shortest path. But the goal should be reliable science with reviewers who can still review. An argument-grounded evaluator threads that needle: it uses the machine where the machine is strong — segmenting structure, verifying claims against long text, flagging redundancy — and keeps the human where the human is irreplaceable: judgment, taste, deciding what the field should value.
References & pointers
- Hua, Nikolov, Badugu, Wang. Argument Mining for Understanding Peer Reviews. NAACL 2019. (AMPERE dataset; Fact / Evaluation / Request roles.)
- Kennard et al. DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions. NAACL 2022.
- Thakkar et al. Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025.
- Yun, Kwon, Kim, Kim. Position on LLM-Assisted Peer Review: Addressing the Reviewer Gap through Mentoring and Feedback. 2026.
- Chang et al. TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. 2025.
- Elaraby, Litman. ArgLegalSumm: Improving Abstractive Summarization of Legal Documents with Argument Mining. COLING 2022.
- Elaraby, Litman. ARC: Argument Representation and Coverage Analysis for Zero-Shot Long Document Summarization. EACL 2026.
- Elaraby et al. SynClaimEval: Evaluating the Utility of Synthetic Data in Long-Context Claim Verification. Eval4NLP @ AACL 2025.
Cite this blog post
Elaraby, M. (2026). AI-Assisted Reviews Should Be Argumentative. Personal blog. https://engsalem.github.io/posts/argumentative-reviews.html