Contents

LLMs can help screen research ideas, but their novelty judgments depend too heavily on prompts and evaluation conditions to serve as final decisions. Ai2’s *Old Ideas, Novel Problems*, released on October 27, 2026, shows why even confident answers need independent verification.

Key Takeaways#

  • On the same pairs of human-written and AI-generated ideas, changing the prompt dropped GPT-5.4’s accuracy from 93.5% to 40.9%.
  • Judges validated on human-written research can behave differently when evaluating AI-generated ideas.
  • In the configurations tested, adding reasoning, literature searches, and expense did not consistently improve results.
  • Use novelty scores to prioritize literature reviews, and check whether judgments hold up across reasonable prompt variations.

What the Benchmark Measures#

Ai2 built the benchmark from ICLR 2026 reviews. Rather than treating acceptance as evidence of novelty, it used comments that explicitly addressed originality.

The human-written ideas consisted of two groups.

  • 154 high-novelty ideas: Accepted papers ranked in the top 10% within their research areas, whose novelty reviewers praised without any reviewer disputing it.
  • 145 low-novelty ideas: Rejected papers ranked in the bottom 10%, with concerns about originality and no reviewer positively acknowledging their novelty.

A separate evaluation paired the 154 high-novelty human ideas with 154 ideas from a basic LLM generator. The generator received a research area and instructions to propose a novel idea, without literature searches or feedback. This condition matters because judging generated proposals is a typical intended use of novelty evaluators.

Small Prompt Changes Sharply Reversed Judgments#

The clearest result came from changing how GPT-5.4 was asked to compare human and AI ideas. One prompt stated that reviewers had found only one idea novel and asked the model to identify it. The other simply asked which idea was novel.

Accuracy fell from 93.5% to 40.9%, a drop of 52.6 percentage points. The model reversed its judgment on more than half of the same pairs. Because the first prompt also supplied information about the expected answer, its high score should not be taken as evidence that an unconstrained novelty judge would be equally reliable.

The effects varied by model and dataset. When comparing human ideas, the same prompt change shifted GPT-5.4’s accuracy by about 7 percentage points and Claude Opus 4.6’s by 21 points. In comparisons between two AI generators, the tested prompt changes reversed roughly one-quarter to one-third of judgments, averaged across six judges. Even when it is unclear which idea is actually more novel, these reversals reveal unstable judgments.

Adding Tools Did Not Solve the Problem#

The study found no consistent improvement from additional search or reasoning. In one comparison, a judge configured with extra-high reasoning effort and repeated arXiv searches achieved a macro F1 of 0.78, while a Claude Opus 4.6 configuration given five retrieved abstracts scored 0.89. At the API prices used in the study, the search agent cost about 10 times as much per judgment.

Specialized evaluators also underperformed a simple baseline on the dataset combining human and AI-generated ideas.

| Evaluator | Macro F1 | Total Evaluation Cost | |---|---:|---:| | Prompted Claude Opus 4.6 | 0.854 | $2.59 | | Idea Novelty Checker | 0.642 | $30.66 | | AI Scientist Novelty Evaluator | 0.772 | $82.05 |

A higher macro F1 indicates better classification of novel and non-novel ideas. These results come from specific settings with different voting methods and call configurations; they do not show that search is generally ineffective. They do show that more complex pipelines need measured evidence of improvement.

Using Judges Responsibly#

Treat LLM judgments as support for review. Useful outputs identify the closest prior work and explain how a proposal differs clearly enough for researchers to verify.

Before screening ideas with a judge, do the following.

1. Validate it on proposals similar to those you will actually evaluate, including AI-generated ideas where relevant. 2. Try at least two reasonable prompts and record how often judgments change. 3. Read the closest cited papers and check whether they already contain the claimed contribution. 4. Refer cases with conflicting judgments to expert review before rejecting an idea or claiming originality.

This benchmark has limitations. Labels for generated ideas have weaker support: the researchers assumed they were less novel than the selected human research. Experts rated the human ideas higher in all 30 pairs they checked, but that sample was skewed toward cases the judges got wrong and did not validate every pair. The results also come from ICLR research, so other fields require their own validation.

Next Step#

Review 10 current research proposals using two novelty-judging prompts, then manually check the closest prior work for every proposal whose judgment changes.

Source: https://huggingface.co/blog/allenai/novelty-bench

Contents