FLOWING BENCHMARKS

文章品質と誤り検出を、別々に評価

7分野、異なる130本の論文、280箇所の固定執筆位置で、Evidence ありのオートコンプリートが分野横断で純優勢31件を記録。Evidence Card は6種類の明確な誤りを91.4%検出しました。

EVIDENCE CARD · 統制テスト・独立レビュー

明確な誤りをあえて入れ、 Evidence Cardが91.4%を検出

Red 誤りの再現率

91.4%32/35

32/35 件を確認。要確認を含めると 33/35 件。

あらかじめ用意した Red 課題 36 件kimi-k2.6 による独立レビュー

Red カード精度

97.1%34/35

34/35 枚を確認・要確認 1 枚・不採用 0 枚

確認できた Red 警告34
46/48 箇所を完了95.8%

Green 対照の正答率

100.0%11/11

Green 対照 11/11 件を正しく判定。誤った Red 判定は 0/11 件。

YELLOW の精度・自然な事例

6/8 枚を独立レビューで確認

75.0%

課題と分母が異なるため、二つの結果は別々に報告します。

報告指標

再現率、カード精度、対照精度、完了率

各指標は別々の分母で計算します。技術的失敗は「問題なし」に含めず、別に報告します。

46/48 箇所を完了

Red 誤りの再現率

91.4%

32/35 件を確認。要確認を含めると 33/35 件。

Red カード精度

97.1%

34/35 枚を確認・要確認 1 枚・不採用 0 枚

Green 対照の正答率

100.0%

Green 対照 11/11 件を正しく判定。誤った Red 判定は 0/11 件。

テスト完了率

95.8%

46/48 箇所を完了。技術的失敗 2 件は「問題なし」に含めず、別に報告。

誤りの種類別

6種類の誤りの結果

数値、方向、因果関係、対象、適用範囲、理論・定義を別々に集計しています。

濃色 = 確認済み・淡色 = 要確認

数値

5/6

技術的失敗なし

方向・極性

6/6

技術的失敗なし

因果関係

4/5

技術的失敗 1 件を別に報告

対象の取り違え

5/6

技術的失敗なし

適用範囲

6/6

技術的失敗なし

理論・定義

6/6

技術的失敗なし

YELLOW の精度・自然な事例

75.0%+43.8 pp

6/8 枚を独立レビューで確認

調整前後の Yellow カード

厳格な精度は 31.3% から 75.0% に向上しました。

調整前は 16 枚を出力し、確認済み 5 枚、要確認 1 枚、不採用 10 枚でした。全件を再実行した後は 8 枚で、確認済み 6 枚、要確認 0 枚、不採用 2 枚。50/51 箇所を完了し、技術的失敗 1 件は別に報告しています。

全件の再実行とカードごとの独立レビューが完了

公開ケース48件

各ケースの入力、カード、レビュー結果

検証対象の主張、固定した原文、生成カード、独立レビューをケースごとに掲載しています。見逃しと技術的失敗も記録に残しています。

表示 48 件
RED数値検出確認済み

溶液法原位大面积制备钙钛矿光电薄膜成膜的同步辐射可视化结晶过程研究

该研究报道,单个器件面积约15 cm²的大面积全柔性三维钙钛矿太阳能电池转换效率提升至15.23%。

ケースを開く ↓

検証対象の主張

该研究报道,单个器件面积约15 cm²的大面积全柔性三维钙钛矿太阳能电池转换效率提升至15.23%。

固定した原文

转换效率提升至5.23%(单个器件面积约15 cm2)

生成されたカード · RED

目标将该研究中约15 cm²全柔性三维钙钛矿太阳能电池的转换效率写为15.23%,而原文对同一器件报告的是5.23%,构成直接数值矛盾。

“基于各层均采用卷对卷 全溶液狭缝涂布方法制备的大面积全柔性三维钙钛矿薄膜太阳能电池转换效率提升至5.23% (单个器件面积 约15 cm2)”— 溶液法原位大面积制备钙钛矿光电薄膜成膜的同步辐射可视化结晶过程研究

独立レビュー · valid

TARGET QUOTE声称'单个器件面积约15 cm²的大面积全柔性三维钙钛矿太阳能电池转换效率提升至15.23%',SOURCE QUOTE对同一器件(单个器件面积约15 cm²、大面积全柔性三维钙钛矿薄膜太阳能电池)报告转换效率为'5.23%'。两个数值15.23%与5.23%在相同实体(约15 cm²大面积全柔性三维钙钛矿太阳能电池)、相同关系(转换效率)上存在直接数值矛盾,构成直接反驳。

RED数値検出確認済み

Genome-Scale CRISPR-Cas9 Knockout Screening in Human Cells

The GeCKO library targeted 18,080 human genes with 46,751 unique guide sequences.

ケースを開く ↓

検証対象の主張

The GeCKO library targeted 18,080 human genes with 46,751 unique guide sequences.

固定した原文

targeting 18,080 genes with 64,751 unique guide sequences

生成されたカード · RED

The stated 46,751 unique guide sequences contradicts the source's 64,751 for the same GeCKO library. The source agrees with the stated 18,080 targeted genes.

“We show that lentiviral delivery of a genome-scale CRISPR-Cas9 knockout (GeCKO) library targeting 18,080 genes with 64,751 unique guide sequences enables both negative and positive selection screening in human cells.”— Genome-Scale CRISPR-Cas9 Knockout Screening in Human Cells

独立レビュー · valid

The TARGET QUOTE states '46,751 unique guide sequences' for the GeCKO library, while the SOURCE QUOTE states '64,751 unique guide sequences' for the same GeCKO library targeting the same 18,080 genes. This is a direct numerical contradiction on the exact same entity (GeCKO library), same relation (number of unique guide sequences), and same scope. The source explicitly confirms the 18,080 genes figure but contradicts the 46,751 guide sequences figure. The proposed 'red' judgment is correct.

RED数値検出確認済み

Empagliflozin, Cardiovascular Outcomes, and Mortality in Type 2 Diabetes

EMPA-REG OUTCOME treated a total of 8,020 patients and followed them for a median of 3.1 years.

ケースを開く ↓

検証対象の主張

EMPA-REG OUTCOME treated a total of 8,020 patients and followed them for a median of 3.1 years.

固定した原文

A total of 7020 patients were treated (median observation time, 3.1 years)

生成されたカード · RED

The EMPA-REG OUTCOME report states that 7,020 patients were treated, not 8,020.

“A total of 7020 patients were treated (median observation time, 3.1 years).”— Empagliflozin, Cardiovascular Outcomes, and Mortality in Type 2 Diabetes

独立レビュー · valid

The target quote states '8,020 patients' while the source quote explicitly states '7020 patients were treated.' This is a direct numerical contradiction on the identical claim of patient count in the EMPA-REG OUTCOME trial. The entity (EMPA-REG OUTCOME trial), relation (patient count), and scope (total treated patients) match exactly, with only the number differing. The proposed color red is correct because the source directly contradicts the target proposition on a material point.

RED数値検出確認済み

TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

The paper's small-task setting allows up to 10,000 training examples, 100 numerical features, and 10 classes.

ケースを開く ↓

検証対象の主張

The paper's small-task setting allows up to 10,000 training examples, 100 numerical features, and 10 classes.

固定した原文

≤ 1 000 training examples, ≤ 100 purely numerical features and ≤ 10 classes

生成されたカード · RED

The paper defines its small-task setting as at most 1,000 training examples, not 10,000.

“has thus learned to solve novel small tabular classification tasks (≤ 1 000 training examples,≤ 100 purely numerical features without missing values and≤ 10 classes)”— TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

独立レビュー · valid

The TARGET QUOTE states 'up to 10,000 training examples,' while the SOURCE QUOTE explicitly states '≤ 1 000 training examples.' This is a direct numerical contradiction on the same entity (training examples), same relation (upper bound on small-task setting), same scope (the paper's definition of small-task setting). The source directly contradicts the target proposition that the limit is 10,000 by specifying it is 1,000.

RED数値検出確認済み

The Progress, Challenges, and Perspectives of Directed Greybox Fuzzing

The survey investigates 24 state-of-the-art fuzzers closely related to directed greybox fuzzing.

ケースを開く ↓

検証対象の主張

The survey investigates 24 state-of-the-art fuzzers closely related to directed greybox fuzzing.

固定した原文

investigation of 42 state-of-the-art fuzzers that are closely related to DGF

生成されたカード · RED

The survey explicitly reports investigating 42 state-of-the-art fuzzers closely related to DGF, not 24.

“Based on the investigation of 42 state-of-the-art fuzzers that are closely related to DGF, we conduct the first in-depth study to summarize the empirical evidence on the research progress of DGF.”— The Progress, Challenges, and Perspectives of Directed Greybox Fuzzing

独立レビュー · valid

The target quote claims '24 state-of-the-art fuzzers closely related to directed greybox fuzzing.' The source quote explicitly states '42 state-of-the-art fuzzers that are closely related to DGF' (where DGF = directed greybox fuzzing). The entity (fuzzers closely related to directed greybox fuzzing), relation (investigation of), and scope (state-of-the-art) match exactly; the only difference is the number (24 vs. 42), which is a direct numerical contradiction on a material point. The proposed color 'red' is correct.

RED数値見逃し

Evaluating the Impact of Brazil's Bolsa Família: Cash Transfer Programs in Comparative Perspective

The review describes Bolsa Família as benefiting roughly 17 million families.

ケースを開く ↓

検証対象の主張

The review describes Bolsa Família as benefiting roughly 17 million families.

固定した原文

benefiting roughly 11 million families

カードは生成されませんでした

RED方向・極性検出確認済み

Empagliflozin, Cardiovascular Outcomes, and Mortality in Type 2 Diabetes

In EMPA-REG OUTCOME, cardiovascular death was significantly higher with empagliflozin than with placebo.

ケースを開く ↓

検証対象の主張

In EMPA-REG OUTCOME, cardiovascular death was significantly higher with empagliflozin than with placebo.

固定した原文

significantly lower rates of death from cardiovascular causes (3.7%, vs. 5.9% in the placebo group)

生成されたカード · RED

The claim that cardiovascular death was higher with empagliflozin contradicts the account of the same trial’s cardiovascular results, which explicitly reports a lower rate of death from cardiovascular causes.

“Cardiovascular Outcomes The primary outcome that was reported previ ously showed a significantly lower risk of 3-point MACE in the pooled empagliflozin group than in the placebo group, a difference that was driven by a lower rate of death from cardiovas cular causes.23”— Empagliflozin and Progression of Kidney Disease in Type 2 Diabetes

独立レビュー · valid

The target quote explicitly claims cardiovascular death was 'significantly higher with empagliflozin than with placebo' in EMPA-REG OUTCOME. The source quote directly contradicts this by stating there was 'a lower rate of death from cardiovascular causes' in the empagliflozin group, which 'drove' the composite benefit. Both quotes concern the same trial (EMPA-REG OUTCOME), the same outcome (cardiovascular death/mortality), the same comparison (empagliflozin vs. placebo), and the same directionality claim—but with opposite directions. The contradiction is material and unambiguous: 'higher' vs. 'lower' for the identical endpoint in the identical trial.

RED方向・極性検出確認済み

Evaluating the Impact of Brazil's Bolsa Família: Cash Transfer Programs in Comparative Perspective

The review concludes that Bolsa Família increased inequality and extreme poverty.

ケースを開く ↓

検証対象の主張

The review concludes that Bolsa Família increased inequality and extreme poverty.

固定した原文

helped reduce inequality and extreme poverty

生成されたカード · RED

The review attributes 21 percent of Brazil’s 1995–2004 Gini decline to Bolsa Família, contradicting the claimed increase in inequality.

“They found that the Gini index for Brazil fell by 4.7 percent from 1995 to 2004 and that Bolsa Familia was responsible for 21 percent of that fall (before 1995 there were no CCT programs in Brazil, so there is confidence that 1995 is a baseline year).”— Evaluating the Impact of Brazil's Bolsa Família: Cash Transfer Programs in Comparative Perspective

独立レビュー · valid

The TARGET QUOTE states 'The review concludes that Bolsa Família increased inequality.' The SOURCE QUOTE from the same review states that Bolsa Família was responsible for '21 percent of that fall' in the Gini index, where 'fall' means a decrease in inequality. A decrease in the Gini index signifies reduced inequality. Therefore, the source directly contradicts the target proposition: the review concludes Bolsa Família decreased inequality (contributed to 21% of the Gini decline), not increased it. The scope matches—both concern the review's conclusion about Bolsa Família's effect on inequality.

1 / 6

適用範囲と制約

  • 1この統制テストは、明確な誤りを見つける力を測るものです。実際の論文で誤りが起きる頻度を推定するものではありません。
  • 2一部の元論文は初期の開発テストにも使われています。この結果は回帰テストと統制下の検出率には使えますが、無作為標本からの頻度推定には使えません。
  • 3調整後に出力された Yellow は8枚だけです。6/8という分母を必ず併記し、75%を全体の精度と解釈しないでください。

← オートコンプリート評価