Decision-model benchmark · October 2026

Which model finds the vulnerable files first?

12 decision models — 6 open-weight, 6 closed — scored every source file of 16 open-source projects. A file counts when a known CVE for that exact version lives in it.

Best open-weight

80.1%

Clef · Apache-2.0

Best closed

79.4%

Jev 1.14

Largest files first

45.5%

random order: 15.2%

Files scored

48,268

479 CVEs · 16 projects

CVE recall in the top 10% of files

Share of known CVEs whose file is ranked in the top 10% of its project, averaged over 16 projects.

Open weightsClosedBaselineAPISelf-hosted (H200)

Model

Recall, top 10%

Cost, all files

AUC

ClefOpen weights · Apache-2.0 · 27B · API

80.1%

$6.36

0.902

Jev 1.14Closed · API

79.4%

$6.39

0.903

d3 v3.1.0Open weights · Apache-2.0 · 26B · self-hosted

79.3%

$22.49

0.905

d1Closed · API

77.6%

$2.31

0.897

Mercury DecideClosed · API

76.4%

$2.98

0.894

Clef FlashOpen weights · Apache-2.0 · 9B · API

76.3%

$3.18

0.886

GPT-6 Luna DecisionsClosed · API

76.2%

$18.06

0.886

StartLux Decision 27BOpen weights · CC-BY-NC-4.0 · 27B · self-hosted

75.8%

$9.14

0.908

Microsoft Decision-1Closed · API

74.6%

$5.72

0.887

Decider Flash 1.2Open weights · Apache-2.0 · 4B · API

73.4%

$1.43

0.882

Drex 2.0Open weights · Open RAIL-M · 35B-A3B · API

71.5%

$8.08

0.877

Span-01 LiteClosed · API

21.5%

free

0.633

File size (baseline)Baseline

45.5%

—

0.780

Random (baseline)Baseline

15.2%

—

0.494

↓ ranking.svg

Recall against cost

API cost at list price; self-hosted cost is H200 GPU time at $4.54/h.

Open weightsClosedBaselineAPISelf-hosted (H200)

Outside the plotted range: Span-01 Lite (21.5%, free).

By project

Recall in the top 10% of files, per project.

Project	Clef	Jev 1.14	d3 v3.1.0	d1	Mercury	Clef Flash	GPT-6 Luna	StartLux 27B	MS Decision-1	Decider Flash	Drex 2.0	Span-01 Lite	File size
FFmpeg 6.0C · 4,144 files · 88 CVEs	42	40	40	43	40	42	43	39	43	41	42	6	44
Fission 1.22Go · 529 files · 17 CVEs	88	82	76	82	76	82	88	53	76	76	53	12	76
Rocket.Chat 7.11TypeScript · 8,723 files · 30 CVEs	90	90	93	87	87	90	83	93	77	87	90	50	40
Tomcat 11.0.18Java · 3,297 files · 44 CVEs	86	93	91	91	91	89	93	86	86	80	84	20	80
Drupal 10.2PHP · 14,267 files · 23 CVEs	87	83	74	74	78	74	83	78	87	78	74	30	52
Calibre 7.13Python · 1,592 files · 18 CVEs	83	83	94	67	89	78	78	89	61	67	83	22	28
uutils 0.5Rust · 600 files · 45 CVEs	84	73	80	71	80	71	67	82	69	76	80	11	44
protobuf 3.19C++/Java · 1,405 files · 10 CVEs	100	100	100	100	90	100	100	90	90	100	90	0	40
Netty 4.2.14Java · 3,609 files · 79 CVEs	90	86	86	89	81	87	84	90	78	81	87	5	53
Coder 2.34Go · 5,392 files · 18 CVEs	94	89	94	94	94	83	89	94	94	89	78	33	67
SurrealDB 3.0.5Rust · 1,465 files · 31 CVEs	39	42	42	45	39	52	26	39	45	36	39	6	16
Grav 2.0.13PHP · 727 files · 21 CVEs	76	76	76	81	76	67	76	76	71	52	76	57	86
Chatwoot 2.1.1Ruby · 1,963 files · 11 CVEs	82	82	82	73	82	64	91	82	91	73	54	46	0
pypdf 6.11Python · 113 files · 27 CVEs	59	70	59	56	59	63	59	52	63	59	63	0	44
lodash 4.7JavaScript · 288 files · 10 CVEs	80	80	80	90	60	80	60	70	60	80	50	30	0
SwiftNIO HTTP/2 1.19Swift · 154 files · 7 CVEs	100	100	100	100	100	100	100	100	100	100	100	14	57

Recall <4040–5555–7070–8080–90≥90

GitLab CE 18.11.2

Top two models only. 48,854 files, 71 CVEs.

Model

CVEs in top 10% of files

AUC

Jev 1.14

66 / 71

0.920

Clef

61 / 71

0.905

File size (baseline)

43 / 71

0.768

Random (baseline)

17 / 71

0.509

Chunk size

StartLux Decision 27B, 4 projects (FFmpeg, Fission, Rocket.Chat, Tomcat), 498 files over 16k tokens. Large files scored three ways.

Large files scored by

Recall, top 10%

AUC

16k chunks, max over chunks

67.8%

0.888

Full context (131k)

68.7%

0.889

First 16k only (truncated)

67.8%

0.885

All results

Model	Weights	License	Size	Run	Top 5%	Top 10%	Top 20%	AUC	Cost
Clefcloudflare/clef	Open	Apache-2.0	27B	API	62.1%	80.1%	91.0%	0.902	$6.36
Jev 1.14typesafe/jev-1.14	Closed	Proprietary	—	API	61.6%	79.4%	90.2%	0.903	$6.39
d3 v3.1.0vllm-sr/d3	Open	Apache-2.0	26B	Self-hosted	62.9%	79.3%	90.3%	0.905	$22.49
d1liquid/d1	Closed	Proprietary	—	API	57.3%	77.6%	86.6%	0.897	$2.31
Mercury Decideinception/mercury-decide	Closed	Proprietary	—	API	62.0%	76.4%	88.9%	0.894	$2.98
Clef Flashcloudflare/clef-flash	Open	Apache-2.0	9B	API	56.6%	76.3%	88.2%	0.886	$3.18
GPT-6 Luna Decisionsopenai/gpt-6-luna-decisions	Closed	Proprietary	—	API	53.3%	76.2%	87.8%	0.886	$18.06
StartLux Decision 27Bstartlux-models/StartLux-Decision-v1.1-27B-Instant	Open	CC-BY-NC-4.0	27B	Self-hosted	59.7%	75.8%	91.3%	0.908	$9.14
Microsoft Decision-1microsoft/microsoft-decision-1	Closed	Proprietary	—	API	52.3%	74.6%	88.5%	0.887	$5.72
Decider Flash 1.2perplexity/pplx-decider-flash-v1.2	Open	Apache-2.0	4B	API	57.3%	73.4%	87.0%	0.882	$1.43
Drex 2.0nace-ai/drex-v2.0	Open	Open RAIL-M	35B-A3B	API	50.5%	71.5%	82.9%	0.877	$8.08
Span-01 Literespan/span-01-lite	Closed	Proprietary	—	API	10.7%	21.5%	39.7%	0.633	free
File size (baseline)	—	—	—	—	32.4%	45.5%	67.9%	0.780	—
Random (baseline)	—	—	—	—	8.8%	15.2%	29.3%	0.494	—

Method

Every file with a source, template or config extension, whole projects, no sampling.
Ground truth: published CVEs affecting each exact release, mapped to the files they live in.
Same prompt for every model. The model sees only the file path and its code.
Ranked by the answer to one fixed question: how likely the file contains a vulnerability.
Files longer than 16k tokens are split; the file takes its highest chunk score.
Recall is averaged per project, so each project counts equally.
Gaps under about 5 points are within noise (bootstrap 95% CI).
Open weights: model weights published on Hugging Face; licence as stated by the publisher.