Takeaways
- Low accuracy on new rulings: Across 65 LLMs tested on motions to dismiss decided after model release dates, only 16% of predictions accurately matched the judges' rulings (and that number is probably too high).
- Training data impact: For motions decided prior to the models' release dates, accuracy nearly tripled to 45%.
- Limited top performers: Only four models predicted correctly at least half the time on post-release cases.
It is undeniable that AI is getting pretty good at many legal tasks. It can output a solid NDA that requires minimal edits or a legal research memo that hits just about every salient legal point. But you often hear people say that AI outputs miss the nuance of a legal argument or subtle fact patterns lurking under the surface of a dispute. It is these subtleties that can, and often do, sway a judge’s opinion.
So, taking inspiration from LegalForecastBench, we set out to determine how well AI could accurately predict the outcome of fully briefed motions to dismiss filed in federal courts. We adapted LegalForecastBench so we could conduct automated pairwise comparisons in Certera to determine which models predicted the outcomes most accurately.
We tested 65 models on five real motions to dismiss and generated 1002 individual predictions. We ran two sets of motions: motions decided both before and after the models’ release dates. To make sure they were not in the model’s training data, we selected three after the models were released, and to compare to results for motions that might be within the LLM’s training data, we tested two motions to dismiss before the models were released. Attorney reviewers then scored each prediction output against the actual ruling with a Brier score and then chose the more accurate AI model.
In short, the models had a difficult time accurately predicting how a judge would rule on the motions. On the three motions handed down after the models’ release dates, only 16% of the predictions were correct (and even then, this number might be inflated by the success rate for one motion in which the plaintiff conceded many of his claims).
Not surprisingly the models fared better on motions decided before their release date and whose rulings could have been in the models' training data, but even then, the AI models were still wrong more than half of the time. For cases decided prior to the models’ release date, the accuracy increased to 45%.
What follows is how we ran the test, and what we found.
Test Setup
We used blind pairwise comparisons produced in Certera. Each model received the same briefing documents the judge reviewed before making a ruling: the complaint, the motion to dismiss, the opposition and the reply.
Further, we structured the test as follows:
- No outside help. Models were told not to use web search, retrieval or any other tool, and to reason only from the record.
- Claim-by-claim forecasts. For each challenged claim, the model gave a 0% to 100% probability that the court would dismiss it. Dismissal with leave to amend counted as dismissed, but a claim that survived in any part did not.
- Blind attorney review. Attorneys then blindly reviewed responses side by side, scored each response's claim-by-claim probabilities against the court's actual ruling using a Brier score (standard measure for probability forecasts) and picked a winner (or a tie, good or bad).
Results From Rulings Issued After Models’ Release
The models were given three cases decided after their release dates to ensure the decision was not in their training data. Cumulatively the accuracy rate across all three motions was 16%. However, as with all AI outputs, one needs to dig a little deeper to verify accuracy. Some might consider the success rate inflated by the fact that for one case, Alonzo v. UT Health, the accuracy rate was 41%, but for the other two cases, the accuracy rate was single digits. Presumably, the higher success rate for Alonzo was attributable to the fact that the plaintiff in that ruling conceded a good portion of the claims at issue.
The results on a case by case basis:
Case | Court | Nature of the case | Ruling | Avg. cost per response | Predictions judged correct |
|---|---|---|---|---|---|
Castro Alonzo v. UT Health Science Center at San Antonio | W.D. Tex. (Judge David Alan Ezra) | A dismissed medical resident sued the university for race and national-origin discrimination and retaliation (Titles VI and VII), plus ADA, due-process, contract and tort claims. | Motion granted | $0.32 | 41% (82 of 200) |
Sable Offshore Corp. v. County of Santa Barbara | C.D. Cal. (Judge Dolly M. Gee) | An oil company and ExxonMobil challenged the County's refusal to transfer permits needed to restart a pipeline. They alleged a taking, federal preemption and breach of a settlement agreement. | Challenged claims dismissed, some with leave to amend | $0.69 | 7% (13 of 196) |
League of Women Voters of Indiana v. Morales | S.D. Ind. (Judge Matthew P. Brookman) | Voting-rights groups challenged Indiana's new proof-of-citizenship voter verification laws under the National Voter Registration Act and the Civil Rights Act. | All claims dismissed for lack of standing, with 28 days to amend | $0.67 | 1.5% (3 of 200) |
All three | $0.56 | 16% (98 of 596) |
The Overall leaderboard: Cases Decided After Model Release
Only four models predicted correctly at least half the time, and only one made a correct prediction in all three cases. The table ranks the LLMs tested on the three post-release cases by the share of its predictions reviewers judged correct.
Rank | Model | Correct / voted | Accuracy | Alonzo | Sable | Indiana | Avg. cost per response |
|---|---|---|---|---|---|---|---|
1 | Gemini 3.5 Flash | 5 / 8 | 63% | 5/5 | 0/1 | 0/2 | $0.48 |
2 | Gemini 3.1 Flash Lite | 3 / 5 | 60% | 3/3 | 0/1 | 0/1 | $0.03 |
3 | Gemini 3.6 Flash | 6 / 11 | 55% | 1/1 | 5/6 | 0/4 | $0.24 |
3 | Mimo V2.5 Pro | 6 / 11 | 55% | 3/3 | 2/4 | 1/4 | $0.13 |
5 | Gemini 3.5 Flash Lite | 3 / 6 | 50% | 2/3 | 1/2 | 0/1 | $0.07 |
5 | Mimo V2.5 | 1 / 2 | 50% | — | 1/2 | — | $0.05 |
7 | Claude Opus 4.8 | 3 / 8 | 38% | 3/4 | 0/2 | 0/2 | $1.64 |
7 | Mistral Medium 3.5 | 3 / 8 | 38% | 2/3 | 0/3 | 1/2 | $0.48 |
9 | Claude Fable 5 | 2 / 6 | 33% | 2/2 | 0/2 | 0/2 | $3.66 |
9 | Claude Sonnet 4.6 | 2 / 6 | 33% | 2/2 | 0/2 | 0/2 | $0.55 |
9 | GPT-6 Luna | 2 / 6 | 33% | 1/2 | 0/2 | 1/2 | $0.01 |
12 | GPT-5.4 | 3 / 10 | 30% | 3/3 | 0/6 | 0/1 | $0.46 |
12 | Gemini 3.8 Flash | 3 / 10 | 30% | 3/3 | 0/5 | 0/2 | $0.48 |
14 | Gemini 3.7 Flash | 2 / 7 | 29% | 1/1 | 1/4 | 0/2 | $0.22 |
15 | Gemini 3.1 Pro | 13 / 51 | 25% | 13/19 | 0/15 | 0/17 | $0.63 |
16 | GPT-6 Sol | 2 / 8 | 25% | 2/2 | 0/2 | 0/4 | $0.22 |
16 | DeepSeek V4 Flash | 1 / 4 | 25% | 1/2 | — | 0/2 | $0.07 |
16 | GPT-5.6 Terra | 1 / 4 | 25% | 1/1 | 0/2 | 0/1 | $0.22 |
19 | Minimax M2.7 HighSpeed | 2 / 9 | 22% | 1/4 | 1/3 | 0/2 | $0.04 |
20 | Grok-4.6 | 3 / 15 | 20% | 3/4 | 0/7 | 0/4 | $0.33 |
20 | Claude Opus 5 | 1 / 5 | 20% | 1/2 | 0/2 | 0/1 | $1.62 |
20 | GLM 5 Turbo | 1 / 5 | 20% | 1/1 | 0/1 | 0/3 | $0.14 |
20 | GPT-5.6 Sol | 1 / 5 | 20% | 1/2 | 0/2 | 0/1 | $0.54 |
20 | Minimax M3 | 1 / 5 | 20% | 1/2 | 0/1 | 0/2 | $0.04 |
25 | GPT-5.5 | 7 / 40 | 18% | 7/14 | 0/10 | 0/16 | $1.13 |
26 | Grok-4.3 | 2 / 12 | 17% | 2/5 | 0/4 | 0/3 | $0.05 |
26 | GPT-6 Astra | 1 / 6 | 17% | 1/4 | 0/2 | — | $1.19 |
26 | Grok-4.20 Reasoning | 1 / 6 | 17% | 1/3 | — | 0/3 | $0.05 |
26 | Mimo V2.6 Pro | 1 / 6 | 17% | 1/2 | 0/3 | 0/1 | $0.06 |
30 | Claude Fable 5.1 | 1 / 7 | 14% | 1/1 | 0/1 | 0/5 | $5.15 |
30 | Claude Opus 4.6 | 1 / 7 | 14% | 1/1 | 0/3 | 0/3 | $2.96 |
30 | Inkling | 1 / 7 | 14% | 1/4 | — | 0/3 | $0.70 |
30 | Mimo V2.6 Flash | 1 / 7 | 14% | 1/3 | 0/2 | 0/2 | $0.01 |
34 | Claude Sonnet 5 | 1 / 8 | 13% | 1/2 | 0/1 | 0/5 | $1.20 |
34 | DeepSeek V4.1 Flash | 1 / 8 | 13% | 1/4 | 0/3 | 0/1 | $0.01 |
34 | Qwen3.6 Max | 1 / 8 | 13% | 0/2 | 1/4 | 0/2 | $0.24 |
37 | DeepSeek V4 Flash 0731 | 1 / 9 | 11% | 1/4 | 0/2 | 0/3 | $0.11 |
37 | Muse Spark 1.2 | 1 / 9 | 11% | 1/3 | 0/4 | 0/2 | $0.26 |
39 | Minimax M2.5 | 1 / 10 | 10% | 1/3 | 0/2 | 0/5 | $0.03 |
39 | Minimax M2.5 HighSpeed | 1 / 10 | 10% | 1/6 | — | 0/4 | $0.05 |
41 | GLM 5 | 1 / 13 | 8% | 0/4 | 1/5 | 0/4 | $0.37 |
42 | GLM 5.1 | 1 / 14 | 7% | 1/4 | 0/4 | 0/6 | $0.22 |
43 | Claude Opus 5.5 | 3 / 50 | 6% | 3/16 | 0/19 | 0/15 | $0.86 |
44 | GLM 5.3 Flash | 0 / 11 | 0% | 0/3 | 0/4 | 0/4 | $0.03 |
44 | Qwen3.6 Plus | 0 / 11 | 0% | 0/2 | 0/6 | 0/3 | $0.83 |
44 | GLM 5.3 | 0 / 9 | 0% | 0/4 | 0/2 | 0/3 | $0.06 |
44 | GLM 5v Turbo | 0 / 9 | 0% | 0/1 | 0/5 | 0/3 | $0.15 |
44 | Gemini 3 Flash | 0 / 9 | 0% | 0/1 | 0/4 | 0/4 | $0.05 |
44 | Grok-4.20 Non-Reasoning | 0 / 9 | 0% | 0/2 | 0/3 | 0/4 | $0.02 |
44 | Kimi K3 | 0 / 9 | 0% | 0/4 | 0/3 | 0/2 | $0.39 |
44 | Kimi K2.6 | 0 / 8 | 0% | 0/2 | 0/2 | 0/4 | $0.33 |
44 | Claude Opus 4.7 | 0 / 7 | 0% | 0/1 | 0/2 | 0/4 | $1.23 |
44 | DeepSeek V4 Pro 0813 | 0 / 7 | 0% | 0/4 | 0/1 | 0/2 | $0.22 |
44 | Minimax M2.7 | 0 / 7 | 0% | 0/2 | 0/1 | 0/4 | $0.04 |
44 | Muse Spark 1.1 | 0 / 7 | 0% | 0/2 | 0/5 | — | $0.14 |
44 | DeepSeek V3.2 Thinking | 0 / 6 | 0% | 0/2 | 0/3 | 0/1 | $0.18 |
44 | Muse Spark 1.3 | 0 / 6 | 0% | 0/3 | 0/1 | 0/2 | $0.27 |
44 | GLM 5.2 | 0 / 5 | 0% | 0/2 | 0/1 | 0/2 | $0.07 |
44 | GPT-5.6 Luna | 0 / 5 | 0% | 0/3 | — | 0/2 | $0.02 |
44 | Grok-4.7 | 0 / 5 | 0% | 0/1 | 0/3 | 0/1 | $0.44 |
44 | Kimi K2.5 | 0 / 5 | 0% | 0/1 | 0/2 | 0/2 | $0.01 |
44 | Qwen3.7 Plus | 0 / 5 | 0% | — | 0/2 | 0/3 | $0.06 |
44 | Grok-4.5 | 0 / 4 | 0% | 0/1 | — | 0/3 | $0.18 |
Caveats
However, there are caveats. The case a model happened to draw affected its score. Gemini 3.5 Flash ranks first, but five of its eight tests were for Alonzo in which the plaintiff conceded some of the claims. But, for the more contested cases, Gemini 3.5 Flash missed all three. Therefore the more convincing results belong to Gemini 3.6 Flash, which went 5 for 6 on one of the cases (Sable), and Mimo V2.5 Pro, was the only model judged correct at least once in every case.
Additionally, one might notice that certain models (especially newly released models) were served up more often than others. That is so because the algorithm is set up to serve up models more often if they have fewer votes in the database to better standardize ranks. Even so, removing the three models with more than 15 decisions barely changes the post-release average: it goes from 16.4% to 16.5%
The Leaderboard Without Alonzo
When you remove the predictions for Alonzo, the accuracy rate plummets to 4% and 27 of the models did not even win a pairwise comparison with it removed.
Rank | Model | Correct / voted | Accuracy | Sable | Indiana | Avg. cost per response |
|---|---|---|---|---|---|---|
1 | Gemini 3.6 Flash | 5 / 10 | 50% | 5/6 | 0/4 | $0.24 |
1 | Mimo V2.5 | 1 / 2 | 50% | 1/2 | — | $0.05 |
3 | Mimo V2.5 Pro | 3 / 8 | 38% | 2/4 | 1/4 | $0.15 |
4 | Gemini 3.5 Flash Lite | 1 / 3 | 33% | 1/2 | 0/1 | $0.08 |
5 | GPT-6 Luna | 1 / 4 | 25% | 0/2 | 1/2 | $0.01 |
6 | Minimax M2.7 HighSpeed | 1 / 5 | 20% | 1/3 | 0/2 | $0.04 |
6 | Mistral Medium 3.5 | 1 / 5 | 20% | 0/3 | 1/2 | $0.42 |
8 | Gemini 3.7 Flash | 1 / 6 | 17% | 1/4 | 0/2 | $0.23 |
8 | Qwen3.6 Max | 1 / 6 | 17% | 1/4 | 0/2 | $0.22 |
10 | GLM 5 | 1 / 9 | 11% | 1/5 | 0/4 | $0.44 |
11 | Claude Opus 5.5 | 0 / 34 | 0% | 0/19 | 0/15 | $1.05 |
11 | Gemini 3.1 Pro | 0 / 32 | 0% | 0/15 | 0/17 | $0.68 |
11 | GPT-5.5 | 0 / 26 | 0% | 0/10 | 0/16 | $1.40 |
11 | Grok-4.6 | 0 / 11 | 0% | 0/7 | 0/4 | $0.37 |
11 | GLM 5.1 | 0 / 10 | 0% | 0/4 | 0/6 | $0.19 |
11 | Qwen3.6 Plus | 0 / 9 | 0% | 0/6 | 0/3 | $0.96 |
11 | GLM 5.3 Flash | 0 / 8 | 0% | 0/4 | 0/4 | $0.03 |
11 | GLM 5v Turbo | 0 / 8 | 0% | 0/5 | 0/3 | $0.16 |
11 | Gemini 3 Flash | 0 / 8 | 0% | 0/4 | 0/4 | $0.06 |
11 | GPT-5.4 | 0 / 7 | 0% | 0/6 | 0/1 | $0.48 |
11 | Gemini 3.8 Flash | 0 / 7 | 0% | 0/5 | 0/2 | $0.58 |
11 | Grok-4.20 Non-Reasoning | 0 / 7 | 0% | 0/3 | 0/4 | $0.02 |
11 | Grok-4.3 | 0 / 7 | 0% | 0/4 | 0/3 | $0.05 |
11 | Minimax M2.5 | 0 / 7 | 0% | 0/2 | 0/5 | $0.03 |
11 | Claude Fable 5.1 | 0 / 6 | 0% | 0/1 | 0/5 | $5.77 |
11 | Claude Opus 4.6 | 0 / 6 | 0% | 0/3 | 0/3 | $3.17 |
11 | Claude Opus 4.7 | 0 / 6 | 0% | 0/2 | 0/4 | $1.24 |
11 | Claude Sonnet 5 | 0 / 6 | 0% | 0/1 | 0/5 | $1.46 |
11 | GPT-6 Sol | 0 / 6 | 0% | 0/2 | 0/4 | $0.24 |
11 | Kimi K2.6 | 0 / 6 | 0% | 0/2 | 0/4 | $0.35 |
11 | Muse Spark 1.2 | 0 / 6 | 0% | 0/4 | 0/2 | $0.33 |
11 | DeepSeek V4 Flash 0731 | 0 / 5 | 0% | 0/2 | 0/3 | $0.13 |
11 | GLM 5.3 | 0 / 5 | 0% | 0/2 | 0/3 | $0.07 |
11 | Kimi K3 | 0 / 5 | 0% | 0/3 | 0/2 | $0.51 |
11 | Minimax M2.7 | 0 / 5 | 0% | 0/1 | 0/4 | $0.05 |
11 | Muse Spark 1.1 | 0 / 5 | 0% | 0/5 | — | $0.17 |
11 | Qwen3.7 Plus | 0 / 5 | 0% | 0/2 | 0/3 | $0.06 |
11 | Claude Fable 5 | 0 / 4 | 0% | 0/2 | 0/2 | $4.47 |
11 | Claude Opus 4.8 | 0 / 4 | 0% | 0/2 | 0/2 | $2.38 |
11 | Claude Sonnet 4.6 | 0 / 4 | 0% | 0/2 | 0/2 | $0.70 |
11 | DeepSeek V3.2 Thinking | 0 / 4 | 0% | 0/3 | 0/1 | $0.20 |
11 | DeepSeek V4.1 Flash | 0 / 4 | 0% | 0/3 | 0/1 | $0.02 |
11 | GLM 5 Turbo | 0 / 4 | 0% | 0/1 | 0/3 | $0.14 |
11 | Grok-4.7 | 0 / 4 | 0% | 0/3 | 0/1 | $0.48 |
11 | Kimi K2.5 | 0 / 4 | 0% | 0/2 | 0/2 | $0.01 |
11 | Mimo V2.6 Flash | 0 / 4 | 0% | 0/2 | 0/2 | $0.01 |
11 | Mimo V2.6 Pro | 0 / 4 | 0% | 0/3 | 0/1 | $0.06 |
11 | Minimax M2.5 HighSpeed | 0 / 4 | 0% | — | 0/4 | $0.06 |
11 | Claude Opus 5 | 0 / 3 | 0% | 0/2 | 0/1 | $2.16 |
11 | DeepSeek V4 Pro 0813 | 0 / 3 | 0% | 0/1 | 0/2 | $0.31 |
11 | GLM 5.2 | 0 / 3 | 0% | 0/1 | 0/2 | $0.09 |
11 | GPT-5.6 Sol | 0 / 3 | 0% | 0/2 | 0/1 | $0.59 |
11 | GPT-5.6 Terra | 0 / 3 | 0% | 0/2 | 0/1 | $0.24 |
11 | Gemini 3.5 Flash | 0 / 3 | 0% | 0/1 | 0/2 | $0.68 |
11 | Grok-4.20 Reasoning | 0 / 3 | 0% | — | 0/3 | $0.03 |
11 | Grok-4.5 | 0 / 3 | 0% | — | 0/3 | $0.20 |
11 | Inkling | 0 / 3 | 0% | — | 0/3 | $1.27 |
11 | Minimax M3 | 0 / 3 | 0% | 0/1 | 0/2 | $0.04 |
11 | Muse Spark 1.3 | 0 / 3 | 0% | 0/1 | 0/2 | $0.43 |
11 | DeepSeek V4 Flash | 0 / 2 | 0% | — | 0/2 | $0.12 |
11 | GPT-5.6 Luna | 0 / 2 | 0% | — | 0/2 | $0.03 |
11 | GPT-6 Astra | 0 / 2 | 0% | 0/2 | — | $1.97 |
11 | Gemini 3.1 Flash Lite | 0 / 2 | 0% | 0/1 | 0/1 | $0.0 |
Motions Decided Before Model Release: Accuracy Nearly Tripled
Not surprisingly, the models were better able to predict outcomes of motions to dismiss decided before their release dates. For these two motions, the LLMs were right 45% of the time. We used the same prompt, the same blind review and the same voting rules.
Case | Court | Nature of the case | Ruling | Predictions judged correct | Avg. cost per response |
|---|---|---|---|---|---|
Reporters Committee for Freedom of the Press v. Rokita | S.D. Ind. (Judge James R. Sweeney II) | News organizations brought a pre-enforcement First Amendment and vagueness challenge to Indiana's law letting police order people to stay back from officers. The state argued lack of standing and ripeness, collateral estoppel and the first-to-file rule. | Denied, September 2024 (Indiana Capital Chronicle) | 53% (105 of 198) | $0.31 |
Mattingly v. Marion Superior Court | S.D. Ind. (Judge James Patrick Hanlon) | A former Marion County magistrate brought Title VII sex and ADEA age discrimination claims. The defense argued that a magistrate is a policymaking appointee excluded from the statutes' definition of "employee," and that the Eleventh Amendment barred the ADEA claim. | Granted, 2023 (The Indiana Lawyer) | 37% (74 of 202) | $0.43 |
Pre-Release Date Overall Leaderboard
Rank | Model | Correct / voted | Accuracy | Rokita | Mattingly | Post-release accuracy | Avg. cost per response |
|---|---|---|---|---|---|---|---|
1 | GPT-5.6 Terra | 7 / 7 | 100% | 3/3 | 4/4 | 25% (1/4) | $0.21 |
1 | GPT-6 Astra | 7 / 7 | 100% | 3/3 | 4/4 | 17% (1/6) | $1.47 |
1 | GLM 5.3 | 4 / 4 | 100% | 1/1 | 3/3 | 0% (0/9) | $0.03 |
1 | Gemini 3.5 Flash Lite | 4 / 4 | 100% | 4/4 | — | 50% (3/6) | $0.04 |
5 | GPT-6.1 Sol | 21 / 24 | 88% | 10/11 | 11/13 | no votes | $0.28 |
6 | Gemini 3.1 Pro | 14 / 17 | 82% | 8/9 | 6/8 | 25% (13/51) | $0.45 |
7 | Grok-4.6 | 3 / 4 | 75% | 2/2 | 1/2 | 20% (3/15) | $0.43 |
8 | Gemini 3.6 Flash | 5 / 7 | 71% | 1/2 | 4/5 | 55% (6/11) | $0.19 |
9 | GPT-5.5 | 14 / 21 | 67% | 4/7 | 10/14 | 18% (7/40) | $0.86 |
9 | Claude Opus 5.5 | 4 / 6 | 67% | 4/5 | 0/1 | 6% (3/50) | $0.50 |
9 | DeepSeek V4 Pro 0813 | 2 / 3 | 67% | 2/2 | 0/1 | 0% (0/7) | $0.21 |
9 | GLM 5.1 | 2 / 3 | 67% | 1/1 | 1/2 | 7% (1/14) | $0.05 |
9 | Gemini 3.5 Flash | 2 / 3 | 67% | 1/2 | 1/1 | 63% (5/8) | $0.41 |
9 | Grok-4.3 | 2 / 3 | 67% | 0/1 | 2/2 | 17% (2/12) | $0.05 |
15 | Kimi K2.6 | 3 / 5 | 60% | 0/1 | 3/4 | 0% (0/8) | $0.30 |
15 | Kimi K3 | 3 / 5 | 60% | 3/3 | 0/2 | 0% (0/9) | $0.28 |
17 | Mimo V2.6 Flash | 4 / 8 | 50% | 1/3 | 3/5 | 14% (1/7) | $0.01 |
17 | DeepSeek V4 Flash | 3 / 6 | 50% | 2/5 | 1/1 | 25% (1/4) | $0.08 |
17 | DeepSeek V4 Flash 0731 | 3 / 6 | 50% | 3/4 | 0/2 | 11% (1/9) | $0.13 |
17 | GPT-6 Luna | 3 / 6 | 50% | 2/5 | 1/1 | 33% (2/6) | $0.01 |
17 | Grok-4.20 Non-Reasoning | 3 / 6 | 50% | 0/1 | 3/5 | 0% (0/9) | $0.04 |
17 | GLM 5.2 | 2 / 4 | 50% | 1/2 | 1/2 | 0% (0/5) | $0.07 |
17 | Gemini 3.7 Flash | 2 / 4 | 50% | 2/3 | 0/1 | 29% (2/7) | $0.14 |
17 | Grok-4.5 | 2 / 4 | 50% | 1/3 | 1/1 | 0% (0/4) | $0.22 |
17 | Minimax M3 | 2 / 4 | 50% | 2/3 | 0/1 | 20% (1/5) | $0.01 |
17 | Mistral Medium 3.5 | 2 / 4 | 50% | 0/1 | 2/3 | 38% (3/8) | $0.26 |
17 | Muse Spark 1.2 | 2 / 4 | 50% | 1/1 | 1/3 | 11% (1/9) | $0.22 |
17 | Qwen3.6 Max | 2 / 4 | 50% | 0/1 | 2/3 | 13% (1/8) | $0.24 |
17 | GPT-5.6 Luna | 1 / 2 | 50% | 1/1 | 0/1 | 0% (0/5) | $0.03 |
17 | Muse Spark 1.3 | 1 / 2 | 50% | 1/1 | 0/1 | 0% (0/6) | $0.71 |
31 | GPT-5.4 | 2 / 5 | 40% | 1/3 | 1/2 | 30% (3/10) | $0.30 |
31 | Inkling | 2 / 5 | 40% | 2/5 | — | 14% (1/7) | $0.63 |
31 | Minimax M2.5 | 2 / 5 | 40% | 0/1 | 2/4 | 10% (1/10) | $0.02 |
34 | Mimo V2.5 | 3 / 9 | 33% | 1/4 | 2/5 | 50% (1/2) | $0.01 |
34 | Gemini 3 Flash | 2 / 6 | 33% | 0/2 | 2/4 | 0% (0/9) | $0.03 |
34 | Mimo V2.6 Pro | 2 / 6 | 33% | 2/5 | 0/1 | 17% (1/6) | $0.04 |
34 | Claude Opus 4.8 | 1 / 3 | 33% | 1/2 | 0/1 | 38% (3/8) | $1.02 |
34 | Claude Sonnet 4.6 | 1 / 3 | 33% | 1/2 | 0/1 | 33% (2/6) | $0.54 |
34 | GLM 5.3 Flash | 1 / 3 | 33% | 1/2 | 0/1 | 0% (0/11) | $0.01 |
34 | GPT-5.6 Sol | 1 / 3 | 33% | 0/2 | 1/1 | 20% (1/5) | $0.36 |
34 | Mimo V2.5 Pro | 1 / 3 | 33% | 1/1 | 0/2 | 55% (6/11) | $0.07 |
42 | Claude Sonnet 5.5 | 18 / 55 | 33% | 18/29 | 0/26 | no votes | $0.24 |
43 | Claude Opus 4.6 | 2 / 7 | 29% | 2/4 | 0/3 | 14% (1/7) | $2.36 |
44 | Claude Fable 5.1 | 1 / 4 | 25% | 1/1 | 0/3 | 14% (1/7) | $3.76 |
44 | Claude Opus 5 | 1 / 4 | 25% | 1/2 | 0/2 | 20% (1/5) | $1.46 |
44 | Claude Sonnet 5 | 1 / 4 | 25% | 1/3 | 0/1 | 13% (1/8) | $0.62 |
44 | DeepSeek V4.1 Flash | 1 / 4 | 25% | 1/2 | 0/2 | 13% (1/8) | $0.02 |
44 | Minimax M2.5 HighSpeed | 1 / 4 | 25% | 0/1 | 1/3 | 10% (1/10) | $0.05 |
44 | Minimax M2.7 | 1 / 4 | 25% | 1/2 | 0/2 | 0% (0/7) | $0.02 |
50 | GPT-6 Sol | 1 / 5 | 20% | 1/2 | 0/3 | 25% (2/8) | $0.19 |
50 | Qwen3.7 Plus | 1 / 5 | 20% | 1/3 | 0/2 | 0% (0/5) | $0.05 |
52 | GLM 5v Turbo | 1 / 6 | 17% | 1/3 | 0/3 | 0% (0/9) | $0.13 |
52 | Minimax M2.7 HighSpeed | 1 / 6 | 17% | 1/2 | 0/4 | 22% (2/9) | $0.05 |
54 | Gemini 3.1 Flash Lite | 1 / 7 | 14% | 1/2 | 0/5 | 60% (3/5) | $0.01 |
55 | Grok-4.7 | 1 / 9 | 11% | 1/5 | 0/4 | 0% (0/5) | $0.50 |
56 | GLM 5 | 0 / 8 | 0% | 0/4 | 0/4 | 8% (1/13) | $0.15 |
56 | Grok-4.20 Reasoning | 0 / 5 | 0% | 0/4 | 0/1 | 17% (1/6) | $0.03 |
56 | Muse Spark 1.1 | 0 / 5 | 0% | 0/2 | 0/3 | 0% (0/7) | $0.13 |
56 | Gemini 3.8 Flash | 0 / 4 | 0% | — | 0/4 | 30% (3/10) | $0.56 |
56 | Qwen3.6 Plus | 0 / 4 | 0% | 0/2 | 0/2 | 0% (0/11) | $0.07 |
56 | Claude Fable 5 | 0 / 3 | 0% | 0/2 | 0/1 | 33% (2/6) | $2.03 |
56 | Claude Opus 4.7 | 0 / 3 | 0% | 0/1 | 0/2 | 0% (0/7) | $0.64 |
56 | DeepSeek V3.2 Thinking | 0 / 2 | 0% | — | 0/2 | 0% (0/6) | $0.21 |
56 | GLM 5 Turbo | 0 / 2 | 0% | — | 0/2 | 20% (1/5) | $0.18 |
56 | Kimi K2.5 | 0 / 2 | 0% | 0/2 | — | 0% (0/5) | <$0.01 |
*Claude Sonnet 5.5 was served up more than other models, appearing in 55 of 200 comparisons. This is so because it was newly released when run and the algorithm serves models with the fewest votes more frequently so it can rate new models quickly. Excluding Sonnet 5.5, which went 0 for 26 on Mattingly, would raise Mattingly's accuracy from 37% to 42% and the pre-release total from 45% to 47%. Thus, the change is negligible and does not change the overall conclusions.
What to Make of All of This
For legal teams: this shows that all things being equal, humans are going to make different decisions than large language models. The models are useful for organizing the record and analyzing arguments, but they leaned toward predicting survival. As noted above, judges often make judgement calls based on information that a litigator may not anticipate.
For AI labs: legal forecasting is a often a test of judgment, as opposed to knowledge. Answers from AI models are presumably objective, but are not yet proficient at weighing how judges actually rule against how legal doctrine reads.