Analysis, insights, and updates on legal AI evaluation, benchmarks, and research.
Low accuracy on new rulings: Across 65 LLMs tested on motions to dismiss decided after model release dates, only 16% of predictions accurately matched the judges' rulings (and that number is probably too high).
Read article
We measured token costs for legal tasks of varying difficulty across most available LLMs.
The Certera Legal AI Leaderboard ranks how AI models perform on legal work. It is based on votes cast by legal professionals who run a prompt, compare responses from different LLMs side by side, and pick the best one.
There is no question that Generative AI is transforming how legal professionals draft legal documents and perform legal research, but it comes with a well-documented risk: AI hallucinations.
To evaluate Large Language Models (LLMs) for legal work, traditional AI benchmarks are often used. While these benchmarks are helpful to be sure, do they tell the whole story? A model might score 90% on a multiple-choice bar exam simulation, but struggle in a multi-task legal prompt or an agentic-based legal workflow.
The legal industry’s relationship with artificial intelligence is seemingly at an inflection point. Despite undeniable advances in not just AI for legal, but AI generally, there is still much debate and reticence in the legal community about using AI.
