The legal industry’s relationship with artificial intelligence is seemingly at an inflection point. Despite undeniable advances in not just AI for legal, but AI generally, there is still much debate and reticence in the legal community about using AI.
By their very nature, those in the legal have always been a skeptical bunch, and, as a result, trust is the main roadblock to technology adoption. For legaI AI that means trusting that AI understands the prompt entered, trusting that data is being handled properly and that the response is accurate.
To date, when legal teams evaluate legal AI software they often must depend on LLM benchmarks. To be sure, these are a useful and needed way to evaluate the accuracy of artificial intelligence, but the question remains: how accurately does model performance on an AI benchmark really reflect how deeply a model understands legal nuance, it’s “legal intuition”, or more importantly, what type of legal outputs attorneys and their allied professionals really prefer in the real world?
This very question is the reason we built Certera.
What is Certera.ai?
Certera.ai is a blind, side-by-side LLM comparison platform built specifically for legal professionals. It applies the concept of blind A/B testing of two anonymized LLM model outputs from legal prompts generated by real lawyers and legal professionals.
Instead of guessing which open-source AI or closed AI model handles specific legal queries best, Certera lets you test multiple AI models against each other in real time without revealing the models’ identity until after the user identifies the better output.
How It Works
The legal prompt evaluation is built on a three-step process:
Enter a Legal AI Prompt:
Legal professionals enter a real legal related prompt. Before it is sent to any model, users can use the built in data anonymization tool to remove confidential and sensitive information.
Review the Blind Outputs:
The query is simultaneously sent to two separate LLMs. In anonymous mode, models are randomly selected. Or, if the user prefers, two specific models may be selected to compare against each other. Answers populate side-by-side so the user can review and determine which is the preferred output.
Vote:
Users then select the preferred response based on legal accuracy, structure, and legal reasoning. After the user selects the best response, the vote is recorded and the models are revealed.
The Dynamic Legal LLM Leaderboard
Every vote cast by a verified legal professional feeds directly into our Legal AI Leaderboard.
Static LLM benchmarks can be artificially gamed if models are trained directly on evaluation data and they rapidly lose utility the moment a new model or iteration is released. By crowdsourcing ground-truth preferences from actual lawyers and allied legal professionals, Certera provides real-time feedback of which models truly lead in legal domain-specific tasks, and what it really costs to use them.
Why Percipient Built Certera
For over a decade, we have been using technology to analyze data with humans verifying the output. That is exactly the workflow needed to improve not just AI for law for AI output for any discipline.
As a result, we are using our unique position in the legal industry to further augment our technology first mission by helping to ensure the legal tech of the future, which is undoubtedly AI based, is the best it can be. Verifying the true accuracy of AI for lawyers and other legal professionals is not best determined by a company’s valuation or marketing hype. It is determined by the precision of the text and analysis on the page.