How Do You Know if An AI Tool Is Actually Good at Legal Work?

How Do You Know if An AI Tool Is Actually Good at Legal Work?

Every AI vendor says their tool is good at legal work. A large law firm came to Synaptiq with a harder question: good at our work, measured how, and compared with what? Public leaderboards couldn’t answer it, and neither could vendor demos. The firm needed a way to measure any AI system on its own tasks, under conditions it controls, and get numbers it could trust.

In this webinar, Synaptiq’s team walks through what we built for them: an evaluation bench on Databricks. You describe in plain English what you want to measure, such as “how well does this model extract liability caps and change-of-control clauses from contracts?” The bench finds suitable data and checks whether the metrics can actually be computed from it. Then it either scores the run or tells you what’s missing and what it would cost to fill the gap.

We’ll cover:

• The why: why leaderboard scores and vendor claims don’t answer the questions a law firm actually asks

• The approach: how we worked with the firm to define what “measurable” means. Datasets provide certain kinds of ground truth and metrics require certain kinds, and that simple rule keeps the bench from producing flattering numbers.

• The demo: about 71,000 public legal documents and 85,000 annotations governed in Unity Catalog, models called through Databricks’ model gateway, and every run, including failed ones, logged back to Delta tables so every score can be traced to the items behind it

• The lessons: benchmark contamination, how a single accuracy number can hide the errors a lawyer cares about, and why a “relevant-looking” dataset often can’t support the metric you want