Reliability of AI legal research tools: measured error rates and two court decisions

Evidence Research Format (ERF) v0.1 · Legal · first edition

Research status: AI-assisted draft Claims checked against published abstracts; details from full papers and court opinions taken by automated extraction and marked as pending manual review. Not reviewed by a qualified lawyer.
Last verified: 2026-10-24
Scope: Published evaluations of named AI tools, and two U.S. court decisions (Mata v. Avianca; Park v. Kim), checked against copies of the opinion text by automated extraction. Every error rate applies only to the tools, versions, questions and test dates of its study.
Format: ERF v0.1 (SYCDS)

Not legal advice. This page summarises published research and court decisions for decision analysis. It does not interpret the rules of any jurisdiction for a specific situation and does not rank or recommend any product. Consult a qualified lawyer for advice.

Research Question

What do published evaluations show about how often specific AI legal research tools produce incorrect or unsupported legal information, and how have courts responded when lawyers filed AI-generated fictitious citations?

Key Findings

Specialised tools reduced but did not remove errors, in the tested versions. In a pre-registered study of three legal research tools tested in March–May 2024, the authors report hallucination rates between 17% and 33% of queries (abstract). Hallucination, accuracy and incomplete answers are separate measures with their own definitions (see E2). [E1, E2, E3]
General-purpose chatbots did worse on direct case questions, in 2023-era models. On verifiable questions about U.S. federal cases, four public models hallucinated at least 58% of the time (abstract); the introduction gives a range up to 88%. [E5]
Numbers do not generalise. Each rate belongs to one study’s tools, versions, questions, scoring and test dates. A 2025 benchmark of other tools reported accuracy around 80% using different tools and scoring; it is not comparable. [E4, E9, C1]
In the two decisions analysed here, the courts acted on unverified citations, not on the use of AI. In Mata v. Avianca the court said nothing is “inherently improper” about using a reliable AI tool and sanctioned conduct that included false statements after the problem was raised; in Park v. Kim the appeals court stressed the duty to read and confirm cited authority. These two U.S. decisions do not establish how other courts act. [E6, E7, E8]
Assessment. The evidence supports verifying every citation against an authoritative source, whatever tool is used. It does not support a single error rate for “AI legal research”, and it does not show how courts in general respond to AI use.

Evidence Table

#	Claim	Evidence	Source	Check
E1	Three specialised legal AI tools hallucinated on a material share of test queries.	Journal abstract: the LexisNexis and Thomson Reuters tools “each hallucinate between 17% and 33% of the time”, in a pre-registered evaluation.	[R1]	Abstract-checked
E2	The study uses three separate outcome measures.	Hallucination: a response that is incorrect, or misgrounded (the cited source does not support the claim). Accurate: correct and grounded. Incomplete: a refusal or an ungrounded response. Headline hallucination rates use all queries as the denominator; rates among responses that gave a direct answer are reported separately. Over 200 queries (80 general research, 70 jurisdiction- or time-specific, 22 false-premise, 30 factual recall). Tested 22 Mar–22 Apr 2024; the Westlaw tool 23–27 May 2024.	[R1]	Partly verified main text, automated extraction; manual review pending
E3	The three measures differed widely between tools.	Hallucination (all queries): Lexis+ AI and Ask Practical Law AI over 1 in 6; Westlaw AI-Assisted Research about one-third. Accurate: Lexis+ AI 65%; Westlaw 42% (introduction) or 41% (section 6.1 — inconsistent within the paper); Ask Practical Law AI 19%. Incomplete (including refusals): Ask Practical Law AI 62%. Exact figures for each measure are in the paper’s figures, which were not read.	[R1]	Partly verified main text, automated extraction; manual review pending
E4	An earlier version of the same study differs.	The 2024 preprint did not evaluate the Westlaw tool (access was not available); figures on this page follow the 2025 journal version.	[R1]	Partly verified preprint text, automated extraction
E5	General-purpose models hallucinated most of the time on direct case questions.	Abstract: LLMs “hallucinate at least 58% of the time”. Introduction: between 58% (ChatGPT 4) and 88% (Llama 2) on verifiable questions about randomly selected federal cases. Models: ChatGPT 4, ChatGPT 3.5, PaLM 2, Llama 2. Measure: hallucination only (no separate accuracy or refusal rate reported here).	[R2]	Abstract-checked 58% figure; 88% upper bound from the introduction, automated extraction
E6	Mata v. Avianca: sanctions for fictitious citations and later false statements.	A filing of 1 March 2023 cited decisions that did not exist (six fake decisions are named in the order). Bad faith found on the basis of “conscious avoidance” and false and misleading statements after the problem was raised. Order: $5,000 penalty, jointly and severally, on two lawyers and their firm, under Rule 11 or alternatively the court’s inherent authority; letters to the client and to each judge falsely named as an author.	[R3]	Partly verified opinion text, automated extraction; manual review pending
E7	Mata v. Avianca: using AI is not itself improper.	“There is nothing inherently improper about using a reliable artificial intelligence tool for assistance”; existing rules make lawyers gatekeepers of the accuracy of their filings.	[R3]	Partly verified opinion text, automated extraction; manual review pending
E8	Park v. Kim: referral for a nonexistent case suggested by ChatGPT.	Second Circuit, per curiam, 30 Jan 2024, affirming a dismissal for discovery failures. The reply brief cited a case the court could not locate; counsel said ChatGPT suggested it. Rule 11 requires attorneys to “read, and thereby confirm the existence and validity of,” cited authority. Counsel referred to the court’s Grievance Panel (Local Rule 46.2) and ordered to give the client a copy of the ruling; no monetary sanction.	[R4]	Partly verified opinion text, automated extraction; manual review pending
E9	A 2025 benchmark of other tools reported higher accuracy.	210 research questions: Alexi 80%, Counsel Stack 81%, Midpage 79%, ChatGPT 80%, lawyer baseline 71%. Weighted score (accuracy 50%, authoritativeness 40%, appropriateness 10%) — a different measure from E2. Thomson Reuters and LexisNexis did not take part.	[R5]	Partly verified secondary report only
E10	In Mata and Park, the courts acted on unverified citations, not on AI use as such.	Judgment based on E6–E8. Limited to these two U.S. decisions; no other decision is analysed on this page.	—	Interpretation

Abstract-checked checked against the published abstract only · Partly verified checked by automated extraction from the full text or opinion, or via a secondary report; manual review pending · Interpretation judgment, not a checkable fact
No claim on this page is full-text-checked or independently reviewed.

Evidence Limitations

How this page was checked: abstracts of R1 and R2 were checked. Details from the full papers and from the two court opinions were obtained by automated extraction; manual review is pending and the labels say so.
Every error rate is tied to a test date. Legal AI products change frequently; the authors of R1 observed answers changing during the study, and at least one tool was replaced by a newer version after testing.
The question sets were designed to probe weaknesses (including false-premise questions). They are not a representative sample of everyday legal research, so they do not measure how often practitioners meet errors.
Manual coding: inter-rater agreement in R1 was Cohen’s kappa 0.77 on a 48-response sample — good, not perfect.
Court decisions record cases that were caught. Their number cannot be turned into an error rate, because the number of filings that used AI is unknown and undetected errors leave no record.
Only two decisions (both U.S.) are analysed here. Further reported decisions in the U.S. and in England and Wales (for example R (Ayinde) v London Borough of Haringey [2025] EWHC 1383 (Admin)) were identified from secondary reports and will be analysed only after the original rulings are read [R6].
Professional ethics guidance (for example ABA Formal Opinion 512, 29 July 2024) has been identified but not read in full; it is not used for any finding here.

Counterevidence

C1 — Some tools scored higher than lawyers on one benchmark. In a 2025 benchmark, several legal AI tools and ChatGPT scored about 80% against a lawyer baseline of 71% (E9). Different tools, questions and scoring from R1; participation was voluntary and the two largest vendors did not take part.
C2 — Retrieval-based legal tools hallucinated less than a general-purpose model (GPT-4) in the same study [R1], so “AI legal research is unreliable” is too broad.
C3 — In Mata, the sanctioned conduct went beyond using AI. The bad-faith finding rested on conscious avoidance and false statements after the problem was raised (E6).

Decision Implications

For choosing or governing an AI research tool, the evidence supports asking vendors for independent evaluations of the specific version and date, and for the hallucination, accuracy and refusal rates separately, rather than relying on claims such as “hallucination-free” (R1 found such claims overstated).
For internal procedure, the evidence supports a non-optional step: every cited authority is checked against an authoritative source before filing. Both decisions analysed here turned on unverified citations (E6–E8).
The evidence supports distinguishing specialised retrieval-based tools from general chatbots; their measured hallucination rates differ (E1, E5).
The evidence does not support applying 17–33% or 58–88% to tools or versions not tested, treating those figures as current, or comparing them with the accuracy scores in E9.
The evidence does not support statements about how courts in general respond to AI use; only two U.S. decisions are analysed here.

What Would Change the Assessment

Independent, pre-registered re-testing of current versions of the major legal research tools showing materially lower or higher error rates.
Participation of the major vendors in public benchmarks.
Appellate guidance or rule changes that set a common standard for AI-related citation errors, for example disclosure requirements.
A study that samples real filings, giving an error rate with a known denominator.
Reading the further decisions and ethics opinions in the original (planned), or review by a qualified lawyer, which would change the scope and research status above.

Discuss with Tac Yang

Use Tac Yang to test a decision that depends on this evidence. Tac does not verify sources; it asks what your decision rests on and what is still missing.

Last Verified / Update History

2026-10-24 — Revised after content review (REVISE): court-opinion and main-text details relabelled “Partly verified — automated extraction; manual review pending” instead of “Abstract-checked”; hallucination, accuracy, incomplete/refusal rates and their denominators separated; conclusions about courts limited to the two decisions analysed (Mata, Park).
2026-10-24 — Correction before publication: the research outline stated that the court in Mata ordered continuing legal education (CLE). That came from a headnote and was wrong; the opinion text shows the firm arranged CLE itself and the court found further mandatory education redundant. The page makes no claim about CLE.
2026-10-24 — First edition (ERF v0.1). Two court decisions checked against copies of the opinion text by automated extraction; other decisions listed as pending. Last verified: 2026-10-24.

Sources

[R1] Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. J Empirical Legal Stud. 2025;22(2):216–242. doi:10.1111/jels.12413. Earlier preprint: arXiv:2405.20362. Link unconfirmed automatic link check returned HTTP 403 for the DOI link; pending manual check in a browser.
[R2] Dahl M, Magesh V, Suzgun M, Ho DE. Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. J Legal Analysis. 2024;16(1):64–93. doi:10.1093/jla/laae003 Link unconfirmed automatic link check returned HTTP 403; pending manual check in a browser.
[R3] Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), No. 22-cv-1461 (PKC), Opinion and Order on Sanctions, 22 June 2023 (Castel, J.). Copy of opinion: national.clla.org.
[R4] Park v. Kim, No. 22-2057 (2d Cir. 30 Jan 2024) (per curiam). Opinion text: caselaw.findlaw.com. Reporter citation not checked. Link unconfirmed automatic link check returned HTTP 403; pending manual check in a browser.
[R5] Vals AI, VLAIR – Legal Research (October 2025), as reported by LawSites, 23 Oct 2025: lawnext.com — secondary report; the original report was not opened.
[R6] R (Ayinde) v London Borough of Haringey; Al-Haroun v Qatar National Bank QPSC [2025] EWHC 1383 (Admin), 6 June 2025. Judgment record: judiciary.uk. Citation and date checked; the judgment text has not been read and is not used for any finding.

SYCDS Evidence Research · ERF v0.1 · Legal · AI-assisted draft, not professionally reviewed · Not legal advice