New study finds emerging legal AI technologies improve quality, speed, and productivity for some legal tasks

Issue 22  

Two weeks ago this newsletter covered the Vals Legal AI Report (“VLAIR”), which was a first of its kind evaluation of four legal industry AI tools across seven legal tasks, which benchmarked the results against a lawyer control group. This issue covers another new study contributing to the growing body of independent analysis of legal AI tools.  

A new Minnesota Legal Studies Research Paper tested a retrieval augmented generation (“RAG”)-tuned AI tool (Vincent AI) and an AI reasoning model (OpenAI’s o1-preview) on six legal tasks, finding that both AI tools significantly enhanced the quality of the legal work compared to the legal work performed without AI in four out of six tasks.[i] Additionally, the study found that both AI tools significantly boosted productivity in five out of six legal tasks, with particular power in tasks like analyzing complaints and drafting persuasive letters.[ii] 

The AI Technology 

• Reasoning Models: The study defined AI reasoning models as models that use computing resources to plan responses before generating them, similar to how a human takes longer to think and outline a response before answering a complex question.[iii] OpenAI has adopted a new naming convention for its reasoning models, beginning with “o1”.[iv]  

• RAG: The study explained RAG as a technique that integrates generative AI with authoritative sources, such as statutes and case law, to enhance accuracy.[v]  

• Automated/Embedded Prompting: In addition to reasoning models and RAG, the study identified a third notable advance in legal AI technology: the development of automated or embedded prompting.[vi] Vincent AI utilizes automated prompting.[vii] The study noted that legal AI tools are increasingly automating legal prompt engineering in order to improve the effectiveness of questions and responses.[viii] I plan to expand on this topic in a future issue of the newsletter. 

The Tasks 

127 law students from the University of Minnesota and University of Michigan law schools participated in the study, with each participant completing six tasks, two without AI assistance, two with the assistance of reasoning model OpenAI 01-preview, and two with the assistance of RAG-tuned Vincent AI.[ix] Participants were divided into three groups so that, for example, approximately one-third of participants completed task one with each AI tool, and one-third of participants completed that task without AI.[x] The tasks were as follows: 

• Task One: Draft an email for a client;  

• Task Two: Draft a legal memo for a partner;  
• Task Three: Analyze a complaint and draft a written analysis; 
• Task Four: Draft a non-disclosure agreement for a client; 
• Task Five: Draft a motion to consolidate; and 
• Task Six: Draft a persuasive letter addressing the enforceability of a covenant not to compete.[xi]  

The tasks were graded on five criteria: accuracy, analysis, organization, clarity, and professionalism.[xii]  

The Findings 

The study found that both AI tools improved the quality of work for three tasks: task two (legal memo), task three (complaint analysis), and task five (motion), while OpenAI o1-preview improved the quality of work for task six (persuasive letter), and Vincent AI improved the quality of work for task one (client email).[xiii]  

While both tools were found to significantly enhance clarity, organization, and professionalism across multiple tasks, OpenAI o1-preview was found to consistently outperform Vincent AI in the frequency, magnitude, and statistical significance of the improvements.[xiv] Further, only OpenAI o1-preview demonstrated statistically significant improvements in legal analysis in some of the tasks.[xv]  

Further, the study found that both AI tools demonstrated statistically significant and substantial speed and productivity improvements in the completion of five out of six tasks.[xvi] 

Neither tool demonstrated a statistically significant improvement in quality or reduction in completion time for task four, drafting a non-disclosure agreement.[xvii] The study noted that it was the only transactional task, as well as the only task where participants were provided a general template to use in their response, which may have reduced the potential for AI-driven quality improvement.[xviii]  

Neither tool consistently showed statistically significant improvements in accuracy.[xix] While the use of both tools produced hallucinations, the study suggested that RAG technology did reduce hallucinations.[xx] 

Notably, the study suggested that because reasoning models and RAG appear to enhance legal work through separate mechanisms, the combination of both technology advancements in a single AI tool (which is already happening in the marketplace of legal industry AI tools) could lead to synergistic advancements.[xxi]  

To learn more, you can download the study here. 

Takeaways for Lawyers Who are Evaluating Their AI Options  

VLAIR and the Minnesota Legal Studies Research Paper both support the idea that AI can enhance a lawyer’s work product on some tasks. What’s more, the Minnesota Legal Studies Paper claims that AI may also improve the speed at which lawyers can complete some tasks, and also raises the possibility that integrating RAG with a reasoning model could yield additive or possibly even multiplicative benefits for users.[xxii]  

This should not be taken as a directive to lawyers to Frankenstein multiple AI tools together in search of those potential multiplicative benefits. As acknowledged by the study, the marketplace of legal industry AI tools is already at work on this problem. Lawyers with a particular interest in this area may wish to spend time building custom solutions for their practices, but there are plenty of opportunities for lawyers to engage with AI by using tools that have already been customized for the legal industry.  

Lawyers who are ready to identify the AI solutions that can make the greatest impact for their organizations should start by clarifying and prioritizing the problems they need to solve with AI. This requires investigating your organization’s technology problems. Where is technology currently serving the people of your organization well, and where is there room for improvement? Is there work performed in your organization that routinely gets written off? What tasks are repetitive? What tasks can be streamlined? What work could be performed more consistently and accurately with technology? Where would a new technology tool make the biggest financial impact? How receptive are the people of your organization to new technology?   

Once you understand your organization’s technology problems, you’ll be in a better position to match those problems with the solutions currently available from AI tools. This is also the point in the AI tool evaluation process where benchmarking reports like VLAIR and studies like the Minnesota Legal Studies Research Paper can help guide decision making. However, considering that there are over 50 use cases for legal industry AI tools, and hundreds of legal industry AI tools on the market, it’s important to recognize that the vast majority of AI tools for lawyers are unlikely to be included in independent benchmarking studies and evaluations in the near future. This is a reality of navigating the AI era, where new developments are happening constantly, and it’s important to recognize that the AI solution that could be most impactful to your organization may not have been included in the benchmarks or studies released to date. 

However, lawyers who are interested in AI tools that lack independent benchmarking data or studies can still conduct their own evaluations and testing of the AI tools that they have identified as being most promising for their unique organizations. Chapter 5 of A Lawyer’s Practical Guide to AI lays out a process to help you identify whether there are AI tools that potentially meet your organization’s needs, and if so, how to evaluate them before implementing them.  Once you know what you want an AI tool to do, you can use the directory of AI tools for lawyers in Chapter 6 of A Lawyer's Practical Guide to AI as a starting point to quickly narrow down the available options and select one or more tools to evaluate further for compatibility with your organization.  

Thanks for being here. 

Jennifer Ballard 

Good Journey Consulting 

P.S. In case you missed my post on LinkedIn earlier this week, on March 18^th the U.S. Court of Appeals for the D.C. Circuit issued an opinion in Stephen Thaler v. Shira Perlmutter et al., No. 23-5233 (D.C. Cir. filed Oct. 11, 2023), affirming the denial of plaintiff’s copyright application for a piece of AI-generated art, finding that the Copyright Act “requires all eligible work to be authored in the first instance by a human being.” Opinion at 3, Stephen Thaler v. Shira Perlmutter et al., No. 23-5233 (D.C. Cir. filed Oct. 11, 2023). Click here to read more.

____________________________________________________

[i] Schwarcz, Daniel and Manning, Sam and Barry, Patrick James and Cleveland, David R. and Prescott, J.J. and Rich, Beverly, AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice (March 02, 2025). Minnesota Legal Studies Research Paper No. 25-16 at 1-2, 7, Available at SSRN: https://ssrn.com/abstract=5162111 or http://dx.doi.org/10.2139/ssrn.5162111. 

[ii] Id. at 2. 

[iii] Id. at 5. 

[iv] Id. at 5. 

[v] Id. at 5-6. 

[vi] Id. at 19. 

[vii] Id. at 21. 

[viii] Id. at 19. 

[ix] Id. at 6. 

[x] Id. at 25. 

[xi] Id. at 26-27. 

[xii] Id. at 29. 

[xiii] Id. at 32. 

[xiv] Id. at 39. 

[xv] Id. 

[xvi] Id. at 41, 47. 

[xvii] Id. at 36, 45. 

[xviii] Id. 

[xix] Id. at 40. 

[xx] Id. 

[xxi] Id. at 2, 57. 

[xxii] Id. at 57.