Why not ChatGPT?

IN CASE YOU MISSED IT: A Lawyer’s Practical Guide to AI is now available for immediate access to purchasers! You can check the guide out here.

Issue 4 

This week I’m going to answer a great question: Why does A Lawyer’s Practical Guide to AI focus on legal industry AI tools and exclude popular general-purpose AI tools like ChatGPT? 

When you evaluate which AI tools would be best for your law practice, it makes sense to focus first on the tools that humans have put thought and care into developing with lawyers’ needs in mind. Many of the generative AI tools created for the legal industry are built by using one of the large language models like OpenAI’s GPT models, Anthropic’s Claude models, and similar. In such cases, the legal industry AI tool company “tunes” that large language model to better perform in its intended niche.  

If there is an AI tool that has been developed specifically to perform a task you want it to do, logically, there’s a good chance it’s going to outperform a general-purpose tool at that task. Here’s an example of this: Stanford RegLab and the Institute for Human-Centered AI conducted a study in 2024 where they tested three AI tools tuned for legal research: Westlaw’s AI-Assisted Research, Ask Practical Law AI (both Thomson Reuters products), and Lexis+ AI, along with GPT-4 (which was the GPT model used at the time for ChatGPT). While the study found that all of the tools generated some inaccurate results, the tools that were tuned for legal research outperformed GPT-4.[1] 

Is it Wrong for Lawyers to Use ChatGPT? 

Some lawyers are using ChatGPT or Claude or Gemini or another large language model for legal work, and it’s not wrong to do so, so long as they take care to only use AI in ways that are aligned with their professional responsibilities. Among other considerations, the ABA has advised lawyers who use AI to be mindful of their duties of competency and confidentiality. General-purpose AI tools may typically be more financially accessible than most of the legal industry AI tools. And it’s possible that some lawyers have devised clever ways to use general-purpose AI tools for use cases that don’t have legal industry AI tool solutions.  

While a general-purpose AI tool could potentially be the solution that makes the most sense for your unique practice, I do not recommend that lawyers start and end their searches for AI solutions with general-purpose AI tools. Doing so would skip a few necessary steps to validate that you will get the most out of your AI solutions.  

How to Identify Whether an AI Tool is Right for You 

There are already a lot of AI tool options available for lawyers. I’ve identified over 150 AI tools made for the legal industry that collectively offer over 50 use cases. If you’re going to go to the effort of adopting an AI tool, I want you to be able to efficiently find the best tool options for your practice. And if at all possible, I want you to avoid adopting an AI tool you’re not impressed with and having to start over.  

In order to get it right the first time, you’ll want to start by developing your AI competency if you haven’t done so already. That way you’ll verify you understand your professional responsibilities in relation to AI and recognize the AI pitfalls to avoid.  

Next, you’ll want to consider where AI can make the biggest impact on your organization. This will involve an analysis of your organization’s existing technology resources and areas for improvement. Once you know what you want an AI tool to do, you can identify AI tools that best meet those needs. General-purpose AI tools could be considered alongside AI tools tuned for the legal industry at this point in the evaluation.  

Once you know which tools are most likely to meet your needs, you can evaluate the risks associated with those tools, consider how compatible each tool is with your organization, and test the tools before making a selection. If you would like more in-depth help figuring out how to explore using AI in your practice, I have a detailed step-by-step process for evaluating and implementing AI tools in Chapter 5 of A Lawyer’s Practical Guide to AI. You can learn more about the guide here. 

If you already selected an AI tool and you’re not impressed with the results, it’s not too late to evaluate whether there might be a different tool that would be a better fit for your organization. I’ll write more about this in an upcoming issue of the newsletter. 

Thanks for being here. 

Jennifer
Good Journey Consulting

[1] Magesh, Varun; Surani, Faiz; Dahl, Matthew; Suzgun, Mirac; Manning, Christopher D; Ho, Daniel E (2024): Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, at 13, Stanford RegLab. Preprint. https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Hallucinations.pdf.