Artificial intelligence (AI) is making waves across industries, and the legal profession is no exception. But as Thomas v. Commissioner recently demonstrated, AI’s convenience can come at a cost. This U.S. Tax Court case offers a cautionary tale about the dangers of relying too heavily on AI-generated content without human oversight—and shows just how quickly things can go awry.
The Problem: A Pretrial Memorandum Gone Wrong
When Gary Thomas challenged the IRS’s decision to disallow certain unreimbursed employee business expenses on his 2016 and 2017 tax returns, his lawyer submitted a Pretrial Memorandum to lay out the case’s key facts and legal arguments. Everything seemed in order—until the judge took a closer look.
The memorandum cited legal cases to support the petitioner’s arguments. But something strange caught the judge’s eye; several of the referenced cases didn’t exist, and other cited cases were real but entirely unrelated to the argued legal points.
This discrepancy raised an alarming question: was this a simple oversight, or had AI been used irresponsibly?
The Lawyer’s Response: AI or Human Error?
In a hearing called to clarify the situation, the lawyer confessed that she had relied on a newly hired paralegal to prepare the document and, more troublingly, had not reviewed it before filing it with the court. She insisted she hadn’t personally used AI but admitted she didn’t know whether her paralegal had turned to an AI tool for assistance.
This uncertainty set off alarm bells. The judge speculated that the memorandum bore the telltale signs of AI-generated content, possibly from a large language model (LLM) like ChatGPT. The case had all the hallmarks of what’s become known as an “AI hallucination,” where an LLM generates convincing but entirely inaccurate text based on patterns from its training data.
The Court’s Warning: The Risk of AI Hallucinations
AI tools, especially large language models, can feel like magic—they analyze vast datasets to produce coherent text that mimics human reasoning. But their outputs aren’t always reliable. In fact, these models can produce what appear to be well-reasoned legal arguments but contain factual or legal errors, known as hallucinations.
The court referred to recent warnings from Chief Justice John Roberts, who highlighted the risks AI poses in legal practice. “AI has the potential to increase access to justice,” Roberts noted in his 2023 Year-End Report on the Federal Judiciary. “But it also carries risks. AI hallucinations have led lawyers to submit briefs citing non-existent cases.” That’s exactly what seemed to have happened here.
The Legal Standard: The Lawyer is Accountable
The court pointed to Tax Court Rule 33, which requires that lawyers certify their filings as accurate and grounded in law. By signing the memorandum, the lawyer essentially vouched for its content—even though she hadn’t reviewed it. Rule 33 also places responsibility on lawyers to supervise their staff. This meant that even if the paralegal had used AI improperly, the ultimate responsibility lay with the attorney.
Despite the errors, the court acknowledged that the memorandum, while inaccurate in its citations, did provide a generally correct summary of the relevant tax law.
The Court’s Ruling: A Symbolic Sanction
In the end, the judge opted for a light, symbolic sanction. The court struck the Pretrial Memorandum from the record, but this didn’t affect the outcome of the case. The facts and legal issues were clear enough to proceed without it.
The court also took into account the financial reality faced by Mr. Thomas. With only $10,000 at stake and an income of just over $50,000, he likely didn’t qualify for free legal assistance but also couldn’t easily afford top-tier legal services. Given these circumstances, the court chose not to impose a monetary penalty, acknowledging the lawyer’s efforts to serve a client who might otherwise have gone unrepresented.
Lessons Learned: Proceed with Caution
The Thomas v. Commissioner case serves as a timely reminder: AI can be a powerful tool for lawyers and tax professionals, but it’s not a replacement for human judgment. Just as predictive coding tools were once scrutinized in the legal field before being cautiously adopted, AI tools must be handled with care.
The court emphasized that it isn’t in the business of dictating how lawyers use new technologies but urged caution when integrating AI into legal and technical practice. Lawyers must verify the outputs AI tools produce, especially when lives, businesses, and legal outcomes are at stake.
The Future of AI in Law
AI offers incredible potential, especially for small firms and clients with limited resources. It can reduce costs and streamline workflows, making legal help more accessible to those who need it. But the Thomas case underscores the importance of caution. AI is a tool, not a substitute for careful, ethical legal and accounting work.
As AI continues to evolve, lawyers and tax professionals will need to strike the right balance between embracing technology and maintaining rigorous oversight. The court’s message is clear: Use AI wisely—but never forget that the final responsibility rests with the humans behind the work.
Financial Guardians is a proud member of InCite, the recently launched online community exclusively for tax professionals, bookkeepers, and accountants. InCite members receive a 30% discount.
