Computer scientists build AI tool to spot risky and unenforceable contract terms

(Press-News.org) Contracts written by employers and landlords often result in second parties—employees and tenants—facing unfair terms because these documents contain unreasonable or ambiguous clauses, leaving the second parties vulnerable to unjust expenses or constraints.

For example, “Tenant must provide written notice of intent to vacate at a reasonable time”—commonly used phrasing in leases—is ambiguous because “reasonable” is undefined. Also, “Employee agrees not to work for any business in the United States for two years following termination,” often included in employee contracts, is unenforceable because many states prohibit broad non-compete agreements. 

To better spot these problematic passages, a team of New York University researchers has created a tool that deploys large language models (LLMs) to analyze contractual agreements and characterize clauses across four categories: missing clauses, unenforceable clauses, legally sound clauses, and legal but risky clauses, identifying the latter as “high risk,” “medium risk,” or “low risk.”

The creators of the tool, ContractNerd, see it as a useful platform for both drafters and signing parties to navigate complex contracts by spotting potential legal risks and disputes. 

“Many of us have to read and decide whether or not to sign contracts, but few of us have the legal training to understand them properly,” says Dennis Shasha, Silver Professor of Computer Science at New York University’s Courant Institute of Mathematical Sciences and the senior author of the research, which appears in the journal MDPI Electronics. “ContractNerd is an AI system that analyzes contracts for clauses that are missing, are extremely biased, are often illegal, or are ambiguous—and will suggest improvements to them.” 

ContractNerd, which analyzes leases and employment contracts in New York City and Chicago, draws from several sources in spotting contractual risk that is “high,” “medium,” and “low.” These sources include Thomson Reuters Westlaw; Justia, a reference for standard rental-agreement language; and Agile Legal, a comprehensive library of legal clauses. It also takes into account state regulations.

To evaluate the tool’s effectiveness, the creators used a series of methods that compared ContractNerd with existing AI systems that analyze contracts. The first comparison showed that ContractNerd yielded the highest scores among these systems based on how accurately each predicted which clauses would be deemed unenforceable in legal cases. 

In the second, an independent panel of laypersons evaluated the output of ContractNerd and  the best of the other systems from the first comparison—goHeather—based on the following criteria:

Relevance: How directly the analysis addressed the content and intent of the clause

Accuracy: Whether the legal references and interpretations were factually and legally correct

Completeness: Whether the analysis covered all significant legal and contextual aspects

For each clause, the reviewers, who were blinded to the actual names of the tools in order to control for bias, indicated which system—“System A” and “System B”—produced the better output. Overall, ContractNerd received the better ratings.

In the third, the creators received the help of NYU School of Law Professor Clayton Gillette, an expert in contracts law, to offer qualitative assessments of both systems using the same criteria. These included analyzing outputs for simple contract clauses, such as “No pets allowed,” to more complicated ones, such as “Tenant shall be responsible for all attorney fees incurred due to breach of this lease agreement.”

In general, Gillette found ContractNerd to be more thorough in its outputs, but saw GoHeather’s analysis easier to comprehend.

“Contracts are of course about law, but they should also be fair to both parties,” says Shasha, who plans on expanding the geographic reach of the tool. “We see ContractNerd as an aid that can help guide users in determining if a contract is both legal and fair, potentially heading off both risky agreements and future legal disputes.”

The paper’s other authors were Musonda Sinkala and Yuge Duan, NYU graduate students at the time the prototype was built, and Haowen Yuan, an NYU undergraduate.

# # #

END