• 14 June 2009

Automating Contract Law: How Advances in Knowledge-Management Technology Can Help Transform the Empirical Study of Contract Law

George S. Geis - University of Virginia School of Law

Posted in , , , , , , , ,

Early one morning in January 1956, Herbert Simon announced to his graduate class at Carnegie Mellon that “[o]ver Christmas, Al Newell and I invented a thinking machine.”1  His claim was a bit premature, but Simon did win the Nobel Prize in Economics twenty-two years later—not for creating a sentient computer but rather for his research on how economic organizations make decisions.  Simon’s main contention was that all decisions are made under uncertainty because it is impossible to gather and process every bit of information that matters.

Despite his acknowledgment of these computational limits, Simon did not just throw up his hands and suggest that decisionmakers abandon the collection of information and resort to, say, flipping a coin.  He concentrated instead on techniques that might improve our ability to muster information, such as finding patterns in data.  Thus Simon famously studied the ability of master chess players to remember thousands of “chunks” of information on chess positions and to draw upon these chunks when planning their moves.  Or he told stories about how artisan watchmakers could increase production by dividing tasks into modular subassemblies instead of trying to put together a complete watch in one go.  Ultimately, however, Simon became fascinated with the potential of technology and computer simulation to organize information beyond the human capacity and thereby improve complex organizational decisionmaking.

The design of contract law is similar to the boardrooms and chessboards studied by Simon in that the creation of contract law also involves difficult and complex tradeoffs.  Just as directors and chess players seek data to reduce uncertainty and to guide their choices, contract scholars have moved toward empiricism as a basis for selecting the rules that govern our agreements, promises, and contractual bargains.2  Unfortunately, engineering the springs and wheels of contract law with empirical data presents profound challenges.  For one thing, it simply takes a long time to do this work.  Further, anyone studying the written opinions of judges or the executed agreements of private parties must limit her sample size to make the study feasible, which raises questions about the general applicability of the conclusions.  Even more troubling are recent assertions that our theoretical models have grown so complex that scholars simply cannot collect or process the needed empirical information.  Thus, while many are eager to see an empirical revolution, the volume of research that actually has emerged in this area remains low.

Yet, as Simon’s work suggests, the problem of harnessing information to support complex decisionmaking is not exclusive to contracts scholars.  In the corporate world, an entire industry has emerged around building knowledge-management technology to help organizations deal with vast volumes of unstructured information.3  Some of these tools are straightforward:  more powerful databases, better search engines, or systematic processes for getting the right information in front of researchers and decisionmakers.  Others, however, are more innovative and use computer algorithms that couple information theory with statistics.  These developments are leading to more powerful systems for managing text, voice, and video information, which large corporations, government security agencies, and others are already using with some success.  It is unfortunate that Simon died in 2001, for the technological progress of the last eight years undoubtedly would have intrigued him.

I believe that these knowledge-management tools may soon make it easier to conduct meaningful, large-scale empirical work in contract law.  Indeed, the general idea of using technology to augment legal research has a vast reach, and many of these concepts might facilitate empirical projects in other areas of legal scholarship.  But contract law is especially well suited to take advantage of knowledge management for two reasons.  First, the choices that parties make when forming contracts often leave a data trail:  Principals typically record the terms of their agreement in a way that is less likely to happen within the context of, say, tort or criminal law.  Second, large, centralized repositories of contract data are now available, including databases of historically executed contracts, litigated case law, and other relevant sources.

By tapping into cutting-edge knowledge-management technology, we can, and soon will, conduct meaningful empirical work in contract law.  Furthermore, it will be possible to automate much of this work to allow constant updating and refreshing of the empirical studies.  To be sure, contract law is complex, and certainly we will never reach uncontested results that allow lawmaking without subjective judgment.  But knowledge management offers some intriguing possibilities, and it is worth considering how the tools that organizations use to manage unstructured information might also assist scholarly inquiry into the incentives and effects of contract law.

 
I.
The Current State of Contract Law Research

Over the past few decades, much of contract law scholarship has focused on developing economic theories that offer a normative approach to setting the legal rules governing binding promises.  Standard economic analysis consists of examining a project that has potential benefits for both buyer and seller.  The researcher then develops an economic model that considers various contingencies and contract terms and uses that model to explore how—and whether—the parties will trade and invest under different legal default rules.  Based upon these results, the researcher may suggest that a particular rule is economically superior to alternatives under the model’s assumptions.  The challenge, of course, is to determine whether these models are accurate reflections—or at least partial reflections—of the way the world really works.  Overall, I think it is fair to describe empirical contract law scholarship as underdeveloped.

Seizing upon the slow rate of progress, some scholars have raised another pessimistic thought:  Maybe this type of work just can’t be done.  In a provocative essay in the Yale Law Journal, Eric Posner argues that there are two fundamental barriers damming a much-needed flood of empirical contracts research.4 First, data required to assess some economic theories of contract law are simply not available.  Second, even if it is possible to get at some (or all) of the necessary data, it may be difficult to sum up the effects of numerous variables into probabilistic choice models that coherently suggest the superiority of a particular legal rule.  How can we possibly model the complex interactions among these variables to arrive at an optimal prescription for courts or legislators?

I agree with Posner that empirical analysis is taxing and that it can be difficult to gather and aggregate the information necessary to reach sound conclusions.  But I would also argue that the future of empirical contracts analysis need not be so bleak.  While Posner’s concerns about a dearth of information may once have been accurate, we are now entering an era where more and more data on variables related to contractual choice—including human preferences and economic cost structures—are becoming available.  Similarly, I am optimistic that scholars can use analytical techniques, such as random sampling and simulation, to combine the most important variables underlying economic models of contract law, and move us beyond an indeterminacy quagmire.

Related to these efforts, the challenge of managing massive quantities of information arises in many different contexts, and some well-funded organizations have profit motives for inventing sophisticated data-management software to aid sound decisionmaking.  How do these tools work, and might they also be helpful for studying and testing our theories of contract law?

 
II.
Technical Advances in Organizational Knowledge Management

Every day we are bombarded with information related to our decisions.  Some of this information is structured—that is, placed in a database or other format that can be readily defined, accessed, and manipulated at an atomic level.  More and more, however, we must wade through unstructured information:  the memo from last month, the video news clip from Tuesday night, this morning’s phone call, or the collection of emails lingering in our inboxes.  This proliferation of unstructured information is not necessarily a bad thing (we enjoy its richer content), but it does complicate life for organizational decisionmakers seeking to use the data to reduce uncertainty and risk.

In response to the challenge created by unstructured data, researchers are developing new algorithms that blend information theory, statistical inference, and other ideas to provide better tools for taming information.  One of the most intriguing innovations in knowledge management involves meaning-based computing.  The general idea is to organize unstructured information in a way that does not depend on keyword tagging (as with Altavista and Excite), anchor-text references (as with Lycos), or linking citations (as with Google).  In other words, meaning-based computing goes beyond most search algorithms in that it tries to estimate (I won’t say understand) what the information actually means.  Two of the most important techniques it uses to do this are Shannon’s information theory and Bayesian inference.

A researcher at AT&T Bell Labs in the 1940s, Claude Shannon worked to encode information so that it could be communicated accurately and efficiently, meaning that a message could still get through even if noise altered the signal.5  Recognizing that a great deal of information is redundant, Shannon wondered whether it was possible to quantify certain units of communication as more important for conveying meaning than others.  The answer was surprising:  The most disordered, or unexpected, term will often denote the most meaning.6  For example, in the sentence “I’m flying to Arizona tomorrow to see the Super Bowl,” the words “Arizona,” “Super,” and “Bowl” probably occur most infrequently.  Thus, the algorithm would treat them as more indicative of the sentence’s meaning.  Knowledge-management software uses this concept on a much grander scale to estimate the most important terms in a piece of unstructured information.

Yet even infrequently occurring words can be ambiguous.  In the sentence quoted in the last paragraph, for example, is the speaker talking about a football game or her grandmother’s terrific china?  To help with this problem, some algorithms use a second important concept:  the statistical notion of Bayesian inference.7  In a nutshell, this is a statistical technique used to update the probability that a prior belief is true in light of newly received information.  In the field of knowledge management, Bayesian inference is used to calculate probabilistic relationships between multiple variables and to determine how one variable impacts or relates to another.  These relationships are important in two ways.  First, within a bundled communication, such as a letter or phone call, they can help refine or deduce the meaning of the message.  Second, researchers can apply Bayesian inference to a larger universe of data in order to estimate which parcels of unstructured information are likely to deal with similar concepts or messages.  To illustrate these applications by returning to our earlier sentence, we might have a prior belief that any time the word “bowl” is used, it refers to something you would use to eat soup or cereal sixty percent of the time; a form of casual recreation involving ten pins and a heavy ball thirty percent of the time; and a forum for a football game just ten percent of the time.  But using Bayesian inference, we might modify that prior belief from a 60-30-10 split to a 10-10-80 split once we discover that the word “super” occurs in close proximity to “bowl.”

These algorithms are proving useful in a wide variety of fields, in part because the technology is more accurate and efficient than other methods of processing unstructured information, such as keyword searches or manual tagging schemes.  The relative advantages of meaning-based computing offer some intriguing possibilities for empirical legal research.  Contract law, in particular, may lend itself to this sort of analysis because private agreements generate a great deal of unstructured information, more and more of which is becoming available for examination.

 
III.
Automating Contract Law

There are at least three broad possibilities for using knowledge-management technology to study contract law.  First, we might use meaning-based computing to improve doctrinal analysis.  Second, the data-gathering technology could help us operationalize or revise normative economic theories of contract law.  And finally, the information we generate might help us to conduct what I call predictive modeling—that is, to run massive computer simulations on various economic models of contract law in an attempt to derive a pretty good, though not necessarily perfect, rule.  I will briefly discuss each idea.

Doctrinal research involves examining published court decisions in order to determine how judges implement legal principles.  A contracts scholar, for example, might read several hundred cases on the mistake doctrine to study how courts differentiate between unilateral and bilateral mistakes and to clarify the conditions necessary to excuse contractual liability.  Focusing on doctrine seems to have fallen out of favor in recent years, but the criticisms are often overstated, and this work can provide important insights into our legal system.

Meaning-based computing algorithms can help doctrinal scholars in two different ways.  First, they make it easier to identify relevant cases.  Instead of relying on Lexis Nexis, Westlaw, or some other aggregator to tag legal cases with subject-matter descriptors manually, researchers might scan every single case involving contract law with meaning-based algorithms that look for all relevant cases—even those using unorthodox terminology.

More substantively, knowledge management might help synthesize the key factors that influence courts in borderline cases.  Ultimately, everyone conducting this type of research must simplify large amounts of unstructured information into manageable distinctions; prior contextual understanding will usually shape this synthesis.  By contrast, unleashing the statistical objectivity of meaning-based computing might lead to a more consistent processing of contract doctrine.  In theory, we could add every contracts case to a centralized database and process each case with a meaning-based algorithm in order to develop an initial subject-matter impression.  This sounds like a lot of work, but organizational knowledge-management systems routinely manage hundreds of thousands or millions of documents.  From here, researchers could cluster the information according to key concepts, perhaps tune the results by “teaching” the algorithm which connections are the most important,8 and use the analysis to refine theories on how and why courts are deciding cases.  And, importantly, there would be no need for human recoding if we needed to extract another variable from the cases.  In other words, automated analysis eliminates the need to predetermine which factors will be included in the tagging taxonomy for doctrinal review.  Finally, computer-aided descriptive analysis might allow for the replication (or automation) of doctrinal research over multiple periods in time.

Of course, doctrinal analysis can only go so far in helping us to reform contract law.  Knowing how courts are deciding cases does not tell us how they should be deciding cases.  Can meaning-based computing aid in the normative inquiry by prescribing the default rules that should operate when parties’ contractual intentions go unexpressed?

Here, unfortunately, the path is steep.  Twenty years ago, I might have argued that we could simply examine a million contracts to glean the popular preferences of contracting parties across diverse terms.  In essence, each contract could serve as a ballot, and we could import the most-voted-for term as a majoritarian default rule.  Meaning-based computing would simply be a way to tally the votes faster.  But work over the past two decades suggests that the selection of contract default rules is much more difficult than simply picking the most popular terms.  We cannot always infer, for instance, that failure to change a default rule constitutes approval rather than an effort to economize on transaction costs.  Furthermore, penalty default rules may be justified, even if unpopular, by their ability to force parties to reveal socially desirable data.  Finally, default-rule theory presents a granularity problem because lawmakers are free to set different default rules for different groups of contracting parties.

All of these concerns make me skeptical of any project that aims to select an optimal default rule.  Yet precisely because this task is so hard, it is worth asking whether powerful computing technology might assist with the process.  After all, lawmakers cannot abandon the job; there must be some legal rule or standard that takes effect in the absence of stated contractual intentions.  So, let me offer a few tentative thoughts on how meaning-based computing might help with the selection of rules.

One possibility is to use knowledge-management algorithms to conduct natural experiments on change and effect in contract law.  The basic idea here is to find a situation where there has been a doctrinal change—perhaps due to a landmark court decision or statutory enactment—and measure the impact of the change on subsequent contract terms in that jurisdiction.  This would be analogous to an event study in finance or economics and might help to shed light on the incentives and effects of differing legal defaults.

A second, related project would involve studying hundreds of thousands of contracts to identify and cluster standard boilerplate provisions.  This would allow us to assess how and when these provisions change and to try to link this evolution to default-rule changes in contract law.  Again, this would not give us a complete normative basis for choosing a particular default over another, but it would speed the partial analysis of incentives and effects.

Third, if we decide that it is appropriate to enact different default rules for different contracting segments, then it might be possible to use computer-aided clustering to help determine a meaningful segmentation.  One of the most difficult problems is identifying appropriate ways to classify parties into different treatment groups—who gets Default A, who gets Default B, and so on.  It might be powerful, therefore, to run knowledge-management clustering analysis on many different types of contracts in order to gather evidence on where salient characteristics diverge in order to help define the segments.

These techniques are imperfect, however, and they are unlikely to prove helpful for all problems related to the design of contract law.  It may still be difficult to collect the necessary data, especially when a decision turns on many variables.  It is also fair to question the extent to which parties will respond as expected to the incentives created by contract law.  Moreover, contract law is a social and linguistic construct, as well as an instrument for the regulation and promotion of exchange, and no computer on Earth can plumb the depths of human society.

 
IV.
Predictive Modeling

Still, there is one last response:  We may not have to get it perfect.  There might still be benefits from using knowledge-management technology to conduct predictive modeling to determine which legal rules will “do a good job” most of the time—even if they are not flawless.  I think the easiest way to introduce this idea is through an analogy.

In 1997, IBM stunned the world by beating the current chess champion, Garry Kasparov, with its Deep Blue supercomputer.  The victory was seen as a technological milestone because chess provides the perfect setting to test concepts like pattern recognition, learning, and planning.  More recently, computers have become the undisputed champions in other arenas of the parlor—including Othello, Scrabble, backgammon, and bridge.  The success of computers in these games is largely due to sophisticated processors that enable programmers to design “brute force” strategies similar to the one used by Deep Blue.  Essentially, the computer trees out all possible moves, countermoves, counter-countermoves, and so on until it has enough information to pick the move with the highest probability of success.

But there is one game where computers have never been able to give humans a good fight:  the East Asian game of Go.  Go requires players to take territory by placing black and white stones on the intersections of a grid to surround their opponent.  It is a complex game, with some stones appearing “dead” until the very end, when they spring back into life at a critical moment.  A computerized brute-force strategy has not succeeded because Go has many more potential moves than chess, and it is very tricky to evaluate any given position.

Recently, however, programmers have started using a different sort of algorithm to give human Go players a tougher match.  Instead of drawing comprehensive decision trees, the computer uses Monte Carlo simulation, a method of statistical sampling using random-number generation.  In a nutshell, the computer will pick one move and randomly play out thousands or millions of games to the finish.  If the computer wins often with this move, say seventy-five or eighty percent of the time, then it will stop analyzing the position and take its chances with this move.  Otherwise, it goes on, selecting a second move, rerunning the Monte Carlo simulation, and recording the probability of success.  The process is repeated only until an acceptable move is found; there is no attempt at perfection.  This strategy is intriguing (and apparently successful) because it mitigates complexity by replacing a search for the optimal move with a search for one that is just good enough most of the time.

In the same way, we may not need to search, via brute force, for the perfect contract-law default rule.  Instead, we might approach the problem like the computer programmer seeking a win at Go.  In some cases, we may be able to use unstructured information to form probability distributions for the parameters that are critical to economic models of contract law.  We could then use Monte Carlo simulation to run millions of different iterations on our predictive model to find a default-rule structure that seems to produce high-Pareto outcomes.  In short, we would split our model economy into two or more parallel universes, give each universe a different default rule, and run millions of random iterations to see which rule typically leads to pretty good outcomes.

To be sure, predictive modeling will not let us empirically solve for the optimal, or most efficient, answer.  It may not be very satisfying to concede that we cannot know which rule of law is best.  But if the game of Go is complicated, the collective choices of billions of people across a diverse range of economic settings are even more so.  Optimal precision in contract law is probably beyond our grasp right now, but we can still use partial economic analysis to inform—and perhaps reform—our legal rules.

 
V.
Conclusion

We are on the edge of innovation in the fields of search and organizational knowledge management.  Recent advances in information theory, statistical inference, and other technologies are helping organizations to draw upon unstructured information in order to make better decisions.  These algorithms are not perfect.  But the technology of organizational knowledge management is getting better at estimating meaning, and there are some intriguing possibilities for using these tools to assist descriptive and normative projects in contract law.

Just to be clear, I am not advocating the substitution of computer algorithms for human choices.  Contract law is a social construct, as well as an economic one, and we will never be able to set our rules without subjective judgment.  Until a supercomputer can simultaneously mimic the mental activity of seven billion brains, we will probably have to settle for contract laws that are good enough—instead of demanding ones that are perfect.  Ultimately, however, we must make choices about what rules to adopt, and we must continue to ask how contract law shapes human action.  We should, therefore, take up every available tool to conduct the partial empirical analysis that might help with these complex problems.dingbat

 

Acknowledgments:

Copyright © New York University Law Review.

George S. Geis is Professor, University of Virginia School of Law.

This Editorial is based on the following full-length Article: George S. Geis, Automating Contract Law, 83 N.Y.U. L. REV. 450 (2008). Click Here for the Full Article

  1. Carnegie Mellon University Libraries, Herbert Simon Collection, Exhibit, Problem Solving Research, http://shelf1.library.cmu.edu/IMLS/MindModels/problemsolving.html (last visited Apr. 8, 2009).
  2. See, e.g., Ian Ayres, Valuing Modern Contract Scholarship, 112 YALE L.J. 881, 900 (2003) (predicting rise in empirical analysis of contract law); Russell Korobkin, Empirical Scholarship in Contract Law: Possibilities and Pitfalls, 2002 U. ILL. L. REV. 1033, 1037 (offering framework for classifying and pursuing future empirical scholarship in contract law); David V. Snyder, Go Out and Look: The Challenge and Promise of Empirical Scholarship in Contract Law, 80 TUL. L. REV. 1009, 1009-10 (2006) (reflecting on current state of contracts scholarship and on heightened interest in empirical analysis). This trend is not confined to contract law, of course; legal scholars of all stripes seek tangible data to evaluate their theories.
  3. For a collection of introductory articles related to knowledge management, see HARVARD BUSINESS REVIEW ON KNOWLEDGE MANAGEMENT (Harvard Bus. Review ed., 1998). For a more comprehensive synthesis of work in this area, see KIMIZ DALKIR, KNOWLEDGE MANAGEMENT IN THEORY AND PRACTICE (2005).
  4. Eric A. Posner, Economic Analysis of Contract Law After Three Decades: Success or Failure?, 112 YALE L.J. 829, 864-65 (2003) (arguing economic approaches fail to explain contract law because they are indeterminate).
  5. C.E. Shannon, A Mathematical Theory of Communication (pts. 1 & 2), 27 BELL SYS. TECHNICAL J. 379, 623 (1948).
  6. CLAUDE E. SHANNON & WARREN WEAVER, THE MATHEMATICAL THEORY OF COMMUNICATION 51 (1964).
  7. For more background on Bayesian inference, see generally ANDREW GELMAN ET AL., BAYESIAN DATA ANALYSIS (2d ed. 2004).
  8. By “teaching,” I mean updating the Bayesian networks that reflect the strength of links between words and concepts (either through human understanding and intervention or through automated review of additional, related content). Many knowledge-management programs allow users to tune systems in this manner.

Comments

  • This summary and the full article include the following sentence:

    “More recently, computers have become the undisputed champions in other arenas of the parlor—including Othello, Scrabble, backgammon, and bridge.”

    I was surprised to hear this of Scrabble, because I follow the scene fairly closely and play intermediate-level tournament Scrabble myself. While the best programs out there (Maven, Quackle) are quite good, on par with the best human players at somewhere in the 40-60% success range (I don’t remember what number I last heard for human successes), they are not currently “undisputed champions”. (Of course, the inherent randomness in Scrabble means that it’s difficult to generalize from anything but extremely large numbers of games, because luck of the draw plays such a notable role in determining who wins the games.) I was curious where the author found his information; the citation for this sentence is this article from The Economist:

    http://www.economist.com/displayStory.cfm?story_id=8585017

    Reading through, the only mention of Scrabble is in this sentence (which also happens to fit in the excerpt at the above URL):

    “[Computers] are the undisputed champions in draughts and Othello. They are generally stronger in backgammon. They are steadily gaining ground in Scrabble, poker and bridge.”

    The article thus misstates its source’s information. It is correct that computers are the undisputed champions of backgammon (draughts) and Othello, but it is incorrect to say or imply the same for Scrabble and bridge, where they are merely steadily gaining (although in the case of Scrabble, I believe this is more a matter of computers increasing in power than of any breakthroughs in Scrabble-game playing by the programs).


Post a Comment (all fields are required)

You must be logged in to post a comment.