Because separate opinions and reversals constitute behavioral manifestations of judges’ discretionary authority, studies of dissensus shed light on critical questions related to the effective functioning and legitimacy of our legal system and the operation of the rule of law. . . . Our findings cut both ways. The evidence we have presented in the preceding pages of this book demonstrates that judging is both a legal and a political activity . . . .²

Surveying the results of recent quantitative studies of judging, Cass Sunstein and Thomas Miles observe that “[f]or those who believe in the rule of law, and in the discipline imposed by the legal system, the results of the New Legal Realism need not be entirely discouraging. The glass is half empty, perhaps, but it is also half full.”³

The rule of law is often said to be a defining aspect of the American system of governance—the foundation stone of our free society—in which judges play a pivotal role.⁴> It is alarming to be informed that serious problems in judging threaten the rule of law or that the rule of law glass is half empty. If these concerns are valid, remedial measures must be sought and implemented without delay.

But are they correct? Have studies of judging shown that the rule of law is in trouble? To evaluate these assertions, one must first know what the rule of law requires of judges; then one must identify or measure how much and in what ways judges are falling short of these requirements. To say that the rule of law glass is half full, continuing with Sunstein’s and Miles’s metaphor, requires knowing what a full glass of the rule of law looks like: there must be rule of law baselines or standards.

A quick look at these studies exposes the need for such baselines. The authors of Are Judges Political? find that “[f]requently the law is clear, and judges should and will simply implement it, no matter who has appointed them.”⁵ Their study provides “considerable evidence to suggest that they do exactly that”;⁶ in five major areas studied they find no ideological effect on judicial decisions, and even when an effect did show, the differences, they admit, were “not huge.”⁷ A study was not necessary to show that judges do not vary greatly by ideology in their legal decisions because typically about 90 percent of federal appellate decisions (more when unpublished cases are counted) are issued without a dissent. Judges, then, agree an overwhelming proportion of the time regardless of ideological differences. By that measure, at least, the rule of law appears to be working well.

The authors of Judging on a Collegial Court find that ideological differences show a statistically significant increase in the probability of a dissent or a concurrence. But it turns out that the size of the effect—its actual impact on the run of decisions—was minuscule: “The difference in absolute terms is rather small, with slightly less than a 0.01 increase in the probability of a concurrence and a 0.02 increase in the probability of a dissent.”⁸ That is hardly worrisome. Confounding the authors’ expectations, furthermore, their study finds no statistically significant correlation between ideological difference and rates of reversal—that is, appellate panels did not reverse trial judges with an opposing ideological disposition at a higher rate. This study, covering decisions by nearly a thousand judges over four decades, would appear to confirm that political views have little impact on judicial decisions, yet, without explaining why, the authors suggest that their findings “cut both ways” on the rule of law.

Behind the disquieting assertions about the rule of law lies an unstated assumption: the proposition that any finding of political influence on judging, no matter how small, is contrary to the rule of law. This, however, is a profoundly unrealistic assumption—ironically so, because these political scientists and law professors claim the mantle of legal realism.

 
I.
Balanced Realism About Judging

For more than a century, judges and jurists in the United States have expressed a view of judging that I call “balanced realism.”⁹ Balanced realism recognizes that there are gaps and uncertainties in the law, that sometimes judges have discretion and must make choices, that different judges can sometimes interpret the same law in different ways owing to differences in perspective and background, that inconsistent precedents or conflicts in the applicable law can exist, and that sometimes judges manipulate the law to reach desired ends. (I call these factors the “skeptical aspects.”) But balanced realism also recognizes that a substantial majority of the time, the rules and their application are clear and predictable, that judges are indoctrinated into a shared legal tradition and legal practices that lead then to interpret and apply legal rules in similar ways, that judging takes place in a thick institutional setting that constrains judges, that most judges strive to abide by the commitment to follow the law, and that the overwhelming majority of judicial decisions are legally determined (the “rule-bound” aspects). Balanced realism acknowledges the limitations inherent in the law and in human judges—limitations that cannot be eliminated—yet it also recognizes that law nonetheless works, that judges can and do render rule-bound decisions.

Judge Benjamin Cardozo famously articulated a balanced realism about judging in The Nature of the Judicial Process:

No doubt there is a field within which judicial judgment moves untrammeled by fixed principles. Obscurity of statute or of precedent or of customs or of morals, or collision between some or all of them, may leave the law unsettled, and cast a duty upon the courts to declare it retrospectively in the exercise of a power frankly legislative in function.¹⁰

Cardozo insisted that when making these decisions a judge must decide in terms of the community view, not the judge’s personal view, but he was aware that it is difficult to keep the two apart: “The perception of the objective right takes the color of the subjective mind.”¹¹ But despite the inherent openness of law and the limitations of human judges, Cardozo reminded his audience, “[w]e must not let these occasional and relatively rare instances blind our eyes to the innumerable instances where there is neither obscurity nor collision nor opportunity for diverse judgment.”¹²

Multiple judges before and after Cardozo have described judging in similar terms. In 1886, for example, Judge Thomas Cooley emphasized that uncertainty in the application of law cannot be eliminated

because in the infinite variety of human transactions it becomes uncertain which of the opposing rules the respective parties contend for should be applied in a case having no exact parallel, and because it cannot possibly be known in advance what view a court or jury will take of questions upon which there is room for difference of opinion.¹³

Differences in the judicial application of law, he wrote, “must always exist so long as there is variety in human minds, human standards, and human transactions.”¹⁴ In 1924, Judge Irving Lehman acknowledged that “no thoughtful judge can fail to note that in conferences of the court, differences of opinion are based at least to some extent upon differences of viewpoint.”¹⁵ Judge Bernard Shientag remarked in 1944,

[n]aturally, it is in cases where the creative faculty of the judicial process operates, where there is a choice of competing analogies, that the personality of the judge, the individual tone of his mind, the color of his experience, the character and variety of his interests and his prepossessions, all play an important role.¹⁶

Judge Albert Tate observed in 1959, “like all other human beings[, judges] have limitations, of vision, knowledge, intelligence, or predisposition which sometimes influence their judicial actions.”¹⁷ In 1963, Judge Charles Clark admitted that cases arise in which there is no clear legal answer, and the judge “is on his own for the ultimate result which must reflect his background, his personality, and his inner convictions.”¹⁸ And so on.

This encapsulates what many judges have said about judging: the bulk of the law is clear, but the law has a margin of uncertainty; judges try their best to rule in an objective fashion, but their personal views sometimes seep through to influence their decisions. The crucial point is that law cannot be made perfectly certain and judges cannot be made to reason like machines, entirely free of background influences. These inherent aspects of judging shape and constrain what is possible. “The rule of law is not the doctrine of perfect decision,” Judge Alvin Rubin counseled: “[I]n many cases a conscientious decision is as much as can be expected, and . . . there is no ultimate ‘right’ answer.”¹⁹

Now it is possible to identify the fundamental flaw in the assumption that any showing of political influence on judicial decisions is inconsistent with the rule of law. A realistic understanding of the rule of law would assume that a certain irreducible amount of ideological influence will be present even in the best system of judging.²⁰ As judges have repeatedly stated, it cannot be otherwise. A realistic view would therefore expect that quantitative studies will find statistically significant correlations in certain contexts between ideology and judicial decisions. This finding in itself, without more, says nothing at all about the rule of law, because it is an inherent aspect of judging. Or to put the point another way, a full glass of the rule of law, like a full glass of milk, is not filled to the brim. The open space between the lip of the glass and the surface of a full glass of the rule of law is where legal uncertainty interacts with the limitations of human judges—where political influences typically come into play.

For the rule of law, what matters is the size of the ideological effect and in what contexts it manifests itself: whether it is greater than, or extends beyond or outside of, what one would expect in a well functioning system of rule-bound judging. Rule of law baselines are necessary to identify what to expect of a full glass of the rule of law. Quantitative studies can raise serious concerns about the rule of law only if their results establish that judicial decisions fall measurably below these baselines. Only then would grounds exist to assert that the rule of law glass is half full, or nearly empty.

 
II.
Constructing Rule of Law Baselines

Throughout this Essay, I have referred to baselines in the plural because a number of standards will be necessary to account for variations in the nature of legal provisions and the circumstances of judging.  Two factors have particular bearing on the formulation of standards: the type of legal issue a judge is called upon to decide, and the level of the court.

In connection with the first factor, certain legal provisions—especially legal standards like “fairness,” “reasonableness,” or “the best interests of the child”—explicitly call upon judges to exercise discretion or to make judgments of a type that allows or invites (or makes it harder to screen out) the expression of personal views. Consequently, a rule of law baseline for this type of question, which remains legally governed and hence should manifest a significant degree of agreement, would anticipate greater variation among judges and higher correlations between their decisions and their ideological views in comparison to a rule of law baseline for narrow legal rules.

The second factor recognizes that the quantum of legal uncertainty is greater at higher court levels. The vast majority of cases are settled (fewer than two percent of federal cases make it through trial) because the applicable law and provable facts are clear, so both parties can weigh the expected costs and benefits of continuing. About ten to fifteen percent of federal appellate cases, by the estimate of a number of federal judges,²¹ involve hard or uncertain legal issues. In contrast, the skeptical aspects of law recognized by balanced realism—uncertainty, disagreement, choice, political pressure—show up in a significantly higher proportion of Supreme Court cases, whereas the rule-bound aspects are proportionally less present (including fewer institutional checks). This represents a virtual inversion of the usual balance of these factors within law and judging generally, although Supreme Court decisionmaking is still thickly draped in legal constraints. In recognition of these differences, rule of law baselines for trial courts and appellate courts should anticipate far greater agreement in legal decisions and significantly lower ideologically correlated variations in comparison with high courts (both state and federal).

Needless to say, the task of formulating rule of law baselines will be complicated, requiring ingenuity and much trial and error. This task can be done in a variety of ways, all contestable. Every baseline produced should be viewed with caution, as a proxy that stands for an approximation of an abstraction—a gross quantitative marker for what to expect from rule-bound judges.

Seeking out comparative measures is one way to proceed. For example, assume that over a seventy-five-year span conservative judges vote in the conservative direction in about fifty-five percent of their cases, whereas liberal judges vote conservative in about fifty percent of their cases.²² The relatively small five percent difference in voting behavior, one might surmise, reflects the irrepressible interaction of legal uncertainty with human judging. This historical norm could supply the basis for a rule of law baseline for federal appellate judging. A warning sign that the judicial system is in trouble, then, might be if judges as a group skew their votes in a one-sided ideological direction in a significantly higher proportion of cases, creating a greater than usual disparity between Republican- and Democratic-appointed judges.²³ Moreover, individual judges whose decisions fall far outside of this historical range might invite scrutiny for failing to rule in a sufficiently rule-bound fashion. One might create similar baselines for the Supreme Court, derived from historical norms or from a comparison of the voting patterns of Justices against one another. This would allow a determination of whether a particular Court or a particular Justice shows a propensity to rule in an ideological direction that exceeds the usual, historical range.

Setting aside levels of courts, one might also compare differences across legal issues to see whether some issues show greater ideologically linked divergence in judicial decisions than others. Applying different rule of law baselines for standards (expecting greater divergence) and for rules (expecting less) will make it possible to tease out whether the observed increase in divergence is a function of the type of legal provision at issue, or of something else (perhaps political salience or entrenched cognitive biases).

These are just illustrative suggestions. Many factors must be considered before rule of law baselines can be constructed—work that has not yet begun.

 
III.
Legitimate Objections to Rule of Law Baselines

Critics may object that it is misguided or wrong to construct such baselines, that the goal itself is ludicrous because the rule of law is a deeply contested ideal with uncertain meaning and implications for judging. Furthermore, critics might argue, quantitative standards that purport to provide a basis to evaluate judging will compress the complexity and nuance of judging in a distorting oversimplification that is susceptible to pernicious uses. These are compelling objections. I would not proffer this proposal but for the worry that leaving this gap unaddressed might be worse than the distortions that result from the effort to fill it.

As the first paragraph of this Essay reveals, political scientists and law professors, perhaps succumbing to the temptation to sell their results, have issued broad, alarming claims about the implications of their quantitative studies for the rule of law. These claims, I have argued, do not follow from the results of their studies in the absence of rule of law baselines, and they paint a false image of the state of judging. Rule of law baselines will impose greater discipline on scholars who wish to draw out broader implications from their results and will provide a sounder footing for their observations.

This effort will also lead to an important advance in the discipline of quantitative studies. Quantitative scholars demonstrate time and again through their studies that judging is not a purely legal activity. This point is not informative. Dozens of judges have admitted for decades that sometimes law is uncertain or runs out, that judges must sometimes make choices, and that sometimes their personal views have an impact on their legal decisions.²⁴ That is the nature of law and of human judging. The formulation of rule of law baselines would constitute a major advance within the field, because these baselines would formally incorporate the recognition of this reality, affecting the orientation and design of the next generation of studies. Future quantitative studies would produce information worthy of attention, not when merely finding indications of ideological influence, but when finding a notable deviation from expected baselines. Absent rule of law baselines, these rapidly multiplying studies will merely confirm what everyone in law already knows.

Acknowledgments:

Copyright © 2010 Duke Law Journal.

Brian Z. Tamanaha is a Professor at the Washington University School of Law.

When I think of “judging” as an anthropologist, I think of two of the classics in legal anthropology:  Paul Bohannan’s Justice and Judgment Among the Tiv ¹ and Max Gluckman’s The Judicial Process Among the Barotse.² Both of these books were ethnographic studies of African tribal courts during the latter days of British colonial rule.  The tribal judges were prominent in both books, and when I teach them in law school I ask students to compare the tribal judges’ roles and conduct to what they know about contemporary American judges.  Like American judges, these African judges sometimes maneuvered the disputants into settlements and sometimes issued final judgments.  Unlike American judges, they never wrote opinions.³ So one could understand them only by watching, listening, and attending to all aspects of their behavior, including such apparently trivial details as how they arranged the seating in the open-air courtroom.

I also think—though far less grandiosely, of course—of the work that Mack O’Barr and I did in American small claims courts in the 1980s and 1990s.⁴ Like their African counterparts, the judges we watched in these courts persuaded, cajoled, threatened, and, when necessary, judged.  Nothing except the basic terms of a final judgment got committed to writing.  Moreover, again as in the African courts, the small claims judges had broad and largely unreviewable authority to impose whatever “law” they saw fit.  In one memorable case that involved a dry cleaner’s liability for a damaged suit, the judge decided to await and abide by the ruling of that august supranational tribunal, the International Fabricare Institute.

Regardless of any methodological preferences or preconceptions we might have had, the absence of written work product forced us—as it had forced Gluckman and Bohannan—to focus on the performance aspects of judging.  Exactly what did the judges say?  Just how did they say it?  What was their non-verbal behavior like?  And a good thing, too: this enforced focus on behavior led us in directions we never would have thought of on our own.  We found a range of judging styles and approaches; in anthropological terms, each courtroom had its own legal culture, created by the judge.  Within these mini-cultures, we found curious echoes of such jurisprudential schools as formalism and realism, as well as applications of the psychological concept of procedural justice.  This is not to say that the judges were consciously enacting these theories, but enacting them they were.

This focus also led to another set of findings, all derived from an initial observation that courtroom interactions are a form of conversation.  (This observation was influenced by an earlier study of British courts, Max Atkinson and Paul Drew’s Order in Court.⁵)  Practitioners of various forms of “ethnomodology” (including conversation analysts, linguistic anthropologists, and the many species of discourse analysts) have long demonstrated that “doing” conversation is an enormously complex social task, one that requires detailed rules.  Moreover, ethnomethodologists argue, these and other things that we think of as social “rules” are not rules in the sense of preexisting principles that need only to be applied.  Rather, they are “phenomena of order,” apparent rules that people “attend to,” and that “emerge” over and over again as people negotiate concrete social situations.  For example, two separate discourse analyses of actual jury deliberations (by Doug Maynard and John Manzo, in 1993,⁶ and Robin Conley and me, this year⁷) have shown, respectively, how principles of justice and standards of proof are produced by jurors in the course of their conversations.  In the small claims courtroom, O’Barr and I saw emergent rules for such judging problems as displaying authority, “doing” due process, showing empathy, and delivering bad news.

But why would these approaches have any relevance for those who study “real” judges?  In a current draft of a paper on the practical impact of “landmark” decisions, Professors George, Gulati, and McGinley⁸ suggest that the crowd divides into two general camps:  the legal scholars, who parse opinions; and the political scientists and their allies, who count and otherwise slice and dice outcomes.  Why might either camp care about methods of studying judging developed while sitting under a tree in Nigeria or in grimy small claims courts around the United States?

Because “real” lawyers and their clients might care.  The most striking observation made by George et al. is that practicing lawyers and judges as well as potentially affected citizens may not define “landmark” in the same way as academics.  In the case that George’s group focused on, what the eggheads expected to be earth-shaking barely registered on the ground.

This leads to a corollary idea more directly related to judging:  practicing lawyers may define “good” and “bad” judges in very different ways than academics.  Law professors tend to praise written opinions that are logically tight, that deal appropriately with precedent, and, most of all, that validate the professors’ policy preferences.  The authors of such opinions are “good” judges.  Similarly, quantitative studies tend to exalt judges who write lots of opinions (some researchers prefer long ones and others short ones), who rarely get reversed, and who get cited often.

But talk to practicing lawyers and you might get a different emphasis.  (I base this generalization on my own six years in full-time practice and 26 as of counsel to a firm and student of the profession.)  First, reversing the priority of their status-conscious academic betters, lawyers may focus more on trial than appellate judges because of per capita impact on their practice.  They are often more interested in how law is administered in their cases than in what law is made in “important” cases.  Second, my guess is that practitioners would identify a much higher proportion of “bad” judges.  If, for instance, one were to rely solely on the testimony of former law clerks now in academia, all federal judges would emerge as hybrids of Oliver Wendell Holmes and St. Thomas More.  But practitioners see some of these same judges as, in varying combinations, lazy, stupid, and mean (to quote my brother, a practicing lawyer).

In identifying “good” judges, practicing lawyers are likely to ask such questions as: How does s/he behave in court?  Does s/he treat lawyers, litigants, and witnesses fairly and courteously?  Can s/he manage a trial competently? Does s/he actually listen or does s/he prejudge things?  Will s/he decide motions in a reasonable period of time, or does s/he ignore them in the hope they’ll go away?  How smart is s/he?

These judicial qualities, I suspect, are very important to those who must live with the judges.  But they do not necessarily emerge from written opinions, nor from aggregations of outcomes.  They would become evident only through an inductive, ethnographic approach to judging that paid particular attention to language.  I wonder what other “phenomena of order” might emerge.  I suspect that important clues to the legitimacy of the legal system might reside in the details of courtroom interactions, and that a significant, “emergent” notion of justice might reveal itself as well.  The point is that, based on anthropology’s experience, including my own, I would not know what hypotheses to test a priori, but would instead expect the unexpected.

An aside: the legitimacy issue was brought home to me when I made my first visit to the European Court of Justice this past summer.  With a group of students, I attended a hearing in a case about import duties (a private company versus the European Commission).    The lawyers argued in German, the judges spoke in French (badly, since none came from France), and I listened alternately in French and English.  The lawyers read prepared remarks and the judges asked no questions; their only utterances were organizational and perfunctory (“We’ll hear now from Herr X”).    Instead, they stared with glazed eyes or flipped through the parties’ written submissions.  I could not imagine what value this presumably expensive exercise had for anyone (except the private-sector lawyer, of course).

After the hearing I discussed it with an ECJ staff lawyer.  I asked, in more polite language, “Why bother with this charade”  I suggested that the court learned nothing and the parties did nothing to advance their cases that couldn’t have been done better in writing.  He didn’t disagree, but said that hearings helped to legitimize the ECJ in the eyes of litigants and, more broadly, to inculcate EU citizenship.  I could not imagine how.  But I thought that if I had been paying closer attention—paying ethnographic attention, as it were—and had a chance to study and restudy my own, linguistic transcript of the hearing, I might have been able to figure out how the court and its constituents were “doing” legitimacy.

All of these considerations lead to the methodological question of how an ethnographer would study “real” judges (federal district judges, let’s say).  One thought might be to collect trial transcripts of cases that had been appealed and do discourse analysis.  That would be interesting.  However, from a sociolinguistic perspective, that would be more a study of transcripts than a study of judicial behavior.  Court reporters perform a subtle kind of interpretation, and their work product is not what a linguist would call a transcript.  Moreover, trial transcripts miss much of what counts as judicial behavior, even in the courtroom.

The only ethnographic solution would be to do what Bohannan and Gluckman did in Africa, and O’Barr and I did in small claims court:  pick some judges, spend days or weeks in detailed observation of their behavior in court (I can’t imagine getting into chambers), tape record it all, and prepare and analyze the detailed, multi-dimensional transcripts that discourse analysts use.  In terms of permission, the watching part would be easy, probably a matter of constitutional right (meddling Institutional Review Boards be damned).  The recording part would require the permission of each judge.  Based on previous experience (in both small claims and, in graduate school, “big” court), some would agree and some wouldn’t.  All would insist on permission from lawyers and litigants.  Perhaps surprisingly, all that I encountered agreed.

Again based on previous experience, it would be a long and demanding project, but by no means undoable.  As models, compare not only my small claims work but Beth Mertz’s 2007 book, The Language of Law School,⁹which analyzed discourse in several first-year law school classes around the country (and, as I think about it, is an excellent illustration of every point I’ve tried to make here).  For the reasons already advanced, I think such a project would be eminently worthwhile, although probably in surprising ways.  I doubt I’ll do it, but I hope someone will.

Reflections on the Conference

Being an anthropologist, I approached the conference itself as an ethnographic opportunity—I don’t know any better.  It was another day among the “natives,” in this case an array of federal and state judges, trial and appellate, and some academics who are interested in studying them.  Two themes stood out.

The first was the sharp distinction between trial and appellate judging.  Trial judges engage in complex conversations.  Many times these conversations are live, oral, and public, as when a judge conducts a trial, a motion argument, or a sentencing hearing.  The judge must not only apply the law in real time, but must also manage the human interaction.  Most trial judges realize that their job in an adversary system is to stay as far in the background as possible while letting the parties produce the trial.  The lawyers are frequently repeat players with whom the judge will have to deal again and again.  This task—“doing trial judging”—strikes me as remarkably challenging, both intellectually and sociologically.  And it must be done while looking over one’s shoulder, since the whole performance is subject to public review, (sometimes unfair) recharacterization, and (sometimes withering) critique.

Appellate judges, by contrast, spend most of their time manipulating written texts.  As they described it at the conference, their work consists largely of reading briefs, memos, cases, and statutes; drafting, circulating, and reading opinions; and attempting to persuade colleagues in written (and, increasingly, electronic) communications.  Live conversations in the form of oral arguments and conferences with colleagues are the exception.  With the exception of the oral arguments, the work of appellate judges is done in privacy and relative anonymity.  When they do perform publicly, they—in contrast to their trial colleagues—are expected to be the stars, interrupting and hectoring the lawyers with questions that sometimes seem intended to show off as much as to focus the arguments.

These functional differences brought home to me the point that studying judging encompasses two quite different projects.  To study the performance of trial judges is to study public behavior and public documents.    It is thus inherently doable, even if extraordinarily labor-intensive.  Appellate judges, however, do most of their work in a secret world that is seemingly impenetrable to ethnographers or others who rely on direct observation. One can study only the judgments and opinions they are required to release to the public; the performance that underlies these carefully crafted documents is immune to scrutiny.

The second observation derives from the first: judges have difficulty being anything but judges, and the trial-appellate distinction carried over into the conference discussion.  The trial judges, by and large, conversed.  They asked and answered questions and seemed eager to have external, scientific perspectives on what and how they are doing.  The appellate judges acted like, well, appellate judges (with apologies to the exceptions).  They monopolized the floor, interrupted, and sometimes went on, and on, and on.  There was nothing unpleasant about this; on the contrary, the whole event was unfailingly cordial. It was just that the appellate judges stayed in courtroom character.  When they discussed their behind-the-scenes work, they saw few problems that were worth studying. They have things under control (aside from the rare colleague who doesn’t work well with others and writes gratuitous dissents).  We academics will never get in, of course, but what would be the point?  (And academic idolatry of federal appellate judges only eggs them on.)

So I came away from the conference with mixed feelings.  Trial judges want to be studied, and there are many ways to do it, with ethnography well-positioned to play a role.  But at the appellate level, it seems that we will be limited to the analysis of outcomes.  As it usually does in legal scholarship, “empirical” will continue to mean “quantitative.”

Acknowledgments:

Copyright © 2010 Duke Law Journal.

John Conley is a professor at the University of North Carolina-Chapel Hill Law School

Researchers trying to rate and rank judges have high aspirations. Some suggest that their measures of quality can be used to select judges for advancement, to determine what backgrounds produce the best judges, and to identify models to whom fellow judges can defer. This response to their efforts has three parts. Part I critiques some of the work to date. (The critique is brief because it was the principal subject of a prior Duke symposium.¹) It argues that ranking—and even rating—judges is unlikely to produce the promised benefits; that the measures thus far used by social scientists miss the mark; and that those measures, to the extent that they influence judges, could encourage bad practices. Part II, the heart of this Essay, lists desirable qualities in appellate judges. Perhaps social scientists can find methods to measure these qualities objectively and accurately. Part III concludes, however, that even if social scientists cannot find such methods, efforts to identify the qualities of a good judge can be beneficial if they spur intelligent, respectful dialogue to encourage conscientious judges to improve their work.

I.
Rating and Ranking Judges

Based on my observations and experience, some of the hopes expressed for rating and ranking judges are quixotic. For example, it would be remarkable if the electorate or an appointing authority made decisions any differently if candidates for judicial advancement were ranked by social scientists. Once a candidate passes the “qualified” hurdle, decisions are more likely to be made based on political and interpersonal skills. Likewise, studies of those judges who are ranked highly are unlikely to reveal what backgrounds produce the best judges; my personal experience is that excellent judges come from all sorts of backgrounds. And although law clerks may tend to refer their judges to opinions by highly ranked authors, the judges themselves are unlikely to pay obeisance. Most judges are independent sorts, with sufficient egos not to be dazzled by prestige. They do not adopt an idea just because it comes from a renowned person; they need to be persuaded on the merits.

In addition, I question whether the task of rating judges can be done well, and I worry that defective rating methods can have adverse consequences. Recent research has attempted to develop objective measures of judicial quality. As some participants in this Symposium have observed, however, the research appears to be driven by the availability of data. That is, social scientists have limited themselves to data that can be collected from available sources and then convinced themselves that certain subsets of that data reflect judicial quality, enabling them to rank judges accordingly. But although such measures can be precise, they may not measure what is useful and may encourage questionable conduct, as has happened in response to ratings of educational institutions. For example, I doubt that a judge’s independence can be captured by counting the number of dissents or disagreements with colleagues (either all colleagues or only those of the same political party). A judge who disagrees may simply be close-minded and unable to persuade colleagues. Worse, if a judge’s reputation turns on the frequency of disagreement, those seeking to improve their stature may try to find grounds on which to disagree, rather than making an effort to find common ground. Or if one measures a judge’s contribution to the law by how often the judge is cited, judges (who are well aware that the best way to get cited is to be the first to opine on a subject) may be tempted to produce more dicta and address issues not presented by the parties.

But replacing objective measures by subjective ones is not the answer. Reputation is a measure that feeds on itself, at times with very little of substance to support it. The compliment “highly underrated” can be as useful in describing judges as football players. Evelyn Waugh once observed that one can acquire a great reputation by being “dogmatic, plausible, and vain.”² Judicial reputations are hardly immune from questionable influences.³

II.
Judicial Quality

As the organizers of this Symposium have suggested, rather than starting with measures for which there are data and then deciding what those measures say about quality, perhaps one should reverse the process, deciding what constitutes judicial quality and then exploring how to measure those constituents. This Section responds to that suggestion with a list of what I believe to be desirable qualities in an appellate judge. Others may use rather different measures. Academics may be most concerned with whether the judge’s opinions are good teaching tools and raise interesting ideas. Attorneys may be most concerned with whether their clients win. Politicians may like judges whose opinions reach results that achieve wide popularity.

My perspective, of course, is that of an appellate judge. I will try to answer the question: “What do you look for in a fellow judge?” I am sure that to some extent my views are idiosyncratic. But I believe that each of my criteria would receive the support of a large fraction, perhaps a majority, of appellate judges.

Before I set forth my criteria, I would point out one possible criterion that is absent: to be a good appellate judge, a colleague does not have to agree with me. Any appellate judge with only a modest amount of experience would recognize that if “agrees with me” is a necessary criterion for a good judge, then only one judge would meet the standard. The simple truth is that no judge’s colleagues are as insightful and wise as the judge himself or herself. Criteria should be attainable in the real world.

I will group my criteria under four headings: Treatment of Colleagues, Treatment of Litigants, Treatment of the Law, and Treatment of the Institution. My order of discussion is not meant to suggest order of importance.

A.     Treatment of Colleagues

A good judge treats colleagues fairly. If it is true that ninety percent of life is just showing up, then treatment of colleagues belongs in the other ten percent. Just showing up won’t cut it. Colleagues should do their share of both the glamorous and the nitty-gritty work of the court. The part of a judge’s work to which the public pays most attention is the production of published opinions. But a judge who focuses only on producing published opinions is not a good colleague. To be sure, most judges would like to devote a greater percentage of their time to working on published opinions that raise challenging issues (although taking a break to resolve a routine case can often bring a welcome sense of accomplishing something). But writing a published opinion in a case that does not warrant it (because it says nothing new) wastes paper and the time of those who feel a need to read it. More importantly, if a judge shirks all duties except the preparation of published opinions, the other, less-interesting duties are shared disproportionately by the judge’s colleagues. These duties include participating in calendars of cases that are likely to result in unpublished opinions, disposing of motions, and serving on one of the many committees that handle court rules and administrative matters. Judges have different interests and may prefer some of these tasks to others. Which they select is immaterial; the important thing is to do one’s share.

B.     Treatment of Litigants

The first duty of appellate judges is to decide the cases before them. The process of decisionmaking must be fair, and appear to be fair, to the litigants. The fairness of treatment on appeal, unlike the fairness of treatment at trial, cannot be measured on the basis of face-to-face encounters. Appellate judges rarely are seen by the parties to a dispute; and most of the time they are not seen even by the parties’ attorneys (and then only for a few minutes, in the highly artificial setting of oral argument). Thus, fairness must be assessed by reading opinions. An opinion reflects fairness to the litigants when it has the following features:

1.   Takes Care with the Facts

The factual context of a case should be stated fairly (including “unfavorable” facts) and accurately. If a fact is worth incorporating in an opinion, it is worth setting forth correctly. Not all facts in an opinion are critical to the resolution of the dispute; but it is important to check the accuracy of even background facts, because an opinion that treats facts cavalierly will suggest that the court has been inattentive to the case. Perhaps the easiest way for a losing attorney to convince a client that the court did not give fair consideration to the client’s contentions is to point out that the court omitted important facts or did not get the facts straight.

2.   Sticks to the Record

An appellate court should decide the case on the record produced in the lower court. When the court goes outside the record, it relies on matters that the parties had no opportunity to contest and that the lower court had no opportunity to take into account. To go outside the record to resolve a dispute will thus offend the sense of justice of the party injured by the practice, and will also likely frustrate the trial court that is being reversed on grounds not presented to it.

3.   Addresses the Parties’ Contentions

A party will not feel that the court has acted fairly if the court does not address the party’s contentions on appeal. This is not to say that every contention must be resolved on the merits. Often a contention is mooted by the resolution of another issue. The court also may refuse to consider a contention because it was not preserved below or not properly presented on appeal. In addition, unfortunately, some briefs are so poorly prepared that the court must characterize the party’s contentions differently than the party did, and thus the issues addressed in the opinion may not match the party’s listed contentions. A good judge, however, will take care that the opinion recognizes all properly presented arguments. I can understand why a court may occasionally write something like, “We have reviewed appellant’s other arguments and none has merit”; but I am not fond of the practice. At the least, I would want to state that the appellate court substantially agrees with the lower court’s analysis of the issue.

4.   Is Evenhanded

A judge should be evenhanded, applying the same rule of law in a similar fashion to all parties. A judge may be either strict or lenient about what it takes to preserve an issue; but the judge should not be lenient when considering preservation by a personal-injury plaintiff and strict when considering preservation by a personal-injury defendant. Similarly, a judge may have lenient or restrictive views on when parties are entitled to standing; but the judge should not be lenient for those exploiting mineral resources and restrictive for those opposed to such exploitation. Of course, there are exceptions to the general rule. Pro se litigants should be granted some leeway, so long as the court does not become their counsel. And the government can be expected to toe the line more closely than others. But lack of evenhandedness can be the most telling indication of appellate bias against an individual litigant or category of litigants.

C.     Treatment of Law

Although an appellate court’s first duty is to decide the case before it, the legal doctrine set forth in the court’s opinion is almost always the most important consequence of the court’s decision to society as a whole. It matters little to the general reader of the opinion whether the court erred in its decision because it got the facts wrong, went outside the record, or failed to consider an argument by the losing party. A distinct set of considerations governs whether a judge does a good job in setting forth legal doctrine. A judge’s opinions should satisfy the following criteria:

1.   Describe the Case Law Honestly

Even when precedent does not determine the result in the case, prior case law is likely to impose significant constraints or be persuasive. The reader of an opinion should be able to assume that the author accurately describes the facts and holdings in cases cited in the opinion. Of course, progress (or at least development) in the law often occurs when precedents are re-examined and recharacterized, and a “better” explanation is then provided for the results in those cases.⁴ But there is no reason for a court today to be deceptive about what it is doing in that regard.

2.   Help Develop the Law

A good judge contributes to the development of the law. I do not mean that the good judge always “gets it right.” Early in my career I agonized about whether I was stating the law absolutely correctly. Then I realized that I simply do not have the experience and intelligence to come up with all the relevant considerations, much less wisely evaluate them, in deciding how to frame a rule of law. All I could expect from myself was to work as diligently and intelligently as I could on the matter. I can hardly expect more from others. But even if the judge does not get it right, the judge can make a contribution by sharing the product of his or her diligence and intelligence. Perhaps the most important component of this contribution is clarity. Rather than just pronouncing the result, the opinion should clearly explain how the court arrived at its conclusion. If the judge believes that there are three important considerations supporting the conclusion, the opinion should recite those considerations and explain how they interact to compel the result. Others may later show that there are really four important considerations, but the judge who came up with the first three has made the analysis easier for those who follow. I value judges who advance the law by sharing the product of their diligent, intelligent efforts, even when I disagree with their conclusions. One of the joys of appellate judging is the interchange of ideas that leads to a better opinion than any single judge on the panel could produce; this interchange can be among judges who agree on a result, but the best work often comes when there is a dissenting voice.

3.   Be Consistent

A judge should be consistent in the process of arriving at doctrine. For example, does the judge have a consistent practice in deciding whether to overturn a precedent or to apply stare decisis? A judge could consistently decide to overturn precedent whenever the judge believes the precedent to have been poorly reasoned; but then the judge is not entitled to rely on stare decisis doctrine to criticize fellow judges for overturning a precedent that they believe to have been poorly reasoned. If a judge bases a constitutional doctrine not on the specific language of the Constitution but on the structure created by the document, the judge cannot criticize others for using the same methodology to reach a result opposed by the judge. Likewise, judges should be consistent in applying canons of statutory construction and in using the results of research in the hard and soft sciences.

D.     Treatment of the Institution

The authority of courts in this country is founded on the reputation of the judiciary. An appellate judge has a duty to contribute positively to that reputation. The judge’s work may well advance the reputation of the judge, but it should not do so at the expense of the courts themselves. A few thoughts on what makes judges good in this respect:

1.   Persuasiveness

Courts may have the power to rule however they wish. But their legitimacy is based on the persuasiveness of their opinions. Accordingly, I value colleagues who write coherently, logically, and convincingly.

2.   Respect for the Courts and the Reader

Persuasiveness of opinions may be the most important factor in establishing the legitimacy of judicial decisions, but it is not the only one. The style and tone of an opinion can also advance or detract from the prestige of the deciding court and of the judicial system as a whole. Opinions that carp at or demean other judges or their work can only cause the public to adopt a similar attitude. And attempts at eloquence that degenerate into bloated prose can provide ammunition for the view that the courts have lost touch with the community.

3.   Modesty

The purpose of writing opinions is not to create a reputation for the author. Grand pronouncements, declarations of “new” legal principles, and treatise-like discussions that go beyond the needs of the case may well establish the author’s brilliance. But they are at least as likely to be examples of “writing more than one knows”⁵ and thus provide numerous opportunities for later judges to correct errors and cabin dicta. Such writing can also create an aura of judicial willfulness. Judicial craftsmanship often consists in explaining how the law expressed in an opinion follows naturally from prior case law. The more an author signals that the law expressed in the opinion is essentially the creation of the author’s brilliance, the less the reader will be convinced that the court’s work represents the rule of law rather than the rule of persons who happen to be judges.

III.
Measuring Quality

Now that I have listed what I believe to be attributes of a good appellate judge, one might ask what good the list does. Perhaps some of the listed attributes can be measured objectively. After all, experts have developed ways of determining how much time it takes trial judges to perform various tasks; and the time may come when someone believes it possible to measure whether an appellate judge is doing a fair share of the court’s work. Also, measures that are not totally objective may be developed to evaluate whether a judge treats litigants evenhandedly or develops legal doctrine in a consistent manner. I have my doubts. Yet even without such measures, a list of desirable qualities can serve a useful purpose. The most productive engine for improving judicial performance is the conscientiousness of those who wear robes. If judges and the consumers of their work can exchange views about desirable qualities for appellate judges, then judges who care about their craft (who comprise the great majority of the profession) can consider those views and conduct periodic self-evaluations. I frequently see my colleagues borrow good practices from one another, often without any discussion between them. I am confident that as judges engage in conversation and introspection regarding quality, we will continue to improve the way we do our jobs.

Acknowledgments:

Copyright © 2010 Duke University Law Journal.

Harris Hartz is a Judge on the U.S. Court of Appeals for the Tenth Circuit.

One kind of empirical study of judicial decisionmaking might be regarded as continuous with the broader goal of social science, which I take to be something like understanding human behavior in general. I see the ultimate aim of this sort of empirical inquiry as developing a theory that explains and predicts judicial decisionmaking in roughly the same way a psychological theory might seek to explain and predict other observed phenomena of human behavior, such as the tendency to obey authority. Just as a successful psychological theory of obedience might, among other things, identify the conditions that explain why and predict whether a given subject will obey an order given by an authority figure in a particular context (for example, personal characteristics of the subject, the subject’s relation to the authority figure, the nature of the order, and its expected consequences), one might likewise consider an empirical theory of judging in this vein successful if it allows particular conditions to be identified—for example, political ideology, characteristics of the litigants, particular features of a case’s history, or the provenance of relevant precedent—that explain and predict judicial outcomes.³

Within this broadly defined empirical project of explaining and predicting judicial decisionmaking as human social behavior—one might call it the project of “naturalizing jurisprudence”⁴—there are multiple theoretical perspectives that might be relevant. A single type of observed judicial decisionmaking might be understood simultaneously through the frameworks of sociology, political science, social psychology, cognitive psychology, and perhaps even neuropsychology. Whether or how all these empirical perspectives might be integrated remains unclear. Presumably, a grand unified theory of judicial decisionmaking is no more and no less likely than a grand unified theory of human behavior in general. All of these scientific perspectives may be viewed as having one common, general aim: they seek to provide causal explanations of judicial decisions—theories that identify the causal predicates of observed decisions and do so with predictive power.⁵

A second type of empirical study of judicial performance seems quite different in its basic aim from the project of naturalizing jurisprudence. The goal of this second type of study might broadly be described as identifying quantified measures of good judicial performance—for example, citation counts, dissent rates, and productivity—that can be used to assess and even rank the quality of sitting judges, judicial candidates, and courts. This type of undertaking can be seen as a way of compiling otherwise inaccessible information that would presumably be of significant value to public officials, the citizenry, and judges themselves in evaluating and monitoring judicial performance. Public ratings of judges and courts based on this information might have the added desirable effect of sussing out the opaque criteria that various political actors use to champion particular judges or candidates, insofar as those ratings would exert pressure on such actors to give public explanations supporting any low-rated candidates they seek to promote.⁶

What is the relationship between these two empirical projects of naturalizing jurisprudence and of measuring judicial performance? One possibility is that the need for objective measures of judicial performance is a function of the current infancy of the science of identifying the causes of judicial decisions. That is, objective measures that serve as proxies for judicial quality are only necessary because of the lack robust theories that would predict how a particular sort of judge would likely decide a particular sort of case. If such a theory existed, there would be no point in trying to develop any general measure of judicial quality.

An example may help draw out the intuitive appeal of this conjecture. Suppose one could prove that in any case possessing the set of features F, involving a party with characteristics P, a second party with characteristics D, and given additional specifiable conditions C, a judge with a set of characteristics J will always decide the case in a way that is favorable to the party with characteristics P, whereas a judge lacking J will always decide against that party. One might argue that this postulate, if true, would undermine the relevance of any generic notion of judicial quality—apart from whatever constitutes J characteristics—in the context of cases with features F. If one could predict how any particular judge would decide this type of case, there would be no need for further information about the judge’s qualities, at least in that limited context. And if analogous predictions could be made with respect to cases with features F1, F2, and so on, and judges with characteristics J1, J2, and so on, there would be no more reason to care about measuring judicial quality in the context of those cases. Thus, the more progress empiricists make in the project of reducing judicial decisionmaking to its natural causes, the less relevance any project of measuring judicial performance will have.

One might argue, in other words, that judicial quality matters because better judges presumably will make better decisions. But if it were possible to predict judicial decisions in the manner postulated, then no one would have reason to care about generic measures of judicial performance. There would be no reason to fret over proxy measures of good judicial decisionmaking if social science could deliver a theory that directly predicts how a particular judge or candidate would decide particular kinds of cases.

I believe that this argument should be rejected. This argument’s fallacy involves its reliance on the implicit assumption that empirical measures of judicial performance are, at their core, nothing more than an indirect attempt to accomplish one of the goals of the project of naturalizing jurisprudence—namely, developing a theory with the power to predict the outcome of judicial decisions on the basis of specifiable causal predicates. But the project of measuring judicial performance need not be assimilated to that of theorizing the causes of judicial decisions. Rather, the project is fundamentally normative and evaluative in character. The basic question is not about the causes of decisions, but about what makes a good judge, or what constitute the basic virtues of a good judge.⁷ Whereas the naturalizing project seeks to provide causal explanations for judicial behavior in a manner continuous with social science, the empirical study of judicial performance seeks to make explicit and then reduce to numbers our value judgments about the relative merits of selected characteristics of judicial performance. The ultimate test of a causal theory is its explanatory and predictive power. The test of a measure of judicial performance is ultimately the normative plausibility of its embedded value judgments about the core virtues of a judge and how well it captures those judgments.⁸ The two projects have incommensurable aims.

There are, however, some important caveats. I do not deny the potential relevance of the findings of naturalized jurisprudence to the project of measuring judicial performance. For example, some observers might think that any evaluation of judicial performance should incorporate criteria that capture something like political independence or capacity for “nonideological” decisionmaking.⁹ But what if there were empirical evidence that indisputably established that, as a matter of fact, political affiliation almost always predicts judicial outcomes in certain types of cases? I suspect that if that were the case, there would be reason to doubt whether criteria aimed at measuring political independence could possibly capture anything meaningful. Standards of judicial quality must be tempered by contemporary knowledge regarding the limits of human psychology. This is a consequence of the basic moral premise that “ought” at least in some sense implies “can.” To that extent, the study of the causes of judicial behavior is potentially relevant to the project of measuring judicial quality. My point is not to deny this possible point of congruence, but rather to emphasize that society’s concept of a good judge is not simply given by how most judges in fact tend to behave. Findings in the science of judicial behavior cannot themselves determine the normative standards by which judges should be measured and evaluated. This is a consequence of another basic moral premise, namely, that “is” does not imply “ought.”

The other caveat is that my remarks assume that it is possible to construct a model of a good judge that is at least to some degree independent of considered preferences relating to case outcomes. This assumption means, among other things, that a judgment about whether a particular individual would make a good judge is not simply reducible to a set of predictions about the outcomes of cases that would come before that individual. But what if one were to reject this assumption? What if the concept of a good judge that best reflected societal and legal norms did in fact turn out to be nothing more than a reflection of collective preferences about case outcomes?¹⁰ In that case, I do think that measures of judicial performance that captured the concept might be collapsible into predictions of judicial behavior. That is, if society’s notion of a good judge turns out to be nothing more than a set of predictions about the likelihood of a judge’s reaching particular outcomes in particular cases, then measures of judicial performance would be nothing more than proxy predictions about what judges would probably do in such cases. It might follow, then, that my claim—that the project of measuring judicial performance is fundamentally distinct from the project of determining the causes of judicial decisions—depends on the defensibility of a particular kind of concept of a good judge, namely, one that is not tied to the desirability of specific outcomes in particular cases.

I do not think that this conclusion does, in fact, follow. Even if this sort of cynicism about the concept of a good judge were warranted, I believe that the project of naturalizing jurisprudence would still remain fundamentally different from the project of quantifying judicial performance, because the latter is essentially a normative endeavor in a way that the former is not. Even if every theorized measure of judicial performance turned out to be nothing more than an elliptical predictor of a particular set of case outcomes, every theorist’s proposed measure of judicial performance would still be answerable to questions about its underlying normative conception of a good judge. Empirical theories about the causes of judicial decisions need not answer these questions.

Pretend, for example, that one could show that measures of judicial performance that depend on citation counts tend to highly rank judges who are more likely to invalidate legislation in federal constitutional cases. If that were true, then commentators might observe that these performance measures are empirically reducible to predictors of how judges will decide those kinds of constitutional cases. This finding would not, however, undermine my central argument, which is that the project of empirically measuring judicial performance, in contrast to the project of identifying the causes of judicial decisions, is fundamentally normative. The discovery of a sufficiently tight predictive correlation between citation counts and a disposition to invalidate legislation might support a claim that any interest in the former as a measure of judicial quality is really nothing more than an indicator of a preference for the latter outcome. But the question that this discovery would not and could not answer is whether anyone should therefore stop using the citation count measure as a benchmark for judicial performance. Arriving at an answer would require normative discussion about principles of judicial evaluation—principles that specify why certain considerations should count as legitimate reasons for or against appointing someone as a judge. Any given measure of performance may collapse into a prediction about substantive case outcomes. This possibility does not show, however, that the measurement project itself collapses into a predictive one. Whatever an empirical theory of judicial performance might in fact be measuring, it must always answer one normative question that a purely predictive one need not answer: should the measures in question form the basis for evaluating judges? For this reason, the project of measuring judicial performance is inescapably normative in a way that the goal of naturalizing jurisprudence is not.

Studies of judicial performance that seek to determine judicial quality by quantitative measures (such as citation count) will ultimately stand or fall on the strength of the normative reasons that can be marshaled for valuing as judges the kind of individuals who do well on those measures. Empirical measures of judicial performance ultimately depend on normative claims about what it means for someone to be a good judge, and the strength of any proposed empirical measure is necessarily a direct function of the strength of the justification of those normative claims. Theories about the causes of judicial performance do not depend on these justifications. And no purely empirical project can supply those justifications. That is a task for normative, not empirical, inquiry.

Acknowledgments:

Copyright © 2010 Duke University.

Patrick S. Shin is an Associate Professor at Suffolk University Law School

Larry Alexander and I have written at some length about the function of rules in legal decisionmaking.¹ Briefly, we argue that, from the central perspective of someone designing a legal system, it is desirable that at least some range of legal disputes be governed by general, determinate rules, and that these rules be treated as authoritative. To treat a rule as authoritative is to follow it in all cases that fall within its terms, without further analysis of the underlying purpose or rationale that motivated the rulemaker to announce the rule. From a systemic perspective, authoritative rules are attractive for several reasons. Rules, if they are generally obeyed, can help their subjects avoid errors and biases that distort case-by-case judgment. Rules can also provide coordination if all or most subjects treat them as authoritative. Individuals can act more effectively if they know what others are likely to do, and an authoritative rule makes it possible to predict others’ conduct in situations governed by the rule.

At the same time, the generality of rules often leads to overinclusiveness in application. As a result, rules will sometimes prescribe outcomes that are wrong, judged by the rules’ own purpose or rationale. Yet the overinclusiveness of rules does not alter the conclusion that rules function most effectively when treated as authoritative. If a rule is well designed—that is, if the rule will prevent more errors overall than it will cause through overinclusiveness—then it is best, from the perspective of the system designer, that those who are subject to the rule follow it in every case to which it applies.²

The difficulty with authoritative rules is that it is not at all clear how rational subjects can treat rules as authoritative. If a rule appears to give the wrong answer in a particular case, the rational choice is to disobey. The subject may understand that rules prevent errors of judgment, but if the subject also understands that rules are overinclusive, the subject may nevertheless conclude that his judgment is correct. The subject may also understand the coordination value of rules, and yet conclude that the error that will result from following a rule in a particular case outweighs the loss of coordination that will result from a single act of disobedience. In each case, the subject may be wrong, but this does not make it subjectively rational to follow the rule.³ The consequence is an unbridgeable gap between the perspective of those who oversee the legal system and the perspective of those subject to the system’s rules.

Judges occupy two positions in this account of legal rules. First, they act as rulemakers. Judicial opinions often contain prescriptive statements that are determinative enough to count as rules and are intended to serve as decisional standards in future cases. Assuming the rules are well designed in the sense described above (assuming, that is, that the rules will prevent more errors than they generate over the long run), the legal system will benefit if future judges, as well as future actors, treat them as authoritative: judicial rules, no less than legislative rules, can reduce errors and provide coordination.

In addition to serving as rulemakers, judges also apply rules to the cases litigants bring to court. When judges apply rules, their perspective is similar to that of rule subjects. If a judge is persuaded that the governing rule provides the wrong answer (as indicated by the rule’s underlying rationale) in a particular case, applying the rule will seem both irrational and unfair. Moreover, the judge may find it difficult to give due recognition to the long-term advantages of the rule, because the judge’s attention is focused on the specific facts of the case.

This view of rules and the roles they play in judicial decisionmaking leads to a number of empirical questions about how judges respond to rules. The most basic question is whether judges treat rules as authoritative. Do judges apply rules only when they are persuaded that the outcome of the rule in the case at hand is consistent with the rule’s purpose or rationale (or is correct, all things considered)? If so, the answer to the basic question is no; rules have no real consequences for adjudication, and the various Legal Realist speculations about what drives judicial decisionmaking come into play.

Assuming that judges sometimes do follow rules simply because they are the rules of the system, the next question is how deeply judges are committed (or how thoroughly they are habituated) to following rules. Do judges follow rules without reflection only when the purpose or rationale of the rule has not been brought to their attention? Do they refuse to consult the rationale or purpose of an established rule if the terms of the rule apply uncontroversially to the case at hand? Do they also follow rules in conscious disregard of the rule’s purpose or rationale, when the rule’s purpose or rationale appears to conflict with the result it prescribes?

If the answer is that judges who follow rules do treat the rules as authoritative against their own best judgment about what outcome is best, subsidiary questions arise about the conditions under which judges are willing to suspend judgment and follow rules. For example, the rule in question may be defective, in the sense that it generates more errors than it prevents overall. Alternatively, the rule may be sound, in that it prevents more errors than it generates overall, and yet misfire in a particular case. Here, the question for investigation is whether differences in the overall quality of the rule affect the level of judicial obedience to the rule.

Another subsidiary question is whether differences in the vividness of the facts presented for adjudication affect judges’ responses to precedent rules. Here, the variable is not the quality of the rule, but the salience or affective impact of a seemingly wrong outcome required by the rule.⁴ The question for investigation is whether judges approach rules consistently in different factual settings, or, alternatively, whether their treatment of rules varies according to the exigencies of particular cases.

These questions are obviously hard to answer. Written opinions may provide some evidence of judges’ attitudes toward rules, but only when judges perceive the dilemma of rule-following and also have reason to discuss it in the course of explaining their decisions. This set of circumstances may be rare; in any event, opinions that directly address rule-following are not likely to yield a representative sample of judicial responses to rules.

A more promising approach might be to conduct controlled experimental studies on judicial subjects. The logistics of this form of research are daunting, but work by Chris Guthrie, Jeffrey Rachlinski, and Andrew Wistrich suggests that it can be done.⁵ As an illustration, consider the following stylized example. (Many refinements would be needed to produce a workable test.) A group of judges is presented with a problem case. The defendant is a retired veterinarian who has just moved to a new neighborhood. In his backyard, he keeps a pet giraffe. The defendant treats the animal well and provides it with ample space. The giraffe is a quiet animal with no violent tendencies, and it has no negative sensory effects on neighbors. The defendant kept the giraffe for several years in another state without incident or complaint. Neighbors have sued, claiming the giraffe is a nuisance and requesting an order requiring the defendant to remove it from the neighborhood.

To capture variations in legal rules, some judges might be told that prior cases establish that keeping a wild animal in a residential neighborhood is a nuisance per se. This is, presumably, a sound rule—over the long run, it will tend to prevent harm and offense, though it may sometimes result in the elimination of unobjectionable pets, such as the defendant’s giraffe. Other judges might be told that prior cases establish that keeping a large animal in a residential neighborhood is a nuisance per se. This rule looks too broad: it may do more harm than good over the range of cases to which it applies. Judicial subjects would be asked to decide the case and possibly to comment on the outcome. A significant number of decisions in the plaintiffs’ favor might suggest that rules influenced the judges’ decisions, particularly if the judges signified discomfort with the outcome of the case. If results were roughly the same for both versions of the problem, the inclination to follow rules would seem impervious to the quality of the rule. If, however, outcomes varied according to the version of the precedent rule presented to  judges, this would suggest that judges deciding whether to follow rules are likely to discriminate according to the quality of the precedent rule.

The facts of the experimental problem could also be varied to detect the influence of vivid facts. For example, some judges might not be given any personal information about the defendant; others might be told that the defendant rescued the giraffe from sordid conditions in a private zoo. In this case, variation in decisions would indicate that judges deciding whether to follow rules discriminate according to the salience of the factual consequences of applying the rule.

A second set of questions relates to the role of judges as rulemakers. Frederick Schauer has suggested, very plausibly, that the quality of judicial rules may suffer as a result of the adjudicatory context in which judges work.⁶ In particular, a judge focused on the salient features of a litigated case may adopt a rule that responds to those features, without adequately considering the effects of the rule in other cases that fall within its terms. The judge may assume, in other words, that the present case is more typical than it is—more representative, as a cognitive psychologist might say—of cases governed by the rule. A tame and odorless giraffe, for example, might not typify wild animals kept as pets.  If so, the rule is likely to prove unsatisfactory over time.

The initial question is whether Professor Schauer’s suggestion—that judicial rules may be adversely affected by the adjudicatory setting from which they emerge—is correct. If so, a further question is whether certain features of the judicial process can help avoid or decrease the distorting effects of vivid facts. For example, are judges less likely to err in their choice of rules when the parties present them with an array of purportedly analogous cases that might arise under a proposed rule?⁷

It is surely possible to find examples of this pattern in decided cases, but decided cases are unlikely to provide a systematic overview of the problem. Again, the most promising alternative is probably experimental. Judges might be given the facts of a case together with a selection of proposed decisional rules, the overall quality of which varies inversely with the attractiveness of their outcomes in the case at hand.

Empirical information about judicial responses to rules would be of great utility in studying the judicial process. Obtaining reliable information would, of course, require tremendous empirical ingenuity: designing effective problems is a challenge. Eliciting a serious but candid response from judges is an even greater challenge: judges must somehow be enticed to do what they ordinarily do, without too much reflection on the nature of the exercise. Fortunately, the objective of this short Essay is only to identify the types of information that would be most helpful to theorists interested in decisionmaking in a regime of legal rules.

Acknowledgments:

Copyright © 2010 Duke Law Journal.

Emily Sherwin is a professor at Cornell Law School.

Some of the discussions at the Symposium highlighted an area where academics’ perceptions of judging conflict with judges’ actual experiences. Judges and academics view the significance of judicial dissents quite differently. Whereas many of the judges believe that dissents primarily reflect the level of cohesiveness and collegiality of the court, academics typically place much more significance on the meaning of judicial dissents. For example, recent academic studies have asserted that judicial dissents often reveal the influence of judges’ retention concerns,¹ the level of judges’ independence,² or certain judges’ higher propensity for risk taking.³

In contrast, many judges at the Symposium considered academics’ emphasis on judicial dissents to be misguided. The judges maintained that dissents reveal less about judges’ retention concerns, impartiality, or risk preferences, and more about the culture of collegiality on the court. Many of the judges believe that dissents primarily reflect the level of cohesiveness among judges, and that various factors and institutions influence this cohesiveness. For example, they explained that the personal relationships, amount of professional and social interaction, and diversity among judges on a court might influence the level of dissent. Many of the judges hypothesized that courts that are more collegial, either because the judges are better friends or share more common values or backgrounds, should have lower dissent rates. Judges on these courts should be less likely to openly criticize the opinions of their colleagues by dissenting.

Although academics have long recognized that institutions such as opinion-assignment procedures and voting order might influence the propensity to dissent,⁴ empirical studies have failed to consider the impact of collegiality and personal relationships on dissent rates. Thus, in this short Essay, I empirically test whether some of the judges’ assertions are consistent with the data. I test whether various measures of diversity are associated with dissent rates in state supreme courts. I find that diversity in many areas—gender, race, age, religion, home state, and political affiliation—is associated with higher levels of dissent. In contrast, diversity in the jobs that judges had before taking the bench is associated with lower dissent rates.⁵

I also test whether the length of time judges have served on the court is associated with dissent rates. Presumably, judges that have served on a court together for many years would have stronger friendships than newer judges, and thus may be more collegial and less likely to dissent. However, my empirical analysis finds the opposite: the greater the number of judges with lengthy tenures on the court, the higher the dissent rate.

Empirical Analysis

To test the influence of diversity and tenure on judges’ propensity to dissent, I use data from the State Supreme Court Data Archive. This data includes an almost universal sample of state supreme court cases in all fifty states from 1995 to 1998. The data include more than twenty-eight thousand decisions involving more than 470 individual state supreme court justices.⁶ The data include variables that reflect case histories, case participants, legal issues, case outcomes, and individual justices’ behavior. I supplemented these data with institutional variables that describe aspects of each state’s judicial system, and with detailed information about each judge’s personal characteristics, background, and career.

Thus, my data consist of individual judge-level votes in each case before the state supreme courts. I use an ordinary probit model to test whether diversity and tenure on a court influence judges’ likelihood of dissenting. The dependent variable in my estimation is an indicator variable for whether an individual judge casts a dissenting vote in each case.

My estimation includes several measures of the diversity of judges’ personal characteristics and backgrounds on each court. I measure the diversity of each personal characteristic on each court with an index that is essentially one minus a Herfindahl index of each characteristic:⁷

Diversity of personal characteristic = 1 − _j (# of judges of each type j / total # of judges)².

For example, diversity along the racial dimension would be measured with:

Diversity of race = 1 − _j (# of judges of each race j / total # of judges)², where j = [White, African American, Asian, Hispanic, and other].

The diversity measure ranges from zero, when the court is composed of judges of only one “type” (that is, race), to one, when each type is represented equally on the court. Thus, increases in this measure indicate an increase in diversity on the court.⁸

I include variables that measure diversity along several different dimensions: gender, race (White, African American, Asian, Hispanic, and other), age (under 45, 46 to 55, 56 to 65, and over 65), religion (Protestant, Catholic, Jewish, Greek Orthodox, Mormon, and other), home state (from the state where the court is located or from another state), political party affiliation (Republican, Democrat, or Independent), and job held before taking the bench (prosecutor, attorney general, elected official, lower-court judge, or nonelected official).

I also include two variables that represent judges’ tenure on the court. I include the percentage of judges on each court that have served for one year or less; presumably, these judges have not served long enough to develop strong personal relationships with other judges. I also include the percentage of judges on each court that have served for six years or more; if these judges have developed stronger friendships, they may be more collegial and less likely to dissent.

In addition to the diversity and tenure variables, the estimations include a series of judge-level, case-level, and state-level variables that might be related to judges’ propensity to cast dissenting votes. The judge-level variables include an indicator for whether a particular judge is the chief justice on the court, and a variable indicating the number of years until the judge’s next retention.⁹ These variables control for voting changes throughout a judge’s career and term.

All estimations also include various case-level variables that may be related to dissenting votes. First, I include indicator variables for whether a case is a criminal case, a juvenile case, a civil case involving the state government, or a civil case involving private individuals. Thus, the base category is nonadversarial cases, such as cases involving certification and advisory opinions. Finally, I include indicator variables for whether at least one litigant is a business, a person, or a representative of the state government.  These indicator variables control for any relationship between dissent rates and the litigants or legal issues in a case.

Next, I include various state-level characteristics that have been found to be related to dissenting votes. First, I include an indicator variable for whether judges in the state face reelection by the voters. In a previous study, I found empirical results suggesting that judges’ reelection concerns are important influences on their propensity to cast dissenting votes.¹⁰

I also include a variable that indicates whether the state has a lower appellate court, and thus, whether the court has discretionary review to hear cases. Numerous studies report that the presence of an intermediate appellate court increases dissent rates, suggesting that discretionary dockets facilitate the expression of dissent.¹¹

The state-level variables also include indicator variables for whether a court utilizes a random opinion-assignment procedure instead of a discretionary procedure, and whether voting takes place in the order of seniority. Random or rotating opinion-assignment procedures prevent judges from being rewarded or sanctioned for their opposition votes, and thus reduce the incentives for consensus.¹² In contrast, when opinions are assigned by the chief justice, opportunities for rewards and sanctions emerge. Similarly, when voting takes place in order of seniority, the most senior judges may influence junior judges, reducing the likelihood of their disagreement.¹³

Moreover, all estimations include year indicators to capture trends in the likelihood of dissent. In the probit estimations, the t-statistics are computed from standard errors clustered by case.

Table 1 reports the primary probit results. In this table, the top number in each cell is the regression coefficient, which indicates the magnitude and direction of the relationship of each variable with judges’ votes. A negative coefficient indicates that a variable reduces the probability that a judge will cast a dissenting vote; a positive coefficient indicates that a variable increases the probability that a judge will dissent. Under each coefficient is the corresponding t-statistic. Coefficients with t-statistics equal to or greater than 1.96 are considered statistically significant at the 5 percent level, meaning that there is 95 percent certainty that the coefficient is different from zero. A t-statistic equal to or greater than 1.96 is typically required to draw conclusions in hypothesis testing.

The results indicate that diversity among the judges hearing a case is significantly related to the propensity to dissent. The positive and statistically significant coefficients on the diversity variables for gender, race, age, religion, home state, and political affiliation indicate that diversity along these dimensions is associated with higher levels of dissent. These results are consistent with the Symposium judges’ belief that dissent reflects lack of cohesiveness on a court; greater diversity probably implies that the judges have fewer common values and experiences, reducing the incentives for collegiality.

In contrast, diversity in the jobs that judges held before taking the bench is negatively associated with dissent. This suggests that more diversity in professional backgrounds is associated with a reduction in the propensity to dissent. Thus, different backgrounds, at least in terms of the judges’ professional lives, do not appear to reduce collegiality on a court.

Moreover, the results indicate that the more judges with very short tenure on the court (one year or less), the less likely judges are to dissent. Similarly, the more judges with lengthy tenures on the court (six years or more), the more likely judges are to dissent. If lengthy tenure is a good proxy for strong personal relationships among judges, then these results are inconsistent with the experiences of some of the judges participating in the Symposium. Instead of lengthy tenures reducing the propensity to dissent because judges are better friends, my results indicate that lengthy tenures increase the propensity to dissent. Nevertheless, the results are consistent with studies that assert that junior judges feel pressure to not dissent against more experienced senior judges.¹⁴ Thus, even if judges build stronger personal relationships during their time on the court, their inclinations to agree with friends may be outweighed by the confidence and experience that comes with longer tenure on the court.

Several other variables also have statistically significant relationships with the propensity to dissent. Judicial elections, discretionary dockets, random opinion assignment, and seniority voting are associated with increases in dissent rates. In contrast, an approaching retention and chief justice status appear to reduce the propensity to dissent.

Conclusion

In this short Essay, I tested whether the empirical evidence is consistent with the experiences of some of the judges participating in the Duke Law Journal’s Symposium on Evaluating Judging, Judges, and Judicial Institutions. Although my results confirmed that institutional and political factors are important influences on dissent rates, I also found that the levels of cohesiveness and collegiality among judges are important. These results indicate that academic studies of judging can greatly benefit if academics consider the real-world experiences of judges.

Table 1

Notes: The table reports coefficients from a probit model. For brevity, the indicator variables for years, case types, and litigant types are not reported here. T-statistics are reported in parentheses. An asterisk represents significance at the 5 percent level.

Acknowledgments:

Copyright © 2010 Duke Law Journal.

Joanna Shepherd is an Associate Professor at Emory Law School.

Before beginning, we should note that we paint with a broad brush here and likely fail to give credit where it is due. This Essay should be read as a comment on general tendencies rather than on individual studies or people. To the extent that it can be understood as reflecting on individuals, we are not ourselves exempt from the criticisms.

I.
Goals of the Workshop

The empirical analysis of judicial behavior is one of the fastest growing areas of scholarship in the legal academy. The three of us bring different perspectives to this literature. Two of us, a legal scholar and a political scientist, have been involved in producing portions of that empirical literature. The third, a former federal prosecutor and United States District Judge and currently a law school dean, has been sometimes a critic but also a proponent. Disagreements among us are intense, with each at times finding the others’ perspectives on courts and judges perplexing and frustrating, if not utterly misguided. Yet our debates have resulted in agreement on three important points: the emergence of this literature in legal academia is something to be celebrated, its potential has not yet been realized, and its potential would be realized more quickly if judges and legal theorists played a larger role in producing it.

One reason to celebrate the growth of this literature is the increased interaction between legal scholars and social scientists. Despite much disagreement between social scientists and legal academics on the how and why of studying judges, a number of scholars from each side have begun talking and working together, realizing that they can gain both new insights from each other and bigger audiences for their work. Because of the research experience and methodological expertise that social scientists bring to this partnership, the resulting body of work is likely to be more rigorous and reliable than if it were produced by legal scholars alone.

But this collaboration also brings dangers—in particular, that methodological considerations will dominate theory rather than serve it, resulting in research that is hyper-technical and theoretically narrow or even irrelevant. If this happens, the research will be of little utility or interest to those who should care most about it, including the primary subjects of the literature: judges and those who depend most upon our judicial institutions. Further, neither judges nor scholars with training in other disciplines will be able to engage and be involved in the research project if it takes such a technical turn.

To our eyes, there are already disturbing signs of a trend in this direction. Specifically, in its themes and methodological approaches, the emerging empirical research in the legal academy tends to resemble the work that social scientists were already doing. Part of the cause, we think, is that legal academics and judges have been too reticent about the strengths they bring to empirical research and therefore have not pushed as hard as they might for work to move in new directions. Or perhaps it is that social scientists have been too aggressive in pushing their own perspectives, sometimes in a framework that is seen by judges as attacking them or mocking their aspirations. Either way, we do not think this trend best serves any of the groups involved.

There is a different direction available, one in which the collaboration between legal scholars and social scientists expands to incorporate more perspectives, particularly those of the primary research subjects, and becomes more accessible, interesting, and relevant. Perhaps another way to think of this approach, congenial to law professors, is to think of the judges and the judiciaries as if they were clients and not subjects.

A skeptic might correctly point out that our goals here cut against the dominant paradigm in empirical research across a range of disciplines, in which social scientists study phenomena from an outsider’s perspective. They observe and measure and theorize about their research subjects, but do not necessarily feel any need to interact with them; and certainly not as collaborators. We are overstating, of course. Our colleagues in anthropology and sociology, especially, incorporate the subject-perspective into their research. But their work has not figured prominently in the current enthusiasm in American law schools for empirical research on courts. We hope that in the future judges’ perspectives will play an increasing role in the research on courts.

There is a different reason for our push toward increased collaboration between researchers and their subjects and that has to do with our goals. These goals are at least partially normative. We hope that the research can have payoffs in the near future in terms of yielding insights into how to improve the functioning of the judicial system. The three of us believe that the quality of the judicial system is important as a social and as an economic matter, and that aspects of the system can be measured and studied to help determine whether it can be improved and how. By contrast, there are others who are skeptical of the measurement project itself—arguing that no measurement is better than partial measurement.¹ The threshold questions, then, are what should be measured and how.

II.
What to Measure and Why

The questions for our workshop—what does it mean to judge well, how well do judges perform, and how can judicial institutions be arranged to promote the best possible performance?—are examples of topics that could benefit from greater intellectual cross-fertilization. Like all public officials entrusted with substantial power, judges should be subjected to critical appraisal: holding them accountable for their performance, identifying judges worthy of promotion, helping to decide who is fit to be a judge in the first place, or reforming judicial institutions to promote better judging. Judges themselves, in our experience, are interested in the question of what makes a good judge and, in many cases, would welcome research that attempted to tackle that question, particularly when the outcome of that research might be concrete suggestions for better judicial techniques or institutional arrangements. We are hardly making radical statements here; evaluative statements about judges and judging are far from rare. Indeed, we have colleagues who, although hostile toward any attempt to quantify aspects of judicial behavior, are comfortable evaluating the quality of this or that judge based on a selection of noteworthy opinions.

The challenge we confront, for which we would welcome help from judges and theorists, is in identifying evaluative standards that are widely held, firmly grounded in theory, and amenable to rigorous empirical assessment. If we were to ask observers of courts about judicial performance, we might well reveal some consensus about how well judges do in general and even about which judges stand out as particularly strong or weak. But if we were to press our respondents to explain the grounds for their judgments, we suspect that the answers would differ, with many struggling to give an explanation or even define their terms.

If we are correct that there is room for improvement in the critical evaluation of judging, the main reason is not that judges and scholars have been uninterested in the topic. It is not uncommon for judges to share their thoughts about proper judging in print,² and one of us can attest that many judges who do not write on the topic still contemplate it privately and with colleagues. There also have been a handful of judges who have engaged the academic debates primarily to criticize academic attempts to measure judicial behavior through empirical study.³ Despite the apparent hostility of some judges, our impression from both reading and observation at our workshop is that the tone of their critiques is driven less by hostility to the idea that judicial behavior can be evaluated (and that there are better and worse performing judges and courts) than by the perception that academics are aiming wide of the mark in terms of conducting the type of research that might help improve the judicial system. ⁴

On the academic side, there is some work directly on the question of how to evaluate judging—including Solum’s (2003) theoretical exploration⁵ and Cann’s (2007) empirical analyses.⁶ Empirical studies of judges and courts have become more common, and many of these studies implicitly adopt some view of judging. Concerns about the quality of judging are an important motivator of recent research into heuristics and biases in judging.⁷ And even if they often go unexpressed, normative considerations about the legitimacy of judges’ behavior underlie the question that has garnered more attention from students of judicial behavior than any other: the extent to which judges’ personal policy preferences or moral views trump impartial interpretations of legal materials in determining their decisions. Outside of empirical studies, one may see the same implicit evaluation issues in certain theoretical work, such as in the literature on constitutional interpretation.

Lack of attention, then, is not a major obstacle to progress in the study of judicial performance. In our view, a far more important obstacle is the dearth of intellectual engagement among judges, theorists, and empiricists. The result is empirical work that is often too far removed from the core concerns of theorists and judges to reward their attention and theoretical work that is typically too abstract to lend itself to empirical testing.

Research into ideological voting illustrates this problem. Empirical scholars have amassed mountains of evidence suggesting that ideology plays an important role in judicial decisions, especially at the United States Supreme Court. But this evidence seems to have had only a limited impact on the way most theorists and judges think. Empiricists are often frustrated by what seems like a stubborn refusal to confront the implications of their findings, but there may be more to the reactions than obstinacy. For example, it may be that the distinction between the “legal” and “attitudinal” models does not capture all, or even a large part, of what is important for the legitimacy of judicial decisions.

If we are right to claim that there is a problem, what can be done about it? At a general level, the crucial step is for judges, theorists, and empiricists to engage in structured conversations. Workshops like ours can help foster such conversations, and we hope to hold more of them. Larger conferences, such as those sponsored by the Society for Empirical Legal Studies, may also serve this purpose. In the end, however, there is no substitute for reading each other’s writings. There have been signs of cross-disciplinary awareness in recent years. For instance, Judge Posner, who has always engaged the social science literature, is especially attentive to it in his latest book on judging;⁸ and two recent books by theorists include extensive discussions of social science research.⁹ Yet these authors are in a small minority. On the other side, many empiricists care about theoretical issues; in fact, as noted, the much-maligned attitudinal versus legal model debate is, at bottom, about the legitimacy of judges’ behavior and self-presentation. Still, caring about theoretical issues is not quite the same as reading theorists and judges closely and designing studies specifically to test their ideas or address their concerns.

Of course, writers cannot place all the blame on readers. Empiricists might boost readership among judges and theorists by: a) explaining their methods and results in ways that are clear and unintimidating even to those without much training in empirical research or statistics; b) avoiding resting their analyses on assumptions that strike others as too unrealistic to take seriously; c) making the theoretical and practical implications of their research more explicit; and d) increasing their understanding of the law or legal framework so as to avoid making inaccurate statements or assertions. Perhaps most beneficial would be a greater focus in the first place on questions that judges and theorists could be expected to care about. For instance, in choosing criteria for evaluating judges or judicial institutions, they could pay close attention to the normative arguments of theorists and the practicalities of real life judging, the latter with an eye toward what we can reasonably demand of human judges or what they can reasonably hope to achieve.

Like empiricists, theorists and judges are more likely to attract readers outside their own circles to the extent they refrain from insularity, eschew jargon, and avoid assumptions of knowledge or beliefs not shared by those outside the circle—admittedly, easier said than done. Most importantly, in thinking about their own work, whether academic or on the bench, theorists and judges could try to be more aware of when that work raises questions about the empirical world or rests on assumptions about the empirical world that are questionable. Identifying such questions could make it more obvious to empiricists why they should read what theorists and judges write.

The benefits of having theorists and judges suggest topics for empirical analysis would not end there. Many empiricists would likely find studying the suggested topics intellectually rewarding. Their work would, in turn, be read by more judges and theorists. The result, we suspect, would be a virtuous circle, with ever-increasing engagement among the different groups.

III.
The Next Stage

Our workshop experiences and impressions from reading tell us that suggesting questions for empirical analysis does not come easily to judges and theorists, perhaps because of reticence, skepticism, or certain habits of thinking. And empiricists are not especially inclined to listen to either theorists or research subjects about what they should be studying and how. That said, despite some apparent distrust or misunderstanding at the initial stages, there was ultimately a high level of intellectual engagement at the workshop. Whatever the causes for the initial difficulties in getting the conversations going, we hope theorists and judges will push to play a larger role in setting the empirical research agenda, whether through calls for action or, if they wish to be more directly involved, through active collaboration with empirical researchers.

We end with four sets of more specific suggestions (or pleas) to different combinations of key players. The first, to academics—both theoretical and empirical—is to consider spreading their attention more evenly across a broad range of courts and judicial behaviors. The law touches people’s lives far more often and directly through state trial courts than through federal appellate courts. And then there are the local courts tackling small claims, traffic violations, and family matters; the administrative law tribunals; the international law courts; and similar court systems. All of these settings potentially provide rich sources of insight into the workings of legal institutions. Some of these settings have been examined by researchers, but these examinations are relatively rare and are frequently ignored in mainstream discussions of judges and courts.

As important as decisions on the merits of cases are, it is just as important for us to understand how judges gather information, evaluate evidence, interpret precedents, rule on motions, choose language for their opinions, and so on. Further, whether in the criminal or civil justice systems, most parties’ experience of the courts is not the traditional trial or a series of opinions culminating in the Supreme Court of the jurisdiction. Rather, it is a settlement system, through plea bargain or negotiation. But these truths, although reflected in many individual studies, are not well reflected in the literature as a whole, especially in political science and legal theory. Of course, we are not advocating that scholars stop paying attention to the U.S. Supreme Court and federal appellate decisions. Those who wish to reach judges and produce research with wider application to the world outside of academia, however, might achieve more success by focusing more on the issues of most concern to the typical judge and the typical litigant on a typical day.

The second suggestion, to empiricists alone, is to consider embracing greater methodological flexibility. Both theorists and judges at our workshop seemed impatient with what they viewed as empirical researchers’ insistence on quantification, usually in the context of large-sample studies. Their criticism is overstated, given the large number of empirical researchers who employ qualitative techniques. Nevertheless, it has some validity both for the literature as a whole and for the emerging branch of that literature in legal journals. By no means do we think it would be appropriate for empiricists to weaken their standards in a way that would allow conclusions to be drawn from data that do not adequately support them. But, as long as they explicitly recognize limitations in their data, it seems to us that it may be worthwhile to sacrifice some reliability¹⁰ and precision if it allows them to get at things that really matter.

Our third plea is to theorists and judges. They were no more shy about expressing criticisms of empirical work at our workshop than they have been in print. But their criticisms are seldom as constructive as they might be. It is of some help to an empirical researcher to hear why a particular method of measuring a key concept is flawed; it is far better to receive suggestions for improving the method. Is there any way of assessing the concept, even if imperfect, that would yield useful information? If not, is there a similar concept that could be empirically observed, allowing at least some light to be shed on the question? The key here, we think, is patience—for theorists and judges to recognize that a methodological difficulty is not necessarily an impossibility and, instead of dismissing the problem, to contribute their insights in an attempt to solve it.

Finally, a request of judges. One of the most important things they could do to promote empirical scholarship that is significant and that matters to them is to actively embrace the spirit of scholarly inquiry. No one much enjoys being the focus of critical scrutiny, especially when being evaluated by measures that seem crude. (Consider how academics regard student evaluations of their teaching or their dean’s annual determination of whether they have “contributed” or not). But to the extent judges can overcome discomfort or resentment, cooperate with researchers’ efforts to study them, and suggest ways for researchers to improve their studies, they can significantly contribute to the research enterprise. And there is no reason why judges should only be subjects of research. They can also engage in research informally or formally, whether keeping their eyes open for how things are done in other courts and comparing those methods with their own, engaging in experimentation to test the effectiveness of different practices or institutions, or even conducting full-scale studies and publishing the results. We recognize that in the current political environment there are groups and persons who seek to damage the judiciary in general and individual judges in particular. From our point of view, this is lamentable. But these malevolent forces and special interests will gather and publicize their own flawed data and empirical studies. We ask the judges to consider that more and better empirical study of judging and judicial institutions has the potential to lead to a stronger judiciary and to better judging. It is also an antidote to slanted and partisan attacks disguised as objective studies.

At the end of the day, we realize we are asking for a lot and that others might not be as optimistic regarding the value of collaboration among judges, theorists, and empiricists. What we saw at the workshop itself was a great deal of openness and willingness to engage. Given what we saw, we are certainly willing to do whatever we can to keep the conversations going.

Acknowledgments:

Copyright © 2010 Duke University School of Law.

Mitu Gulati is a Professor of Law at the Duke University School of Law.

David E. Klein is an Associate Professor of Politics at the University of Virginia.

David F. Levi is the Dean and a Professor of Law at the Duke University School of Law. Previously, Dean Levi was the Chief Judge of the United States District Court for the Eastern District of California.