Devising Rule of Law Baselines: The Next Step in Quantitative Studies of Judging

Brian Z. Tamanaha - Washington University School of Law

Posted in , , , , ,

Political scientists and law professors have lately taken to asserting that quantitative studies of judging reveal worrisome findings about the rule of law in the U.S. judicial system. The authors of Are Judges Political? declare: “variations in panel composition lead to dramatically different outcomes, in a way that creates serious problems for the rule of law.”1 The authors of Judging on a Collegial Court similarly conclude:

Because separate opinions and reversals constitute behavioral manifestations of judges’ discretionary authority, studies of dissensus shed light on critical questions related to the effective functioning and legitimacy of our legal system and the operation of the rule of law. . . . Our findings cut both ways. The evidence we have presented in the preceding pages of this book demonstrates that judging is both a legal and a political activity . . . .2

Surveying the results of recent quantitative studies of judging, Cass Sunstein and Thomas Miles observe that “[f]or those who believe in the rule of law, and in the discipline imposed by the legal system, the results of the New Legal Realism need not be entirely discouraging. The glass is half empty, perhaps, but it is also half full.”3

The rule of law is often said to be a defining aspect of the American system of governance—the foundation stone of our free society—in which judges play a pivotal role.4> It is alarming to be informed that serious problems in judging threaten the rule of law or that the rule of law glass is half empty. If these concerns are valid, remedial measures must be sought and implemented without delay.

But are they correct? Have studies of judging shown that the rule of law is in trouble? To evaluate these assertions, one must first know what the rule of law requires of judges; then one must identify or measure how much and in what ways judges are falling short of these requirements. To say that the rule of law glass is half full, continuing with Sunstein’s and Miles’s metaphor, requires knowing what a full glass of the rule of law looks like: there must be rule of law baselines or standards.

A quick look at these studies exposes the need for such baselines. The authors of Are Judges Political? find that “[f]requently the law is clear, and judges should and will simply implement it, no matter who has appointed them.”5 Their study provides “considerable evidence to suggest that they do exactly that”;6 in five major areas studied they find no ideological effect on judicial decisions, and even when an effect did show, the differences, they admit, were “not huge.”7 A study was not necessary to show that judges do not vary greatly by ideology in their legal decisions because typically about 90 percent of federal appellate decisions (more when unpublished cases are counted) are issued without a dissent. Judges, then, agree an overwhelming proportion of the time regardless of ideological differences. By that measure, at least, the rule of law appears to be working well.

The authors of Judging on a Collegial Court find that ideological differences show a statistically significant increase in the probability of a dissent or a concurrence. But it turns out that the size of the effect—its actual impact on the run of decisions—was minuscule: “The difference in absolute terms is rather small, with slightly less than a 0.01 increase in the probability of a concurrence and a 0.02 increase in the probability of a dissent.”8 That is hardly worrisome. Confounding the authors’ expectations, furthermore, their study finds no statistically significant correlation between ideological difference and rates of reversal—that is, appellate panels did not reverse trial judges with an opposing ideological disposition at a higher rate. This study, covering decisions by nearly a thousand judges over four decades, would appear to confirm that political views have little impact on judicial decisions, yet, without explaining why, the authors suggest that their findings “cut both ways” on the rule of law.

Behind the disquieting assertions about the rule of law lies an unstated assumption: the proposition that any finding of political influence on judging, no matter how small, is contrary to the rule of law. This, however, is a profoundly unrealistic assumption—ironically so, because these political scientists and law professors claim the mantle of legal realism.

Balanced Realism About Judging

For more than a century, judges and jurists in the United States have expressed a view of judging that I call “balanced realism.”9 Balanced realism recognizes that there are gaps and uncertainties in the law, that sometimes judges have discretion and must make choices, that different judges can sometimes interpret the same law in different ways owing to differences in perspective and background, that inconsistent precedents or conflicts in the applicable law can exist, and that sometimes judges manipulate the law to reach desired ends. (I call these factors the “skeptical aspects.”) But balanced realism also recognizes that a substantial majority of the time, the rules and their application are clear and predictable, that judges are indoctrinated into a shared legal tradition and legal practices that lead then to interpret and apply legal rules in similar ways, that judging takes place in a thick institutional setting that constrains judges, that most judges strive to abide by the commitment to follow the law, and that the overwhelming majority of judicial decisions are legally determined (the “rule-bound” aspects). Balanced realism acknowledges the limitations inherent in the law and in human judges—limitations that cannot be eliminated—yet it also recognizes that law nonetheless works, that judges can and do render rule-bound decisions.

Judge Benjamin Cardozo famously articulated a balanced realism about judging in The Nature of the Judicial Process:

No doubt there is a field within which judicial judgment moves untrammeled by fixed principles. Obscurity of statute or of precedent or of customs or of morals, or collision between some or all of them, may leave the law unsettled, and cast a duty upon the courts to declare it retrospectively in the exercise of a power frankly legislative in function.10

Cardozo insisted that when making these decisions a judge must decide in terms of the community view, not the judge’s personal view, but he was aware that it is difficult to keep the two apart: “The perception of the objective right takes the color of the subjective mind.”11 But despite the inherent openness of law and the limitations of human judges, Cardozo reminded his audience, “[w]e must not let these occasional and relatively rare instances blind our eyes to the innumerable instances where there is neither obscurity nor collision nor opportunity for diverse judgment.”12

Multiple judges before and after Cardozo have described judging in similar terms. In 1886, for example, Judge Thomas Cooley emphasized that uncertainty in the application of law cannot be eliminated

because in the infinite variety of human transactions it becomes uncertain which of the opposing rules the respective parties contend for should be applied in a case having no exact parallel, and because it cannot possibly be known in advance what view a court or jury will take of questions upon which there is room for difference of opinion.13

Differences in the judicial application of law, he wrote, “must always exist so long as there is variety in human minds, human standards, and human transactions.”14 In 1924, Judge Irving Lehman acknowledged that “no thoughtful judge can fail to note that in conferences of the court, differences of opinion are based at least to some extent upon differences of viewpoint.”15 Judge Bernard Shientag remarked in 1944,

[n]aturally, it is in cases where the creative faculty of the judicial process operates, where there is a choice of competing analogies, that the personality of the judge, the individual tone of his mind, the color of his experience, the character and variety of his interests and his prepossessions, all play an important role.16

Judge Albert Tate observed in 1959, “like all other human beings[, judges] have limitations, of vision, knowledge, intelligence, or predisposition which sometimes influence their judicial actions.”17 In 1963, Judge Charles Clark admitted that cases arise in which there is no clear legal answer, and the judge “is on his own for the ultimate result which must reflect his background, his personality, and his inner convictions.”18 And so on.

This encapsulates what many judges have said about judging: the bulk of the law is clear, but the law has a margin of uncertainty; judges try their best to rule in an objective fashion, but their personal views sometimes seep through to influence their decisions. The crucial point is that law cannot be made perfectly certain and judges cannot be made to reason like machines, entirely free of background influences. These inherent aspects of judging shape and constrain what is possible. “The rule of law is not the doctrine of perfect decision,” Judge Alvin Rubin counseled: “[I]n many cases a conscientious decision is as much as can be expected, and . . . there is no ultimate ‘right’ answer.”19

Now it is possible to identify the fundamental flaw in the assumption that any showing of political influence on judicial decisions is inconsistent with the rule of law. A realistic understanding of the rule of law would assume that a certain irreducible amount of ideological influence will be present even in the best system of judging.20 As judges have repeatedly stated, it cannot be otherwise. A realistic view would therefore expect that quantitative studies will find statistically significant correlations in certain contexts between ideology and judicial decisions. This finding in itself, without more, says nothing at all about the rule of law, because it is an inherent aspect of judging. Or to put the point another way, a full glass of the rule of law, like a full glass of milk, is not filled to the brim. The open space between the lip of the glass and the surface of a full glass of the rule of law is where legal uncertainty interacts with the limitations of human judges—where political influences typically come into play.

For the rule of law, what matters is the size of the ideological effect and in what contexts it manifests itself: whether it is greater than, or extends beyond or outside of, what one would expect in a well functioning system of rule-bound judging. Rule of law baselines are necessary to identify what to expect of a full glass of the rule of law. Quantitative studies can raise serious concerns about the rule of law only if their results establish that judicial decisions fall measurably below these baselines. Only then would grounds exist to assert that the rule of law glass is half full, or nearly empty.

Constructing Rule of Law Baselines

Throughout this Essay, I have referred to baselines in the plural because a number of standards will be necessary to account for variations in the nature of legal provisions and the circumstances of judging.  Two factors have particular bearing on the formulation of standards: the type of legal issue a judge is called upon to decide, and the level of the court.

In connection with the first factor, certain legal provisions—especially legal standards like “fairness,” “reasonableness,” or “the best interests of the child”—explicitly call upon judges to exercise discretion or to make judgments of a type that allows or invites (or makes it harder to screen out) the expression of personal views. Consequently, a rule of law baseline for this type of question, which remains legally governed and hence should manifest a significant degree of agreement, would anticipate greater variation among judges and higher correlations between their decisions and their ideological views in comparison to a rule of law baseline for narrow legal rules.

The second factor recognizes that the quantum of legal uncertainty is greater at higher court levels. The vast majority of cases are settled (fewer than two percent of federal cases make it through trial) because the applicable law and provable facts are clear, so both parties can weigh the expected costs and benefits of continuing. About ten to fifteen percent of federal appellate cases, by the estimate of a number of federal judges,21 involve hard or uncertain legal issues. In contrast, the skeptical aspects of law recognized by balanced realism—uncertainty, disagreement, choice, political pressure—show up in a significantly higher proportion of Supreme Court cases, whereas the rule-bound aspects are proportionally less present (including fewer institutional checks). This represents a virtual inversion of the usual balance of these factors within law and judging generally, although Supreme Court decisionmaking is still thickly draped in legal constraints. In recognition of these differences, rule of law baselines for trial courts and appellate courts should anticipate far greater agreement in legal decisions and significantly lower ideologically correlated variations in comparison with high courts (both state and federal).

Needless to say, the task of formulating rule of law baselines will be complicated, requiring ingenuity and much trial and error. This task can be done in a variety of ways, all contestable. Every baseline produced should be viewed with caution, as a proxy that stands for an approximation of an abstraction—a gross quantitative marker for what to expect from rule-bound judges.

Seeking out comparative measures is one way to proceed. For example, assume that over a seventy-five-year span conservative judges vote in the conservative direction in about fifty-five percent of their cases, whereas liberal judges vote conservative in about fifty percent of their cases.22 The relatively small five percent difference in voting behavior, one might surmise, reflects the irrepressible interaction of legal uncertainty with human judging. This historical norm could supply the basis for a rule of law baseline for federal appellate judging. A warning sign that the judicial system is in trouble, then, might be if judges as a group skew their votes in a one-sided ideological direction in a significantly higher proportion of cases, creating a greater than usual disparity between Republican- and Democratic-appointed judges.23 Moreover, individual judges whose decisions fall far outside of this historical range might invite scrutiny for failing to rule in a sufficiently rule-bound fashion. One might create similar baselines for the Supreme Court, derived from historical norms or from a comparison of the voting patterns of Justices against one another. This would allow a determination of whether a particular Court or a particular Justice shows a propensity to rule in an ideological direction that exceeds the usual, historical range.

Setting aside levels of courts, one might also compare differences across legal issues to see whether some issues show greater ideologically linked divergence in judicial decisions than others. Applying different rule of law baselines for standards (expecting greater divergence) and for rules (expecting less) will make it possible to tease out whether the observed increase in divergence is a function of the type of legal provision at issue, or of something else (perhaps political salience or entrenched cognitive biases).

These are just illustrative suggestions. Many factors must be considered before rule of law baselines can be constructed—work that has not yet begun.

Legitimate Objections to Rule of Law Baselines

Critics may object that it is misguided or wrong to construct such baselines, that the goal itself is ludicrous because the rule of law is a deeply contested ideal with uncertain meaning and implications for judging. Furthermore, critics might argue, quantitative standards that purport to provide a basis to evaluate judging will compress the complexity and nuance of judging in a distorting oversimplification that is susceptible to pernicious uses. These are compelling objections. I would not proffer this proposal but for the worry that leaving this gap unaddressed might be worse than the distortions that result from the effort to fill it.

As the first paragraph of this Essay reveals, political scientists and law professors, perhaps succumbing to the temptation to sell their results, have issued broad, alarming claims about the implications of their quantitative studies for the rule of law. These claims, I have argued, do not follow from the results of their studies in the absence of rule of law baselines, and they paint a false image of the state of judging. Rule of law baselines will impose greater discipline on scholars who wish to draw out broader implications from their results and will provide a sounder footing for their observations.

This effort will also lead to an important advance in the discipline of quantitative studies. Quantitative scholars demonstrate time and again through their studies that judging is not a purely legal activity. This point is not informative. Dozens of judges have admitted for decades that sometimes law is uncertain or runs out, that judges must sometimes make choices, and that sometimes their personal views have an impact on their legal decisions.24 That is the nature of law and of human judging. The formulation of rule of law baselines would constitute a major advance within the field, because these baselines would formally incorporate the recognition of this reality, affecting the orientation and design of the next generation of studies. Future quantitative studies would produce information worthy of attention, not when merely finding indications of ideological influence, but when finding a notable deviation from expected baselines. Absent rule of law baselines, these rapidly multiplying studies will merely confirm what everyone in law already knows.


Copyright © 2010 Duke Law Journal.

Brian Z. Tamanaha is a Professor at the Washington University School of Law.

This material is based upon work supported by the National Science Foundation under Grant No. SES-0946437.

  3. Thomas J. Miles & Cass R. Sunstein, The New Legal Realism, 75 U. CHI. L. REV. 831, 844 (2008).
  4. See BRIAN Z. TAMANAHA, ON THE RULE OF LAW: HISTORY, POLITICS, THEORY 2, 104 (2004) (explaining that “the defining characteristic of the Western political tradition is ‘freedom under the rule of law’” and discussing the role of judges in “find{ing} a balance” between individual freedom and the rule of law).
  5. SUNSTEIN ET AL., supra note 1, at 5.
  6. Id.
  7. Id. at 129.
  8. HETTINGER ET AL., supra note 2, at 65.
  11. Id. at 110–11.
  12. Id. at 129.
  13. Thomas M. Cooley, Another View of Codification, 2 COLUMBIA JURIST 464, 465 (1886).
  14. Id. at 465–66.
  15. Irving Lehman, The Influence of the Universities on Judicial Decisions, 10 CORNELL L.Q. 1, 6 (1924).
  17. Albert Tate, Jr., Forum Juridicum: The Judge as a Person, 19 LA. L. REV. 438, 439 (1959).
  18. Charles E. Clark, The Limits of Judicial Objectivity, 12 AM. U. L. REV. 1, 12 (1963).
  19. Alvin B. Rubin, Views from the Lower Court, 23 UCLA L. REV. 448, 453–54 (1976).
  20. Nor is it clear that the legal system would be better if these aspects could be eliminated. These factors, the openness of law and the influence of background views of judges, help law change in sync with changes in society.
  21. See TAMANAHA, supra note 9, at 125–31, 144.
  22. This example is a simplified and modified version of a study reported by Judge Richard Posner. See RICHARD A. POSNER, HOW JUDGES THINK 21 (2008).
  23. With respect to voting trends, Judge Posner’s study indeed shows an increase in the ideological disparity among currently sitting judges. Id.
  24. See TAMANAHA, supra note 9, chs. 7–8.

Post a Comment (all fields are required)

You must be logged in to post a comment.