When I think of “judging” as an anthropologist, I think of two of the classics in legal anthropology:  Paul Bohannan’s Justice and Judgment Among the Tiv ¹ and Max Gluckman’s The Judicial Process Among the Barotse.² Both of these books were ethnographic studies of African tribal courts during the latter days of British colonial rule.  The tribal judges were prominent in both books, and when I teach them in law school I ask students to compare the tribal judges’ roles and conduct to what they know about contemporary American judges.  Like American judges, these African judges sometimes maneuvered the disputants into settlements and sometimes issued final judgments.  Unlike American judges, they never wrote opinions.³ So one could understand them only by watching, listening, and attending to all aspects of their behavior, including such apparently trivial details as how they arranged the seating in the open-air courtroom.

I also think—though far less grandiosely, of course—of the work that Mack O’Barr and I did in American small claims courts in the 1980s and 1990s.⁴ Like their African counterparts, the judges we watched in these courts persuaded, cajoled, threatened, and, when necessary, judged.  Nothing except the basic terms of a final judgment got committed to writing.  Moreover, again as in the African courts, the small claims judges had broad and largely unreviewable authority to impose whatever “law” they saw fit.  In one memorable case that involved a dry cleaner’s liability for a damaged suit, the judge decided to await and abide by the ruling of that august supranational tribunal, the International Fabricare Institute.

Regardless of any methodological preferences or preconceptions we might have had, the absence of written work product forced us—as it had forced Gluckman and Bohannan—to focus on the performance aspects of judging.  Exactly what did the judges say?  Just how did they say it?  What was their non-verbal behavior like?  And a good thing, too: this enforced focus on behavior led us in directions we never would have thought of on our own.  We found a range of judging styles and approaches; in anthropological terms, each courtroom had its own legal culture, created by the judge.  Within these mini-cultures, we found curious echoes of such jurisprudential schools as formalism and realism, as well as applications of the psychological concept of procedural justice.  This is not to say that the judges were consciously enacting these theories, but enacting them they were.

This focus also led to another set of findings, all derived from an initial observation that courtroom interactions are a form of conversation.  (This observation was influenced by an earlier study of British courts, Max Atkinson and Paul Drew’s Order in Court.⁵)  Practitioners of various forms of “ethnomodology” (including conversation analysts, linguistic anthropologists, and the many species of discourse analysts) have long demonstrated that “doing” conversation is an enormously complex social task, one that requires detailed rules.  Moreover, ethnomethodologists argue, these and other things that we think of as social “rules” are not rules in the sense of preexisting principles that need only to be applied.  Rather, they are “phenomena of order,” apparent rules that people “attend to,” and that “emerge” over and over again as people negotiate concrete social situations.  For example, two separate discourse analyses of actual jury deliberations (by Doug Maynard and John Manzo, in 1993,⁶ and Robin Conley and me, this year⁷) have shown, respectively, how principles of justice and standards of proof are produced by jurors in the course of their conversations.  In the small claims courtroom, O’Barr and I saw emergent rules for such judging problems as displaying authority, “doing” due process, showing empathy, and delivering bad news.

But why would these approaches have any relevance for those who study “real” judges?  In a current draft of a paper on the practical impact of “landmark” decisions, Professors George, Gulati, and McGinley⁸ suggest that the crowd divides into two general camps:  the legal scholars, who parse opinions; and the political scientists and their allies, who count and otherwise slice and dice outcomes.  Why might either camp care about methods of studying judging developed while sitting under a tree in Nigeria or in grimy small claims courts around the United States?

Because “real” lawyers and their clients might care.  The most striking observation made by George et al. is that practicing lawyers and judges as well as potentially affected citizens may not define “landmark” in the same way as academics.  In the case that George’s group focused on, what the eggheads expected to be earth-shaking barely registered on the ground.

This leads to a corollary idea more directly related to judging:  practicing lawyers may define “good” and “bad” judges in very different ways than academics.  Law professors tend to praise written opinions that are logically tight, that deal appropriately with precedent, and, most of all, that validate the professors’ policy preferences.  The authors of such opinions are “good” judges.  Similarly, quantitative studies tend to exalt judges who write lots of opinions (some researchers prefer long ones and others short ones), who rarely get reversed, and who get cited often.

But talk to practicing lawyers and you might get a different emphasis.  (I base this generalization on my own six years in full-time practice and 26 as of counsel to a firm and student of the profession.)  First, reversing the priority of their status-conscious academic betters, lawyers may focus more on trial than appellate judges because of per capita impact on their practice.  They are often more interested in how law is administered in their cases than in what law is made in “important” cases.  Second, my guess is that practitioners would identify a much higher proportion of “bad” judges.  If, for instance, one were to rely solely on the testimony of former law clerks now in academia, all federal judges would emerge as hybrids of Oliver Wendell Holmes and St. Thomas More.  But practitioners see some of these same judges as, in varying combinations, lazy, stupid, and mean (to quote my brother, a practicing lawyer).

In identifying “good” judges, practicing lawyers are likely to ask such questions as: How does s/he behave in court?  Does s/he treat lawyers, litigants, and witnesses fairly and courteously?  Can s/he manage a trial competently? Does s/he actually listen or does s/he prejudge things?  Will s/he decide motions in a reasonable period of time, or does s/he ignore them in the hope they’ll go away?  How smart is s/he?

These judicial qualities, I suspect, are very important to those who must live with the judges.  But they do not necessarily emerge from written opinions, nor from aggregations of outcomes.  They would become evident only through an inductive, ethnographic approach to judging that paid particular attention to language.  I wonder what other “phenomena of order” might emerge.  I suspect that important clues to the legitimacy of the legal system might reside in the details of courtroom interactions, and that a significant, “emergent” notion of justice might reveal itself as well.  The point is that, based on anthropology’s experience, including my own, I would not know what hypotheses to test a priori, but would instead expect the unexpected.

An aside: the legitimacy issue was brought home to me when I made my first visit to the European Court of Justice this past summer.  With a group of students, I attended a hearing in a case about import duties (a private company versus the European Commission).    The lawyers argued in German, the judges spoke in French (badly, since none came from France), and I listened alternately in French and English.  The lawyers read prepared remarks and the judges asked no questions; their only utterances were organizational and perfunctory (“We’ll hear now from Herr X”).    Instead, they stared with glazed eyes or flipped through the parties’ written submissions.  I could not imagine what value this presumably expensive exercise had for anyone (except the private-sector lawyer, of course).

After the hearing I discussed it with an ECJ staff lawyer.  I asked, in more polite language, “Why bother with this charade”  I suggested that the court learned nothing and the parties did nothing to advance their cases that couldn’t have been done better in writing.  He didn’t disagree, but said that hearings helped to legitimize the ECJ in the eyes of litigants and, more broadly, to inculcate EU citizenship.  I could not imagine how.  But I thought that if I had been paying closer attention—paying ethnographic attention, as it were—and had a chance to study and restudy my own, linguistic transcript of the hearing, I might have been able to figure out how the court and its constituents were “doing” legitimacy.

All of these considerations lead to the methodological question of how an ethnographer would study “real” judges (federal district judges, let’s say).  One thought might be to collect trial transcripts of cases that had been appealed and do discourse analysis.  That would be interesting.  However, from a sociolinguistic perspective, that would be more a study of transcripts than a study of judicial behavior.  Court reporters perform a subtle kind of interpretation, and their work product is not what a linguist would call a transcript.  Moreover, trial transcripts miss much of what counts as judicial behavior, even in the courtroom.

The only ethnographic solution would be to do what Bohannan and Gluckman did in Africa, and O’Barr and I did in small claims court:  pick some judges, spend days or weeks in detailed observation of their behavior in court (I can’t imagine getting into chambers), tape record it all, and prepare and analyze the detailed, multi-dimensional transcripts that discourse analysts use.  In terms of permission, the watching part would be easy, probably a matter of constitutional right (meddling Institutional Review Boards be damned).  The recording part would require the permission of each judge.  Based on previous experience (in both small claims and, in graduate school, “big” court), some would agree and some wouldn’t.  All would insist on permission from lawyers and litigants.  Perhaps surprisingly, all that I encountered agreed.

Again based on previous experience, it would be a long and demanding project, but by no means undoable.  As models, compare not only my small claims work but Beth Mertz’s 2007 book, The Language of Law School,⁹which analyzed discourse in several first-year law school classes around the country (and, as I think about it, is an excellent illustration of every point I’ve tried to make here).  For the reasons already advanced, I think such a project would be eminently worthwhile, although probably in surprising ways.  I doubt I’ll do it, but I hope someone will.

Reflections on the Conference

Being an anthropologist, I approached the conference itself as an ethnographic opportunity—I don’t know any better.  It was another day among the “natives,” in this case an array of federal and state judges, trial and appellate, and some academics who are interested in studying them.  Two themes stood out.

The first was the sharp distinction between trial and appellate judging.  Trial judges engage in complex conversations.  Many times these conversations are live, oral, and public, as when a judge conducts a trial, a motion argument, or a sentencing hearing.  The judge must not only apply the law in real time, but must also manage the human interaction.  Most trial judges realize that their job in an adversary system is to stay as far in the background as possible while letting the parties produce the trial.  The lawyers are frequently repeat players with whom the judge will have to deal again and again.  This task—“doing trial judging”—strikes me as remarkably challenging, both intellectually and sociologically.  And it must be done while looking over one’s shoulder, since the whole performance is subject to public review, (sometimes unfair) recharacterization, and (sometimes withering) critique.

Appellate judges, by contrast, spend most of their time manipulating written texts.  As they described it at the conference, their work consists largely of reading briefs, memos, cases, and statutes; drafting, circulating, and reading opinions; and attempting to persuade colleagues in written (and, increasingly, electronic) communications.  Live conversations in the form of oral arguments and conferences with colleagues are the exception.  With the exception of the oral arguments, the work of appellate judges is done in privacy and relative anonymity.  When they do perform publicly, they—in contrast to their trial colleagues—are expected to be the stars, interrupting and hectoring the lawyers with questions that sometimes seem intended to show off as much as to focus the arguments.

These functional differences brought home to me the point that studying judging encompasses two quite different projects.  To study the performance of trial judges is to study public behavior and public documents.    It is thus inherently doable, even if extraordinarily labor-intensive.  Appellate judges, however, do most of their work in a secret world that is seemingly impenetrable to ethnographers or others who rely on direct observation. One can study only the judgments and opinions they are required to release to the public; the performance that underlies these carefully crafted documents is immune to scrutiny.

The second observation derives from the first: judges have difficulty being anything but judges, and the trial-appellate distinction carried over into the conference discussion.  The trial judges, by and large, conversed.  They asked and answered questions and seemed eager to have external, scientific perspectives on what and how they are doing.  The appellate judges acted like, well, appellate judges (with apologies to the exceptions).  They monopolized the floor, interrupted, and sometimes went on, and on, and on.  There was nothing unpleasant about this; on the contrary, the whole event was unfailingly cordial. It was just that the appellate judges stayed in courtroom character.  When they discussed their behind-the-scenes work, they saw few problems that were worth studying. They have things under control (aside from the rare colleague who doesn’t work well with others and writes gratuitous dissents).  We academics will never get in, of course, but what would be the point?  (And academic idolatry of federal appellate judges only eggs them on.)

So I came away from the conference with mixed feelings.  Trial judges want to be studied, and there are many ways to do it, with ethnography well-positioned to play a role.  But at the appellate level, it seems that we will be limited to the analysis of outcomes.  As it usually does in legal scholarship, “empirical” will continue to mean “quantitative.”

Acknowledgments:

Copyright © 2010 Duke Law Journal.

John Conley is a professor at the University of North Carolina-Chapel Hill Law School

When federal appellate judges decide a case, we focus on the relevant legal materials, including the record from the trial court or agency; the challenged judgment, decision, or verdict; the precise issues raised and preserved by the litigants; the parties’ arguments raised in their written briefs and oral arguments; the applicable constitutional, treaty, statutory, regulatory, or contractual provisions; relevant case precedent; and the applicable standards of review. And, because we typically sit in panels of three, we do not act alone in considering the outcome counseled by these materials; rather, we deliberate as a group with the goal of reaching a consensus on the appropriate result.

There is no doubt that, in what I call “hard” and “very hard” cases, judges sometimes exercise discretion in order to reach an outcome that best fits with existing law. Given this reality, some commentators hypothesize that judges’ decisionmaking is significantly determined by their personal political or ideological predilections. Legal empiricists have tried to test this hypothesis through the application of statistical analysis to case outcomes. Rather than considering the reasoning contained in opinions, these scholars treat case outcomes as raw data and attempt to statistically correlate those outcomes to a judge’s presumed personal ideological and political views. These views are often identified by reference to the party of the President who appointed the judge. I find these studies seriously flawed due to conceptual and methodological problems, and I have concluded that they tell us very little about how appellate judges decide cases.¹

This past year, I examined the judgments of the D.C. Circuit reviewing administrative agency actions between 2000 and 2008. I selected this eight-year period because a Republican President was in the White House, Congress was controlled by Republicans for a majority of the eight years, and a clear majority of the judges on the D.C. Circuit were appointed by Republican Presidents. I selected administrative agency actions both because the article was prepared in conjunction with a symposium focused on administrative law and also because it is well understood that this large category of cases includes some of the most difficult and controversial appeals heard by my court. If judges’ personal political and ideological predilections played a significant role in their decisions during this period, a prime place to look for this effect would be the court’s decisions on agency actions. One might also expect to find sharp divisions among judges along political lines. That is not what our case dispositions indicate.

Rather, our case dispositions demonstrate: (1) most of the decisions involving administrative agency actions were issued without dissent; (2) judges routinely crossed over presumed political lines in the few cases in which dissents were issued; (3) “mixed panels” of the court (meaning panels consisting of judges appointed by both Republican and Democratic Presidents) routinely issued unanimous decisions resolving complex, difficult, and important administrative law cases; and (4) the full court rarely reheard a case en banc.

Some legal empiricists argue that unanimous decisions prove little, since, in their view, judges may join a decision even though they disagree with the reasoning supporting it. In other words, some empiricists claim that, due to “ideological dampening,” a judge sitting on a “mixed panel” will be less likely to vote according to his or her own ideological preferences. The problem with this claim regarding alleged “panel effects” is that it is based on rank speculation—no empiricist can prove it. Indeed, my colleague, Dean Richard Revesz, who has written extensively on empirical legal studies, has pointed out that “panel effects” can be explained by a “deliberation hypothesis,” pursuant to which judges modify their views because they are persuaded by their colleagues on the appropriate application of the law.

The low rate of dissents in appellate decisions shows that judges appointed by both Democrats and Republicans routinely agree on what the law requires, regardless of their personal political and ideological leanings. Studies indicate that about 90% of all published appellate decisions are unanimous. However, unpublished decisions constitute over 80% of the cases decided by federal appellate courts, and judges rarely issue dissents in unpublished decisions. If both published and unpublished decisions are counted, the rate of dissent in federal appellate courts would border on negligible.

Quite apart from the low level of dissents—which empirical ideologists cannot explain—it is also important to focus on the conceptual and methodological problems inherent in the empirical studies that claim to measure appellate decisionmaking.

Empirical studies often are overly informed by the attitudinal model of judicial behavior. The attitudinal model is premised on the assumption that judicial decisions are determined principally by the personal political and ideological preferences or attitudes of judges, and that judges’ written opinions are merely “smoke screens” designed to hide this reality. The attitudinal model thus fails to take account of the effect of law, precedent, and deliberations on judicial decisionmaking. Consequently, at least in its starkest forms, the attitudinal model speaks about judicial opinions solely in terms of case outcomes. The attitudinal model also assumes that individual judges’ personal views are immutable and can be accurately characterized pursuant to a simplistic liberal/conservative dichotomy. These are patently unrealistic assumptions.

Even those legal empiricists who recognize the problems inherent in the attitudinal model face unsolved methodological difficulties that render suspect all but the most modest empirical conclusions about appellate decisionmaking. Empiricists normally draw on a defined data set of cases and look at “dependent” and “independent” variables. Dependent variables concern the object of study, while independent variables are those that are hypothesized to affect the dependent variable. In studies of judicial decisionmaking, the dependent variables relate to judicial decisions, typically case “outcomes.” And in defining case outcomes, a number of empiricists rely on the “U.S. Courts of Appeals Database.”

Although this database was created to facilitate empirical analysis, it is seriously flawed in that it does not include appellate cases resolved by unpublished decisions. Over 80% of all federal appellate decisions are unpublished. Unpublished decisions typically are unanimous and involve the most straightforward applications of the law. Importantly, then, unpublished decisions offer valuable information regarding appellate judges’ adherence to precedent. For legal empiricists whose stated concern is whether judges follow the law or personal preferences, every judgment must count if the basis of appellate decisionmaking is to be accurately characterized.

The second major methodological obstacle faced by empiricists involves coding difficulties that can distort the dependent case outcome variable. There are many possible dispositions of appellate cases. However, empiricists routinely collapse these dispositions into simple binary outcomes, such as “liberal/conservative.” These characterizations necessarily simplify and distort a court’s holding, reducing to a simple often uninformative label what may be a complex and nuanced decision. For example, the court’s disposition in an administrative law case might include a judgment on standing that appears to be “conservative,” a judgment on “arbitrary and capricious” review that appears to be “liberal,” and a judgment under Chevronthat is neither. All of these nuances are lost in a binary outcome characterization.

Another problem is that only the outcomes of decisions are coded, not the content. A disposition on procedural grounds against an environmental group is treated exactly the same as a decision on the merits, although the consequences can be quite different. Whether an opinion hews closely to precedent, or decides a case on first principles, is usually ignored. Coding only for outcome treats as identical opinions that are, in many ways, quite different.

Difficulties in the coding of independent variables also cause problems for empiricists. Legal empiricists are most interested in studying the personal political and ideological preferences of judges, which are not easily quantified. Scholars consequently seek to describe these preferences through reference to proxies. The proxy typically employed is the party of the appointing President or “PAP.” Relying on this proxy, researchers assume that judges appointed by Republican Presidents are “conservative,” and judges appointed by Democratic Presidents are “liberal.”

Assuming we could agree on the meanings of “conservative” and “liberal,” it is not the case that all Republican Presidents are conservative and all Democratic Presidents are liberal. Moreover, Presidents are not solely motivated to appoint judges who reflect their politics. Commentators have noted alternative motivations for presidential appointments, including personal relationships and party building.

The link between the party of the appointing President and judicial “ideology” is even more attenuated. As Professor Gregory Sisk has explained:

The International Encyclopedia of the Social Sciences defines ideology as “one variant form of those comprehensive patterns of cognitive and moral beliefs about man, society, and the universe in relation to man and society, which flourish in human societies.” Nothing nearly so sophisticated is in operation in most empirical research conducted on the courts, whether undertaken by political scientists or law professors.²

The critically important legal influences on appellate decisionmaking, including case records, the applicable law, precedent, and judicial deliberations, pose even more difficult coding challenges for empiricists and consequently have been largely ignored. To code precedent, for example, formalized and repeatable procedures would have to be developed for identifying and numerically describing the legal issues present in a case, the scope of authoritative and persuasive law, and the effect of that law on the outcome of the case. Legal empiricists have yet to figure out how to reliably code “precedent” as an independent variable. The coding of deliberations presents an even more insurmountable task, for judges’ deliberations are confidential. What is said as judges deliberate over how best to resolve the issues before them is critical to the decisionmaking process, but is not public and therefore cannot be coded.

Persons who read about empirical studies often do not understand the coding problems underlying the use of proxies for political beliefs and ideology. Nor may they appreciate the consequences of empiricists’ failure to account for unpublished decisions, case records, the applicable law, the effect of precedent, and the impact of judicial deliberations. The problem is compounded when some empirical scholars fail to fully reveal the limitations of their studies, and instead suggest that a finding of a “significant correlation” is dispositive. In his thoughtful and measured book, Decision Making in the U.S. Courts of Appeals, Professor Frank Cross usefully explains that

[a] reader [of empirical studies] should not place undue importance on a finding of statistical significance, because such a finding shows a correlation between variables but by itself does not prove the substantive significance of that correlation. One must also consider the magnitude of the association.³

In other words, even when a regression study indicates a strong relationship, the meaning of that relationship may still be unclear. Uninformed readers do not understand this.

There are two areas of disagreement that might arise with respect to my critique of empirical legal studies. First, empiricists might point out that it is often the case in studies involving correlational analyses that researchers do not have direct access to the phenomena they want to measure and must resort to proxies. So long as there is a general correlation between the proxy and the underlying phenomenon (and so long as the proxy is not correlated significantly with the absence of that phenomenon), the imperfections merely burden the estimates with a degree of randomness. Some empiricists might also argue that a significant relationship between PAP and case outcomes is telling because any such relationship is unconnected to anything intrinsic to legal reasoning.

These claims do not hold up under close scrutiny. The hypothesis that judicial decisionmaking is influenced by the ideology of judges is remarkable only if and to the extent that this ideology is extrinsic to law. It is well understood, however, that legal reasoning partakes of political and moral judgments in a number of cases in which judges must exercise delegated or common-lawmaking authority. If one accepts that such reasoning is legal reasoning, then any statistical model that uses a measure of ideology that potentially captures this reasoning cannot tell us much about appellate decisionmaking beyond the bland assertion that judicial disagreement explains variation in outcomes. Ideology may inappropriately affect variation in legal outcomes only if (a) ideology or politics takes on an impermissible, extralegal characteristic—something that empirical scholarship has not shown—or (b) we are wrong in our view that some political and ideological questions are intrinsic to law itself.

Second, empiricists might point out that where there is no correlation between independent variables, the omission of one independent variable should have no measurable impact on the estimated effect of another on the dependent variable. There are at least two problems with this argument. First, we have good reason to believe that the quality of judicial deliberations affects appellate decisionmaking. Therefore, if an empirical model omits the deliberations variable, it will falsely suggest that the influence of personal ideology is immutable and endemic to judicial decisionmaking rather than the source of a correctable pathology that is likely concentrated in relatively discrete segments of the federal circuit courts. Second, it is wrong to assume that as ideological correlations go up, precedent correlations invariably go down. Higher correlations between ideological and political preferences and case outcomes tell us nothing about the relationship between precedent and case outcomes.

The simple truth is that, even accepting these empirical studies on their own terms—that is, with all of their inherent flaws—they predict very little about the effects of extralegal factors on appellate decisionmaking. In his book, Professor Cross concluded that, while appointment variables had measurable effects, they had “very limited explanatory power,” especially when compared to legal variables for which “there was consistently a statistically significant association that was robust to different samples and control variables.”⁴ Unsurprisingly, none of the studies refutes the claim that case records, the applicable law, precedent, and judicial deliberations are the critically important determinants of appellate decisionmaking.

Finally, I want to be clear that I do not mean to dispute the reality that Presidents often appoint judges whose views are consistent with their own. Indeed, when a court is composed of judges who come from a variety of personal, professional, and political backgrounds, this can make for better informed deliberations. My principal point is that it does not follow from the political reality of partisan appointments that judges act in a partisan way in deciding cases once on the bench. Rather, what I believe is that, on an appellate court that adheres to collegial practices, the applicable law, precedent, and the deliberative process are the primary determinants of case outcomes. Certainly, no empirical study has shown otherwise.

Acknowledgments:

Copyright © 2009 Duke University School of Law.

Harry T. Edwards is Senior Circuit Judge & Chief Judge Emeritus, U.S. Court of Appeals for the D.C. Circuit; Visiting Professor of Law, New York University School of Law.

Michael A. Livermore, who co-authored the full-length Article, is Executive Director, Institute for Policy Integrity at New York University School of Law.

This Editorial is based on the following full-length Article:  Harry T. Edwards & Michael A. Livermore, Pitfalls of Empirical Studies That Attempt to Understand the Factors Affecting Appellate Decisionmaking, 58 DUKE L.J. 101 (2009).