• 17 June 2009

Pitfalls of Empirical Studies That Attempt to Understand the Factors Affecting Appellate Decisionmaking

Hon. Harry T. Edwards - Senior Circuit Judge, D.C. Circuit Court

Posted in , , ,

I recently read a paper in which a prominent legal scholar argued that empirical evidence conclusively demonstrates that decisionmaking in the federal courts of appeals is highly politicized. No body of empirical evidence supports this claim.

When federal appellate judges decide a case, we focus on the relevant legal materials, including the record from the trial court or agency; the challenged judgment, decision, or verdict; the precise issues raised and preserved by the litigants; the parties’ arguments raised in their written briefs and oral arguments; the applicable constitutional, treaty, statutory, regulatory, or contractual provisions; relevant case precedent; and the applicable standards of review. And, because we typically sit in panels of three, we do not act alone in considering the outcome counseled by these materials; rather, we deliberate as a group with the goal of reaching a consensus on the appropriate result.

There is no doubt that, in what I call “hard” and “very hard” cases, judges sometimes exercise discretion in order to reach an outcome that best fits with existing law. Given this reality, some commentators hypothesize that judges’ decisionmaking is significantly determined by their personal political or ideological predilections. Legal empiricists have tried to test this hypothesis through the application of statistical analysis to case outcomes. Rather than considering the reasoning contained in opinions, these scholars treat case outcomes as raw data and attempt to statistically correlate those outcomes to a judge’s presumed personal ideological and political views. These views are often identified by reference to the party of the President who appointed the judge. I find these studies seriously flawed due to conceptual and methodological problems, and I have concluded that they tell us very little about how appellate judges decide cases.1

This past year, I examined the judgments of the D.C. Circuit reviewing administrative agency actions between 2000 and 2008. I selected this eight-year period because a Republican President was in the White House, Congress was controlled by Republicans for a majority of the eight years, and a clear majority of the judges on the D.C. Circuit were appointed by Republican Presidents. I selected administrative agency actions both because the article was prepared in conjunction with a symposium focused on administrative law and also because it is well understood that this large category of cases includes some of the most difficult and controversial appeals heard by my court. If judges’ personal political and ideological predilections played a significant role in their decisions during this period, a prime place to look for this effect would be the court’s decisions on agency actions. One might also expect to find sharp divisions among judges along political lines. That is not what our case dispositions indicate.

Rather, our case dispositions demonstrate: (1) most of the decisions involving administrative agency actions were issued without dissent; (2) judges routinely crossed over presumed political lines in the few cases in which dissents were issued; (3) “mixed panels” of the court (meaning panels consisting of judges appointed by both Republican and Democratic Presidents) routinely issued unanimous decisions resolving complex, difficult, and important administrative law cases; and (4) the full court rarely reheard a case en banc.

Some legal empiricists argue that unanimous decisions prove little, since, in their view, judges may join a decision even though they disagree with the reasoning supporting it. In other words, some empiricists claim that, due to “ideological dampening,” a judge sitting on a “mixed panel” will be less likely to vote according to his or her own ideological preferences. The problem with this claim regarding alleged “panel effects” is that it is based on rank speculation—no empiricist can prove it. Indeed, my colleague, Dean Richard Revesz, who has written extensively on empirical legal studies, has pointed out that “panel effects” can be explained by a “deliberation hypothesis,” pursuant to which judges modify their views because they are persuaded by their colleagues on the appropriate application of the law.

The low rate of dissents in appellate decisions shows that judges appointed by both Democrats and Republicans routinely agree on what the law requires, regardless of their personal political and ideological leanings. Studies indicate that about 90% of all published appellate decisions are unanimous. However, unpublished decisions constitute over 80% of the cases decided by federal appellate courts, and judges rarely issue dissents in unpublished decisions. If both published and unpublished decisions are counted, the rate of dissent in federal appellate courts would border on negligible.

Quite apart from the low level of dissents—which empirical ideologists cannot explain—it is also important to focus on the conceptual and methodological problems inherent in the empirical studies that claim to measure appellate decisionmaking.

Empirical studies often are overly informed by the attitudinal model of judicial behavior. The attitudinal model is premised on the assumption that judicial decisions are determined principally by the personal political and ideological preferences or attitudes of judges, and that judges’ written opinions are merely “smoke screens” designed to hide this reality. The attitudinal model thus fails to take account of the effect of law, precedent, and deliberations on judicial decisionmaking. Consequently, at least in its starkest forms, the attitudinal model speaks about judicial opinions solely in terms of case outcomes. The attitudinal model also assumes that individual judges’ personal views are immutable and can be accurately characterized pursuant to a simplistic liberal/conservative dichotomy. These are patently unrealistic assumptions.

Even those legal empiricists who recognize the problems inherent in the attitudinal model face unsolved methodological difficulties that render suspect all but the most modest empirical conclusions about appellate decisionmaking. Empiricists normally draw on a defined data set of cases and look at “dependent” and “independent” variables. Dependent variables concern the object of study, while independent variables are those that are hypothesized to affect the dependent variable. In studies of judicial decisionmaking, the dependent variables relate to judicial decisions, typically case “outcomes.” And in defining case outcomes, a number of empiricists rely on the “U.S. Courts of Appeals Database.”

Although this database was created to facilitate empirical analysis, it is seriously flawed in that it does not include appellate cases resolved by unpublished decisions. Over 80% of all federal appellate decisions are unpublished. Unpublished decisions typically are unanimous and involve the most straightforward applications of the law. Importantly, then, unpublished decisions offer valuable information regarding appellate judges’ adherence to precedent. For legal empiricists whose stated concern is whether judges follow the law or personal preferences, every judgment must count if the basis of appellate decisionmaking is to be accurately characterized.

The second major methodological obstacle faced by empiricists involves coding difficulties that can distort the dependent case outcome variable. There are many possible dispositions of appellate cases. However, empiricists routinely collapse these dispositions into simple binary outcomes, such as “liberal/conservative.” These characterizations necessarily simplify and distort a court’s holding, reducing to a simple often uninformative label what may be a complex and nuanced decision. For example, the court’s disposition in an administrative law case might include a judgment on standing that appears to be “conservative,” a judgment on “arbitrary and capricious” review that appears to be “liberal,” and a judgment under Chevronthat is neither. All of these nuances are lost in a binary outcome characterization.

Another problem is that only the outcomes of decisions are coded, not the content. A disposition on procedural grounds against an environmental group is treated exactly the same as a decision on the merits, although the consequences can be quite different. Whether an opinion hews closely to precedent, or decides a case on first principles, is usually ignored. Coding only for outcome treats as identical opinions that are, in many ways, quite different.

Difficulties in the coding of independent variables also cause problems for empiricists. Legal empiricists are most interested in studying the personal political and ideological preferences of judges, which are not easily quantified. Scholars consequently seek to describe these preferences through reference to proxies. The proxy typically employed is the party of the appointing President or “PAP.” Relying on this proxy, researchers assume that judges appointed by Republican Presidents are “conservative,” and judges appointed by Democratic Presidents are “liberal.”

Assuming we could agree on the meanings of “conservative” and “liberal,” it is not the case that all Republican Presidents are conservative and all Democratic Presidents are liberal. Moreover, Presidents are not solely motivated to appoint judges who reflect their politics. Commentators have noted alternative motivations for presidential appointments, including personal relationships and party building.

The link between the party of the appointing President and judicial “ideology” is even more attenuated. As Professor Gregory Sisk has explained:

The International Encyclopedia of the Social Sciences defines ideology as “one variant form of those comprehensive patterns of cognitive and moral beliefs about man, society, and the universe in relation to man and society, which flourish in human societies.” Nothing nearly so sophisticated is in operation in most empirical research conducted on the courts, whether undertaken by political scientists or law professors.2

The critically important legal influences on appellate decisionmaking, including case records, the applicable law, precedent, and judicial deliberations, pose even more difficult coding challenges for empiricists and consequently have been largely ignored. To code precedent, for example, formalized and repeatable procedures would have to be developed for identifying and numerically describing the legal issues present in a case, the scope of authoritative and persuasive law, and the effect of that law on the outcome of the case. Legal empiricists have yet to figure out how to reliably code “precedent” as an independent variable. The coding of deliberations presents an even more insurmountable task, for judges’ deliberations are confidential. What is said as judges deliberate over how best to resolve the issues before them is critical to the decisionmaking process, but is not public and therefore cannot be coded.

Persons who read about empirical studies often do not understand the coding problems underlying the use of proxies for political beliefs and ideology. Nor may they appreciate the consequences of empiricists’ failure to account for unpublished decisions, case records, the applicable law, the effect of precedent, and the impact of judicial deliberations. The problem is compounded when some empirical scholars fail to fully reveal the limitations of their studies, and instead suggest that a finding of a “significant correlation” is dispositive. In his thoughtful and measured book, Decision Making in the U.S. Courts of Appeals, Professor Frank Cross usefully explains that

[a] reader [of empirical studies] should not place undue importance on a finding of statistical significance, because such a finding shows a correlation between variables but by itself does not prove the substantive significance of that correlation. One must also consider the magnitude of the association.3

In other words, even when a regression study indicates a strong relationship, the meaning of that relationship may still be unclear. Uninformed readers do not understand this.

There are two areas of disagreement that might arise with respect to my critique of empirical legal studies. First, empiricists might point out that it is often the case in studies involving correlational analyses that researchers do not have direct access to the phenomena they want to measure and must resort to proxies. So long as there is a general correlation between the proxy and the underlying phenomenon (and so long as the proxy is not correlated significantly with the absence of that phenomenon), the imperfections merely burden the estimates with a degree of randomness. Some empiricists might also argue that a significant relationship between PAP and case outcomes is telling because any such relationship is unconnected to anything intrinsic to legal reasoning.

These claims do not hold up under close scrutiny. The hypothesis that judicial decisionmaking is influenced by the ideology of judges is remarkable only if and to the extent that this ideology is extrinsic to law. It is well understood, however, that legal reasoning partakes of political and moral judgments in a number of cases in which judges must exercise delegated or common-lawmaking authority. If one accepts that such reasoning is legal reasoning, then any statistical model that uses a measure of ideology that potentially captures this reasoning cannot tell us much about appellate decisionmaking beyond the bland assertion that judicial disagreement explains variation in outcomes. Ideology may inappropriately affect variation in legal outcomes only if (a) ideology or politics takes on an impermissible, extralegal characteristic—something that empirical scholarship has not shown—or (b) we are wrong in our view that some political and ideological questions are intrinsic to law itself.

Second, empiricists might point out that where there is no correlation between independent variables, the omission of one independent variable should have no measurable impact on the estimated effect of another on the dependent variable. There are at least two problems with this argument. First, we have good reason to believe that the quality of judicial deliberations affects appellate decisionmaking. Therefore, if an empirical model omits the deliberations variable, it will falsely suggest that the influence of personal ideology is immutable and endemic to judicial decisionmaking rather than the source of a correctable pathology that is likely concentrated in relatively discrete segments of the federal circuit courts. Second, it is wrong to assume that as ideological correlations go up, precedent correlations invariably go down. Higher correlations between ideological and political preferences and case outcomes tell us nothing about the relationship between precedent and case outcomes.

The simple truth is that, even accepting these empirical studies on their own terms—that is, with all of their inherent flaws—they predict very little about the effects of extralegal factors on appellate decisionmaking. In his book, Professor Cross concluded that, while appointment variables had measurable effects, they had “very limited explanatory power,” especially when compared to legal variables for which “there was consistently a statistically significant association that was robust to different samples and control variables.”4 Unsurprisingly, none of the studies refutes the claim that case records, the applicable law, precedent, and judicial deliberations are the critically important determinants of appellate decisionmaking.

Finally, I want to be clear that I do not mean to dispute the reality that Presidents often appoint judges whose views are consistent with their own. Indeed, when a court is composed of judges who come from a variety of personal, professional, and political backgrounds, this can make for better informed deliberations. My principal point is that it does not follow from the political reality of partisan appointments that judges act in a partisan way in deciding cases once on the bench. Rather, what I believe is that, on an appellate court that adheres to collegial practices, the applicable law, precedent, and the deliberative process are the primary determinants of case outcomes. Certainly, no empirical study has shown otherwise.dingbat



Copyright © 2009 Duke University School of Law.

Harry T. Edwards is Senior Circuit Judge & Chief Judge Emeritus, U.S. Court of Appeals for the D.C. Circuit; Visiting Professor of Law, New York University School of Law.

Michael A. Livermore, who co-authored the full-length Article, is Executive Director, Institute for Policy Integrity at New York University School of Law.

This Editorial is based on the following full-length Article:  Harry T. Edwards & Michael A. Livermore, Pitfalls of Empirical Studies That Attempt to Understand the Factors Affecting Appellate Decisionmaking, 58 DUKE L.J. 101 (2009).

  1. See Harry T. Edwards & Michael A. Livermore, Pitfalls of Empirical Studies That Attempt to Understand the Factors Affecting Appellate Decisionmaking, 58 DUKE L.J. 101 (2009).
  2. Gregory C. Sisk, The Quantitative Moment and the Qualitative Opportunity: Legal Studies of Judicial Decision Making, 93 CORNELL L. REV. 873, 892 (2008) (book review) (footnote omitted).
  4. Id. at 229.

Post a Comment (all fields are required)

You must be logged in to post a comment.