Measuring selection bias in publicly available judicial opinions

Abstract

To have an informed discussion about judicial performance and efficiency, we will sometimes want to explore what judges actually do on an everyday level. But in many ways, courts have not always been paragons of transparency. Often the parties are the only people who are aware of what action a court has taken in a case.

This paper explores that dynamic, in the context of decisions made by federal trial courts at one particular procedural stage--decisions made on motions to dismiss for failure to state a claim--Rule 12(b)(6) motions. There is growing interest in the work of federal trial courts, and to date, most legal researchers have turned to the same resource: commercially available databases. But they have done so conscious of the risk that many district court decisions will never find their way onto Westlaw or LEXIS. If the universe of opinions available on commercially available services is not representative of the entire universe of district court decisions, it is harder to draw conclusions about the work of federal trial courts and therefore more difficult to draw conclusions about judicial performance.

This paper shows that the fear is justifiable: certain kinds of decisions, issued by judges sitting in certain judicial districts hearing particular kinds of cases, are overrepresented in services like Westlaw and LEXIS. This has the potential to affect the kinds of conclusions one might draw from observations gleaned by reviewing decisions available only on commercial databases. More broadly, it has the potential to affect how lawyers and judges argue and adjudicate cases, and therefore to affect the corpus of law itself