The Search for Search Standards: The Hunt at DESI IV

What makes an e-discovery search legitimate and defensible?

While virtually every case involves a search for relevant electronically stored information ("ESI"), there is no industry-based definition or measure of a “legally defensible” search. Reminiscent of Supreme Court Justice Potter Stewart's famous quip, some think we know a good search “when we see it,” but the simple and embarrassing truth is that we do not have an operative definition of search acceptability. The lack of any such industry standard for searching and finding ESI in a case wreaks havoc in the field and leaves it to courts to determine, on a case by case basis, whether a particular search passes muster. Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.d. 251 (D. Md. 2008).

But while judicial officers are many things, they are not search experts. United States v. O’Keefe, 537 F. Supp. 2d. 14 (D.D.C. 2008); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331(D.D.C. 2008). In fact, many judges were elevated to judgeship years or even decades before ESI became prevalent, and thus lack any practical experience in searching for, processing, or producing ESI. Putting the question to the courts will therefore result (and has resulted) in disparate answers that vary from jurisdiction to jurisdiction depending both on how the search issues are presented, and on the quality and quantity of resources each side brings to the court. Do we want an approach that may lead to different search standards, say, in a federal court in Chicago versus a state court in Los Angeles?

The risk is simply too great and the issues too pressing, to allow a generation or two of common law decisions across multiple jurisdictions to be cobbled together to shape an overarching definition of a "good" search that counsel and clients can rely upon -- one that will stand up to judicial scrutiny.  That's where "ICAIL," the International Conference of Artificial Intelligence and Law, and its Discovery of Electronically Stored Information (DESI) Workshop, comes in.

Actual search practice in litigation is a chaotic cacophony of divergent practices applied with more or less vigor and attentiveness. What is the best approach for any particular case? Much litigation is still stuck on the case-specific level of discussing and testing the application of Boolean search terms. As a result, the more general and overarching issues of search tools and other technologies involved --which form a large part of whether a search methodology passes muster as a whole -- often take a back seat.

The current lack of legally defensible search standards is especially paradoxical given the tremendous strides that we are making in searching as a general rule. The public release of the Enron (and other voluminous) digital data collections has allowed for search algorithm testing and development on genuine ESI collections, and unleashed exciting new search technologies and methodologies. See.e.g.  We now have our terra firma of a sufficient expanse, to conduct trial and error testing to find sustainable methods of searching. Moreover, an increasingly scientific approach to legal search is dispelling intuitive “the world is flat” biases -- such as the belief that the most reliable search (the so-called “gold standard”) is human, linear review of documents. See and A purely linear approach is an invitation to cost and quality disasters, as the court implied in Multiven, Inc. v. Cisco Systems, 2010 WL 2813618 (N.D. Cal. July 9, 2010), and harkens back to Ken Withers' admonishment to move beyond the dark ages of "proto-digital" e-discovery. See also Kenneth J. Withers, "E-Discovery in the Next Decade: Finding a Way out of Purgatory," Keynote Address at Fifth Annual Advanced E-Discovery Institute (Nov. 20, 2009) and

As our search technologies and research advances, these capabilities will need to be incorporated into the definition of a “legally defensible search” that will give comfort to counsel and client, and that will not be second-guessed in the trenches of our adversarial litigation process. Because, ironically, while we struggle for precision in our searches, today we cannot precisely define a defensible search.

This is not to say there is no guidance at all. Federal magistrate judges have provided some invaluable guidance, and The Sedona Conference® as always has provided important thought leadership on general search technologies and approaches. See generally and The Sedona Conference®, Commentary on Achieving Quality in E-Discovery. EDRM has produced excellent search commentaries. And e-discovery blogs such as Ralph Losey’s “E-Discovery Team,” and John Tredennick's "E-Discovery Search Blog,", have also advanced the dialogue. However, we are far from a definition of “legally defensible search” that judges can use to assess search efforts.

Into this vast and uncharted territory, rides the Discovery of Electronically Stored Information (DESI) Workshop of ICAIL, which held its fourth meeting this month at the University of Pittsburgh. The questions posed to the Workshop by Jason Baron and the DESI organizing steering committee composed of Jason Baron, Laura Ellsworth, Dave Lewis, Debra Logan, and Doug Oard to the 150 participants (vendor representatives, academicians, and lawyers) were:

(a) whether e-discovery search is capable of standardization, and

(b) if so what models might be suitable for the standard setting task.

In short, what would a defensible search standard look like? The day was divided between excellent presentations/discussions and small break-out groups working on discrete topics. We were honored and energized by an extended videoconference visit by Federal magistrate judge Paul Grimm, who is well-known for issuing thoughtful opinions on the issue of e-discovery, who enthusiastically endorsed the committee's work and provided a penetrating critique of the impact of the lack of search standards from the perspective of the bench.

I am not the official reporter of the DESI IV meeting and the opinions here are my own, but I think it safe to say that a consensus emerged on a number of points, two in particular:

  • A cross-vendor search standard will not be a particular tool or methodology, but an institutionalized quality assurance process that is required in many industries but that that has not penetrated the legal industry generally, much less e-discovery in particular. For example, manufacturers and retailers refuse to accept supplier products and services that are not “produced” subject to ISO standards. Legal processes should not be an exception. The ISO 9001 certification standard will require quality controls, precision and recall metrics, a proper mix of automated process and human direction and iteration, sampling parameters, exception reporting, refinement, auditing, and a senior management, institutional process commitment to on-going, enhanced measureable and verifiable quality. Clients, lawyers, and courts may ultimately, and perhaps soon, require ISO certifications before accepting search results.

ICAIL’s DESI IV Workshop has launched a long overdue undertaking. There will be little justice without e-discovery and little meaningful ESI without good, defensible search in our exploding digital data ecosphere. See Baron and Losey,"E-Discovery: Did You Know?  

At stake is the legitimacy and integrity of our judicial processes. In trying to pave roads through a near-barren landscape, DESI deserves our unwavering support.