The Search for Search Standards: The Hunt at DESI IV

What makes an e-discovery search legitimate and defensible?

While virtually every case involves a search for relevant electronically stored information ("ESI"), there is no industry-based definition or measure of a “legally defensible” search. Reminiscent of Supreme Court Justice Potter Stewart's famous quip, some think we know a good search “when we see it,” but the simple and embarrassing truth is that we do not have an operative definition of search acceptability. The lack of any such industry standard for searching and finding ESI in a case wreaks havoc in the field and leaves it to courts to determine, on a case by case basis, whether a particular search passes muster. Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.d. 251 (D. Md. 2008).

But while judicial officers are many things, they are not search experts. United States v. O’Keefe, 537 F. Supp. 2d. 14 (D.D.C. 2008); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331(D.D.C. 2008). In fact, many judges were elevated to judgeship years or even decades before ESI became prevalent, and thus lack any practical experience in searching for, processing, or producing ESI. Putting the question to the courts will therefore result (and has resulted) in disparate answers that vary from jurisdiction to jurisdiction depending both on how the search issues are presented, and on the quality and quantity of resources each side brings to the court. Do we want an approach that may lead to different search standards, say, in a federal court in Chicago versus a state court in Los Angeles?

The risk is simply too great and the issues too pressing, to allow a generation or two of common law decisions across multiple jurisdictions to be cobbled together to shape an overarching definition of a "good" search that counsel and clients can rely upon -- one that will stand up to judicial scrutiny.  That's where "ICAIL," the International Conference of Artificial Intelligence and Law, and its Discovery of Electronically Stored Information (DESI) Workshop, comes in.

Actual search practice in litigation is a chaotic cacophony of divergent practices applied with more or less vigor and attentiveness. What is the best approach for any particular case? Much litigation is still stuck on the case-specific level of discussing and testing the application of Boolean search terms. As a result, the more general and overarching issues of search tools and other technologies involved --which form a large part of whether a search methodology passes muster as a whole -- often take a back seat.

The current lack of legally defensible search standards is especially paradoxical given the tremendous strides that we are making in searching as a general rule. The public release of the Enron (and other voluminous) digital data collections has allowed for search algorithm testing and development on genuine ESI collections, and unleashed exciting new search technologies and methodologies. See.e.g.  We now have our terra firma of a sufficient expanse, to conduct trial and error testing to find sustainable methods of searching. Moreover, an increasingly scientific approach to legal search is dispelling intuitive “the world is flat” biases -- such as the belief that the most reliable search (the so-called “gold standard”) is human, linear review of documents. See and A purely linear approach is an invitation to cost and quality disasters, as the court implied in Multiven, Inc. v. Cisco Systems, 2010 WL 2813618 (N.D. Cal. July 9, 2010), and harkens back to Ken Withers' admonishment to move beyond the dark ages of "proto-digital" e-discovery. See also Kenneth J. Withers, "E-Discovery in the Next Decade: Finding a Way out of Purgatory," Keynote Address at Fifth Annual Advanced E-Discovery Institute (Nov. 20, 2009) and

As our search technologies and research advances, these capabilities will need to be incorporated into the definition of a “legally defensible search” that will give comfort to counsel and client, and that will not be second-guessed in the trenches of our adversarial litigation process. Because, ironically, while we struggle for precision in our searches, today we cannot precisely define a defensible search.

This is not to say there is no guidance at all. Federal magistrate judges have provided some invaluable guidance, and The Sedona Conference® as always has provided important thought leadership on general search technologies and approaches. See generally and The Sedona Conference®, Commentary on Achieving Quality in E-Discovery. EDRM has produced excellent search commentaries. And e-discovery blogs such as Ralph Losey’s “E-Discovery Team,” and John Tredennick's "E-Discovery Search Blog,", have also advanced the dialogue. However, we are far from a definition of “legally defensible search” that judges can use to assess search efforts.

Into this vast and uncharted territory, rides the Discovery of Electronically Stored Information (DESI) Workshop of ICAIL, which held its fourth meeting this month at the University of Pittsburgh. The questions posed to the Workshop by Jason Baron and the DESI organizing steering committee composed of Jason Baron, Laura Ellsworth, Dave Lewis, Debra Logan, and Doug Oard to the 150 participants (vendor representatives, academicians, and lawyers) were:

(a) whether e-discovery search is capable of standardization, and

(b) if so what models might be suitable for the standard setting task.

In short, what would a defensible search standard look like? The day was divided between excellent presentations/discussions and small break-out groups working on discrete topics. We were honored and energized by an extended videoconference visit by Federal magistrate judge Paul Grimm, who is well-known for issuing thoughtful opinions on the issue of e-discovery, who enthusiastically endorsed the committee's work and provided a penetrating critique of the impact of the lack of search standards from the perspective of the bench.

I am not the official reporter of the DESI IV meeting and the opinions here are my own, but I think it safe to say that a consensus emerged on a number of points, two in particular:

  • A cross-vendor search standard will not be a particular tool or methodology, but an institutionalized quality assurance process that is required in many industries but that that has not penetrated the legal industry generally, much less e-discovery in particular. For example, manufacturers and retailers refuse to accept supplier products and services that are not “produced” subject to ISO standards. Legal processes should not be an exception. The ISO 9001 certification standard will require quality controls, precision and recall metrics, a proper mix of automated process and human direction and iteration, sampling parameters, exception reporting, refinement, auditing, and a senior management, institutional process commitment to on-going, enhanced measureable and verifiable quality. Clients, lawyers, and courts may ultimately, and perhaps soon, require ISO certifications before accepting search results.

ICAIL’s DESI IV Workshop has launched a long overdue undertaking. There will be little justice without e-discovery and little meaningful ESI without good, defensible search in our exploding digital data ecosphere. See Baron and Losey,"E-Discovery: Did You Know?  

At stake is the legitimacy and integrity of our judicial processes. In trying to pave roads through a near-barren landscape, DESI deserves our unwavering support.

Trackbacks (0) Links to blogs that reference this article Trackback URL
Comments (2) Read through and enter the discussion with the form at the end
Matthew Berman - June 29, 2011 12:52 AM


Great post, this is exactly what I was looking for. This reminded me of a site I came across which is a perfect for finding a
New Orleans Lawyer . Hope you enjoy.



Larry - June 30, 2011 8:14 AM

While I see the possibility of a defined 'process' or the potential for an agreed upon set of 'practices', I think it is a leap to think there could be a "Standard" developed regarding eDiscovery.

Snips from (FAQs):

Q: What is a standard?

A: A standard is a document, established by consensus that provides rules, guidelines or characteristics for activities or their results. (As defined in ISO/IEC Guide 2:2004)

Q: Why are standards important?

A: Standards play an important role in everyday life. They may establish size or shape or capacity of a product, process or system. They can specify performance of products or personnel. They also can define terms so that there is no misunderstanding among those using the standard.

Q: What is conformity assessment?

A: Conformity assessment is defined as any activity concerned with determining directly or indirectly that relevant requirements are fulfilled. (As defined in ISO/IEC Guide 2:2004)

While a standard IS a technical expression of how to make a product safe, efficient, and compatible with others, a standard alone cannot guarantee performance. Conformity assessment, however, provides assurance to consumers by increasing consumer confidence when personnel, products, systems, processes or services are evaluated against the requirements of a voluntary standard.

So from this, at absolute best, a 'voluntary standard' might be developed, but to establish a body of participants that would satisfy the four cardinal principles of "balance, openness, due process and consensus" required to produce a TRUE Standard would required involvement from parties having disparate views on the goals of eDiscovery.

If a Court, or even a body such as Sedona, were to develop such guidance on its own, it certainly wouldn't be viewed as balanced or open without entertaining arguments from opposing views and resolving their concerns to THEIR satisfaction, there would be no due process OR consensus.

The term "Standard" is frequently misused to describe what someone is really seeking to develop- a guideline or recommended practice to establish a common means of achieving something- I think this is what the DESI may want to focus on.

Post A Comment / Question Use this form to add a comment to this entry.

Remember personal info?
Send To A Friend Use this form to send this entry to a friend via email.