The Search for Search Standards: The Hunt at DESI IV

What makes an e-discovery search legitimate and defensible?

While virtually every case involves a search for relevant electronically stored information ("ESI"), there is no industry-based definition or measure of a “legally defensible” search. Reminiscent of Supreme Court Justice Potter Stewart's famous quip, some think we know a good search “when we see it,” but the simple and embarrassing truth is that we do not have an operative definition of search acceptability. The lack of any such industry standard for searching and finding ESI in a case wreaks havoc in the field and leaves it to courts to determine, on a case by case basis, whether a particular search passes muster. Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.d. 251 (D. Md. 2008).

But while judicial officers are many things, they are not search experts. United States v. O’Keefe, 537 F. Supp. 2d. 14 (D.D.C. 2008); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331(D.D.C. 2008). In fact, many judges were elevated to judgeship years or even decades before ESI became prevalent, and thus lack any practical experience in searching for, processing, or producing ESI. Putting the question to the courts will therefore result (and has resulted) in disparate answers that vary from jurisdiction to jurisdiction depending both on how the search issues are presented, and on the quality and quantity of resources each side brings to the court. Do we want an approach that may lead to different search standards, say, in a federal court in Chicago versus a state court in Los Angeles?

The risk is simply too great and the issues too pressing, to allow a generation or two of common law decisions across multiple jurisdictions to be cobbled together to shape an overarching definition of a "good" search that counsel and clients can rely upon -- one that will stand up to judicial scrutiny.  That's where "ICAIL," the International Conference of Artificial Intelligence and Law, and its Discovery of Electronically Stored Information (DESI) Workshop, comes in.

Actual search practice in litigation is a chaotic cacophony of divergent practices applied with more or less vigor and attentiveness. What is the best approach for any particular case? Much litigation is still stuck on the case-specific level of discussing and testing the application of Boolean search terms. As a result, the more general and overarching issues of search tools and other technologies involved --which form a large part of whether a search methodology passes muster as a whole -- often take a back seat.

The current lack of legally defensible search standards is especially paradoxical given the tremendous strides that we are making in searching as a general rule. The public release of the Enron (and other voluminous) digital data collections has allowed for search algorithm testing and development on genuine ESI collections, and unleashed exciting new search technologies and methodologies. See.e.g.  We now have our terra firma of a sufficient expanse, to conduct trial and error testing to find sustainable methods of searching. Moreover, an increasingly scientific approach to legal search is dispelling intuitive “the world is flat” biases -- such as the belief that the most reliable search (the so-called “gold standard”) is human, linear review of documents. See and A purely linear approach is an invitation to cost and quality disasters, as the court implied in Multiven, Inc. v. Cisco Systems, 2010 WL 2813618 (N.D. Cal. July 9, 2010), and harkens back to Ken Withers' admonishment to move beyond the dark ages of "proto-digital" e-discovery. See also Kenneth J. Withers, "E-Discovery in the Next Decade: Finding a Way out of Purgatory," Keynote Address at Fifth Annual Advanced E-Discovery Institute (Nov. 20, 2009) and

As our search technologies and research advances, these capabilities will need to be incorporated into the definition of a “legally defensible search” that will give comfort to counsel and client, and that will not be second-guessed in the trenches of our adversarial litigation process. Because, ironically, while we struggle for precision in our searches, today we cannot precisely define a defensible search.

This is not to say there is no guidance at all. Federal magistrate judges have provided some invaluable guidance, and The Sedona Conference® as always has provided important thought leadership on general search technologies and approaches. See generally and The Sedona Conference®, Commentary on Achieving Quality in E-Discovery. EDRM has produced excellent search commentaries. And e-discovery blogs such as Ralph Losey’s “E-Discovery Team,” and John Tredennick's "E-Discovery Search Blog,", have also advanced the dialogue. However, we are far from a definition of “legally defensible search” that judges can use to assess search efforts.

Into this vast and uncharted territory, rides the Discovery of Electronically Stored Information (DESI) Workshop of ICAIL, which held its fourth meeting this month at the University of Pittsburgh. The questions posed to the Workshop by Jason Baron and the DESI organizing steering committee composed of Jason Baron, Laura Ellsworth, Dave Lewis, Debra Logan, and Doug Oard to the 150 participants (vendor representatives, academicians, and lawyers) were:

(a) whether e-discovery search is capable of standardization, and

(b) if so what models might be suitable for the standard setting task.

In short, what would a defensible search standard look like? The day was divided between excellent presentations/discussions and small break-out groups working on discrete topics. We were honored and energized by an extended videoconference visit by Federal magistrate judge Paul Grimm, who is well-known for issuing thoughtful opinions on the issue of e-discovery, who enthusiastically endorsed the committee's work and provided a penetrating critique of the impact of the lack of search standards from the perspective of the bench.

I am not the official reporter of the DESI IV meeting and the opinions here are my own, but I think it safe to say that a consensus emerged on a number of points, two in particular:

  • A cross-vendor search standard will not be a particular tool or methodology, but an institutionalized quality assurance process that is required in many industries but that that has not penetrated the legal industry generally, much less e-discovery in particular. For example, manufacturers and retailers refuse to accept supplier products and services that are not “produced” subject to ISO standards. Legal processes should not be an exception. The ISO 9001 certification standard will require quality controls, precision and recall metrics, a proper mix of automated process and human direction and iteration, sampling parameters, exception reporting, refinement, auditing, and a senior management, institutional process commitment to on-going, enhanced measureable and verifiable quality. Clients, lawyers, and courts may ultimately, and perhaps soon, require ISO certifications before accepting search results.

ICAIL’s DESI IV Workshop has launched a long overdue undertaking. There will be little justice without e-discovery and little meaningful ESI without good, defensible search in our exploding digital data ecosphere. See Baron and Losey,"E-Discovery: Did You Know?  

At stake is the legitimacy and integrity of our judicial processes. In trying to pave roads through a near-barren landscape, DESI deserves our unwavering support.

Tools for Evaluating and Selecting E-Discovery Vendors

Helpful, effective guidance on evaluating and selecting e-discovery vendors is rare. For one, the constant development of technology makes it difficult to keep up with the latest vendor offerings. Not to mention that more vendors pop up on almost a daily basis.  Court rulings also play a role in changing the e-discovery landscape and therefore the tools needed to keep up with it.

On the scholarly end, The Sedona Conference® published a white paper on e-discovery vendor selection (June 2007), and Gartner, Inc. offered a similar guide for attorneys and litigation support professionals (Dec. 2009). Gartner's guide also provided a market rating on 18 of the biggest players in the e-discovery industry. Some of the important evaluation criteria considered in both reports include:

  • Project identification (scope of work)
  • Vendor's viability (business organization, office locations, sub-contracting, staffing, technical expertise, conflicts, financials)
  • Vendor's market understanding and sales strategies
  • Vendor's experience or work quality
  • Customer feedbacks or client references
  • Products and services
  • Products support and maintenance (document repositories, media type, project/data security, recovery, forensics, project/records management, internal/external IT infrastructure integration)
  • Pricing
  • Delivery (deadlines, project scope changes)

The The Sedona Conference® 2007 includes a sample decision matrix to weigh various factors that are critical to project success.  The 2009 Gartner report provides a rating summary for reviewed vendors. 

But despite these available, helpful guides and road maps, it is still a time-consuming process to conduct internal audits, compile RFI (Request for Information) and RFPs (Request for Proposals), evaluate responses, and utlimately select the appropriate vendor(s) to handle the project.

One possible solution?  Enter Apersee.

Of late, Socha Consulting LLC and Gelbmann & Associates have jointly developed an automated vendor evaluation tool, which they call Apersee.

Still in "beta" mode, Apersee is a ranking tool meant to assist practitioners in selecting e-discovery providers as well as associated products and services. After the user defines and weighs the priorities for an e-discovery company, product or service (a weight is defined on a scale of from 1 (appreciated but unimportant) to 5 (critical & required)), the Apersee Selection Engine sets out to analyze how important a given element is to the decision process by applying a proprietary algorithm to score every product in its database.

If a product or service met all of the requirements, it would receive a score of 100 percent. This scoring method gives preference to criteria the user has designated as most important, and returns the list of matching products or services in ranked order. Essentially, Apersee follows the Electronic Discovery Reference Model (EDRM) in its evaluative process. It includes the following matrix:

  • EDRM stages (information management, identification, preservation, collection, and processing with review and production)
  • Functions (tools, project management, media types, security, capacity, training, support, fee structures and provider integration)
  • Geography and vendor office locations
  • ESI types and languages covered
  • Qualifications (experience, staffing, certifications, cases handled, affiliations and awards)
  • Product type (applications, cloud, SaaS, service)

In most scenarios, parties to litigation have limited time to prepare for e-discovery and vendor selection. Any tools and strategies that can help shorten the decision-making process would prove beneficial. At first glance, Apersee seems to incorporate all the important elements in evaluating e-discovery vendors. Its success will be contingent on the size and quality of its vendor database and the services the vendors offer. It could prove to be a welcoming tool to streamline the arduous process of e-discovery vendor selection.  Only time will tell.

The Wi$dom of Collaborative E-Discovery (Especially in Smaller Cases)

Sometimes it seems like it all comes down to money, doesn't it?  Especially now, so many of us are looking for ways to work - and litigate - smarter, leaner, and cheaper.  Fortunately, some of the finest legal minds are hard at work on solutions to costly e-discovery conundrums, and, luckily, many valuable resources are just a (free!) mouse-click away.

So much of what we know about the outlines of E-Discovery law comes from multi-million dollar, bet-the-company litigation between corporate behemoths.  But what about the everyday family law or small business disputes that are the bread and butter for most attorneys across the country?  How do you get the information you need and protect your client from sanctions without spending more on e-discovery than the case is worth?

Sharon Nelson (who writes the clever E-Discovery Blawg "Ride the Lightning") and John Simek created this helpful podcast highlighting both the e-discovery opportunities and pitfalls for the practitioner in smaller litigation matters.  Key topics include finding a vendor for smaller cases, the dangers of using a company's own internal IT folks on e-discovery projects, how to preserve electronic evidence in a cost-effective manner, and calculating expert expenses. 

A particularly helpful pointer Nelson and Simek offer:  cooperating early and often with opposing counsel, especially in smaller cases, can get you what you need and save you money.  The Sedona Conference® agrees, and its Cooperation Proclamation invites lawyers to abandon the traditional adversarial mode and work more collaboratively during the discovery phase of litigation so that counsel and parties can devote more "time and attention (and money) . . . to litigating the merits of the dispute." 

Wow - cooperating with opposing counsel?  For many of us - whether in-house or outside counsel - that represents a huge cultural shift.  Then again, with a huge percentage of litigation costs devoted to discovery, supporters of the Proclamation argue that, far from abandoning the obligation to be zealous advocates for their clients, lawyers who take a collaborative approach to discovery issues save their clients money and can focus on the substantive legal and factual issues in dispute.  You can read more here and here

In these tough economic times, which have led companies large and small to cut their legal departments and litigation budgets, and with numerous jurisdictions starting to adopt rules encouraging cooperation and collaboration in discovery, collaborative discovery appears to be the wave of the future.  "Cooperative Discovery Bytes" has an interesting ring . . .

Ode to E-Discovery in 2008

Flooding the internet, they consistently accrue:
Blawgs offering e-discovery 'Year in Review's;
But these go on about facts and case histories too,
Before getting to the point of what you can and can't do.

Why not cut to the chase? Why not give it up straight?
Stripped below are the basics of two thousand and eight.
We'll start off with the general dos and the don'ts;
The haven'ts, the shouldn'ts, the emphatically won'ts.

Quite instructive are Canon's and Keithley's examples
Of "lackadaisical attitude" of defendants. As samples:
Do not find that hard drive behind the client's home door,
When discovery has been ongoing for a year or for more.

Do not stumble on computer reports you said "did not exist"
In an e-folder marked "Reports" that you for some reason missed.
And periodically remind clients and their IT personnel
Of the need to preserve the source code that was written on that Dell.

When you don't produce e-mails, the court said in Peskoff
Explain your search method and why, at production, you scoffed.
But if you contributed to information deletion or loss
And the court orders recovery, you won't get your costs!

Do not say you've e-searched when it's just a tall tale:
This was sanctioned under Rule 26 in R&R Sails.
There were costs sanctions also in Ajaxo, among a larger plethora.
And sanction of termination in Arteria and also Pandora.

In Keithley sanctions were imposed even on a party pro se
And in Schwarzenegger for "foot dragging" and a "litany of delays."
But take heed, warned O'Keefe -- don't request termination on whim.
Do not "strike at a king" unless you're sure you'll "kill him."

O'Keefe (plus Equity, Victor) gave lawyers heart attacks.
For saying that search term effectiveness is for experts to crack;
And that if lawyers pick and evaluate the key words instead
They are moving toward places "where angels fear to tread."

The courts warned that when using a method of searching
Learn first of its weaknesses through prior researching.
This was why D'Onofrio rejected what both experts said
And created a brand new search protocol method instead.

Rule 502 on preventing waiver through "reasonable steps"
Saw decisions pronouncing judgment on various missteps.
Alcon acknowledged that the Rule's very recent debut
Was designed to avoid "expensive, painstaking review."

Despite this pronouncement, some courts have cried "waived"
As to attempts made in hindsight to have privilege saved.
Rhoads found possible waiver for documents mistakenly produced
If they were not in the privilege log – there could be no excuse.

And failure to take measures that could prevent waiver
Like claw-back agreements, or Sedona-type saviors
Led to Victor’s conclusion, which uncommonly held
That the attorney-based privilege at issue was quelled.

Moving on, Mancia addressed the Rule 26 obligation
To meet early on regarding e-preservation,
Proclaiming "adversarial conduct" in e-discovery condemned
As a "burden to the American judicial system."

Some courts dove in early to prevent such discord,
Ordering forensic exams to preserve evidentiary records.
To conserve ephemeral info in Xpel, it was fair;
And when defendants were evading service, it was ordered in Allcare.

Other examples included when a party was unable
or unwilling (in Canon) to preserve/produce on the table.
Just remember: as emphasized in Sterle and Square D
Do not interfere with a court-ordered forensic decree.

Rodman, Reinhard and Younessi addressed nonparty subpoenas
And the protection of confidential, trade secret arenas.
Where nonparties are concerned and offer up much resistance
In-house searches are fine, or neutral expert assistance.

The debates continue on metadata versus non-native tracks
And Aquilar labeled metadata as being "the new black."
That court ordered re-production of non-natives with meta
Though the recipient was required to pay costs, as pro rata.

But not all courts required conversion to a metadata mode.
Extra burden led D'Onofrio to an "only if necessary" ode.
And Autotech said doc requests must actually require "native" --
You can't ask for it in hindsight by getting creative.

Yet if e-documents already exist in original native form
And the requests do not contain any language that informs,
White condemned the conversion to non-native in litigation
Since this is done just to increase the opponent's frustration.

Finally, social networks are making an appearance in law
And becoming a most popular e-discovery draw.
The field is wide open on the extent to which these
Are discoverable and admissible, or cannot be seized.

Flagg required defendants to give ISPs consent
And to produce ISP-retrieved records of texts that it sent.
And in Australia a court made clients even more nervous
By allowing Facebook to be used as a method of service!

We hope you've enjoyed this short "Year in Review"
And that all of this knowledge is useful to you.
We await more developments in two thousand and nine;
And wonder whether and where courts will draw any lines.


**For a complete list of the cases discussed above, please contact the author.