Four Lessons Counsel can learn about Da Silva Moore and Predictive Coding

There’s good news in the world of electronic discovery. This February in New York, Magistrate Judge Andrew Peck and counsel for the parties in Da Silva Moore v. Publicis Groupe gave us a magnificent e-discovery lesson and pushed open the door for the utilization of advanced search technologies -- namely predictive coding, an increasingly used methodology of computer-assisted review.

The Case

The plaintiff filed a Title VII class action gender discrimination claim against defendant Publicis Groupe, alleging she and other female employees at Publicis Groupe endured discriminatory terminations, demotions and job reassignments. The plaintiff (who had very little, if any, electronically stored information (ESI) of her own to produce) demanded that Publicis Groupe produce documents (including ESI) that related to whether Publicis Groupe:

  1. Compensated female employees less than comparably situated males through salaries, bonuses or perks.
     
  2. Precluded or delayed selection and promotion of females into higher-level jobs held by male employees.
     
  3. Disproportionately terminated or reassigned female employees when the company was reorganized in 2008.

Based on the records requested and the number of custodians, the parties anticipated the document pool would be around three million documents, which would have likely cost in excess of $1 million with traditional keyword search methods. Instead of going this route, the parties agreed to something bold: review the documents using what has come to be called predictive coding, a methodology of computer-assisted review. By using these methods, the parties hoped to reduce the number of manually-reviewed documents from 3 million to 20,000.

The implementation of predictive coding is not simple. Fortunately, Da Silva Moore v. Publicis Groupe provides a lengthy guide on important topics such as methods to identify the initial seed set, iterative training rounds to refine the “predictive coding to assure reasonable recall” and methods of sampling to validate levels of confidence and confidence intervals.
 

What the Case Means for Your Business

Although discovery in Publicis Groupe is far from over, and the parties have each filed motions challenging portions of Judge Peck’s ruling, there are already lessons to be learned for how to effectively deploy computer-assisted review to reduce the cost of electronic discovery in your cases:

1. Have an expert, knowledgeable about the review tool you intend to use.  Judge Peck turned to the parties’ technical experts to explain the effect of the review protocol on the validity of the ultimate production. Surely judges less familiar with the technology could benefit from hearing from an expert in the field. Since experts tend to disagree (as they did in Publicis Groupe), it’s an absolute requirement to provide testimony about the operation and testing of the search tool chosen for the case.

2. Be willing to accept that you will not receive every potentially relevant document. Judge Peck put it best when he reminded counsel, “By the time you go to trial, even with six plaintiffs, if you have more than 100 trial exhibits it will be a miracle.” He also explained that, “The idea is not to make this perfect, it’s not going to be perfect. The idea is to make it significantly better than the alternative (human review) without nearly as much cost.”

Consequently, you have to be willing to risk that a computer will miss more documents than the recent law grad you would normally pay to sift through each page. Keep in mind that human review and key word search strings are far from perfect. Predictive coding when properly applied will likely enhance both recall and precision.

3. Cooperate with the opposition. The utilization of this technology requires engaged cooperation between the parties. Counsel must review and share the initial seed set with the opposition, and agree on statistical sampling techniques. Notwithstanding subsequent disputes, Da Silva Moore v. Publicis Groupe illustrates competent counsel working closely on e-discovery to meet the interests of both the plaintiffs and the defendant. Keep in mind that both sides agreed to utilize this advanced technology in this case. Of course, the devil is in the details, where reasonable litigants can disagree.

4. Understand the technology.  Even with technology experts at the ready, counsel were still necessary to advocate for their client’s interest in balancing the cost of discovery against the completeness of the final set of documents produced. Predictive coding is not right for all cases. It is not inexpensive—counsel must expend considerable up-front fees identifying the seed set and fine tuning the technology.

Touted as a practical, cost-saving and revolutionary solution, computer-assisted review is finally getting its chance to show what it’s worth. The private bar is watching anxiously to see whether it lives up to its billing.
 

 

This article was originally published in Inside Counsel.

"Because Something Must Be Done": The Dangers of Trying to Save E-Discovery Costs by Treating Data Like Paper

This article was published by Wendy Akbar and Bill Hamilton.

Remember the days before the computer, the typewriter, the printing press, and even carbon paper? The days when, to copy a book, one needed to sit down and re-write it by hand? Every letter of every word of every sentence-dappled paragraph, had to be painstakingly copied one by one. With all the technology available today, no publisher would ever consider copying a book by hand rather than re-printing a copy saved on the computer. To do so would be a waste of time -- a return to the Dark Ages.

When it comes to e-discovery, unfortunately, the Dark Ages still occasionally guest stars in modern-day electronically stored information (ESI) retrieval and production. The dangers of being such an e-discovery ostrich were most recently highlighted in Multiven, Inc. v. Cisco Systems, 2010 WL 2813618 (N.D. Cal. July 9, 2010). Plaintiff Multiven, along with the two counterclaim defendants, originally elected to undertake a manual review of the full set of voluminous ESI it possessed prior to production. Sound conscientious? Not exactly. It means they turned the clock back about 15 years (in e-discovery time, akin to 150 years) and:

(1) Refused to use an outside vendor to help organize ESI information;
(2) Refused to utilize any search terms to narrow the "giant mass" of data to be reviewed;
(3) Instead used approximately five attorneys for six months to a year, to manually review every unfiltered page of "that giant mass" for responsive documents.

Why? They wanted to save on cost, particularly the expense of hiring an outside vendor to help whittle down potentially responsive information. Perhaps they adopted a manual review for other strategic reasons. (Never mind, of course, the hourly billing rate of five attorneys doing eight hours of document review five days a week for over six months, which more likely than not was more expensive than hiring a vendor to narrow the "giant mass" to a more reasonable review load for the attorneys).   The end result, however, was not exactly what was intended . . .
 

What sounded like a Great Idea to Multiven ultimately backfired. The process bogged everyone down, taking months and months. Finally, the district court became distressed when it became apparent that the delays associated with the manual review of all electronic files was causing the case to extend well beyond the boundaries of the scheduling order:

[I]t has become clear to this Court that [the plaintiff and counterdefendants] cannot complete their review and production of documents with enough time before the close of discovery to allow Cisco time to actually do anything with them. [They] so far have insisted on a review process that guarantees that they will not finish this extensive project in any reasonable amount of time.

Id. at *2. The trial court thus adopted the Report and Recommendation of Magistrate Judge Lloyd, required Multiven to utilize a vendor to assist in e-discovery, and appointed a Special Master to manage certain e-discovery disputes that potentially threatened to derail the litigation. In short, the Court concluded (id. (emphasis added)):

Because something must be done, this Court recommends that the District Court order the parties to promptly retain a third party vendor to assist with this increasingly perilous situation.

So first, Multiven was not only stuck having to hire the very vendor it had hoped to avoid, but on top of it had to pay its attorneys the hundreds of thousands it cost to conduct the months of document review that were done to avoid hiring a vendor in the first place. Instead of saving money, their gamble multiplied the cost of e-discovery in the case. Second, by this point, even Multiven was so fed up with the document review process that it was amenable to hiring a vendor before the court order even issued. Sure, part of it was because the defendant, Cisco, had offered to pay for part of the vendor cost, but Multiven had finally learned that it was in over it head, and that in trying to save money, it had wound up spending significant more cost on e-discovery.

While the Court's intervention was necessary to get the case back on track, one wonders what possessed the Multiven and its counsel to undertake a prolonged manual review, especially when Cisco appeared to be urging Multiven to employ modern technology all along. Delay was only one by-product of Multiven's decision. A second by-product was the likely extraordinary costs such manual reviews entail. The most substantial e-discovery costs arise from the attorney review process, regardless of whether that review is done internally by firm lawyers or outsourced to vendors in the United States or abroad.

One might think that a manual review is the "gold standard" and the most thorough, comprehensive way of searching. In other words, while you can place fistfuls of hay at a time into a hay-sorter to separate out those few stray needles, there's still a margin of error. You're more likely to find them if you use the time-consuming method of picking up one strand at a time and eyeballing it. Right? Oddly, no. In truth, the so-called gold standard of manual review is mythological, and the human eye can miss a lot more than a well-crafted keyword search protocol can capture. A widely-regarded study, Blair et al., Wittgenstein, 'Language and Information:  Back to the Rough Ground!' 302 (2006), found that manual review teams only found 20% of the relevant documents!  Ralph Losey and Jason Baron have also demonstrated that adding conceptual search tools to key-word searching further improves results and reduces cost. So Multiven's Dark Ages strategy not only multiplied costs and delayed the proceedings, but likely would have missed most of the relevant documents -- leading to potential sanctions and do-overs. Some cost-saving strategy!

In truth, there are very limited circumstances in which a case or production might require every e-document to be reviewed: the all-consuming, unfiltered, manual document review is dead, and for good reason. Today's staggering volume of ESI makes such a manual review both cost and time prohibitive. Utilizing the broad array of tools to cull down, de-duplicate and search data volumes is generally accepted and is even becoming mandated practice, as Multiven learned too late. While all searches are not created equal, the Text REtrieval Conference (TREC) Project -- co-sponsored by the National Institute of Standards and Technology (NIST) and U.S. Department of Defense-- has demonstrated a better search results by machines than plowing through data manually. While a search methodology utilized by a party has to be defensible, standards are slowly emerging, as documented by The Sedona Conference® in its May 2009 Commentary on Achieving Quality in the E-Discovery Process: Best Practices for Document Retention and Production. There is no reason to be stuck in the Dark Ages of linear, manual review.