The two most commonly used technology-led approaches to electronic disclosure are:

1. 1. 1. The application of keywords to a pool of documents; and
      2. The use of predictive coding to a pool of documents.

Both approaches above combine the use of technology with a manual review element.

This note explains the benefits of using a predictive coding approach to a large-scale electronic disclosure exercise compared to the use of keywords.

Parties to a complex electronic disclosure exercise are encouraged by the Court’s rules to use technology in order to undertake a proportionate review.

Paragraph 9.6 of PD57AD confirms that the parties should seek to reduce the burden and cost of the disclosure exercise and ways of doing this can include[1] the use of:

“(3)(a) software or analytical tools, including technology assisted review software and techniques; and

(b) coding strategies, including to reduce duplication”.

The explanatory note to Section 2 of the Disclosure Review Document also provides that parties should use a technology assisted review (“TAR”) to conduct a proportionate review of the data set (particularly if the review data set is likely to be in excess of 50,000 documents)[2].

The majority of case law and secondary sources on the use of TAR originates from American law. However, the use of TAR has been considered and approved by the English Court in Pyrrho v NWB[3] (which relied on the U.S. precedent of Moore v Publicis Groupe[4]) and Brown v BCA trading. [5]

Predictive Coding is one automated TAR method and is recognised as such in the definitions appended to PD57AD (the terms Predictive Coding/TAR will be used interchangeably in the remainder of this note).

Keywords (and by that, we mean Boolean search expressions[6]) only find documents that exactly match the search term being used. For example, a search for “cat”, “dog”, “house,” or “bank,” would return documents with one or all of these terms exactly as they are keyed into a search problem.

Keywords also may miss conceptually or contextually similar documents to the disclosure issues that do not contain the exact keywords. When considering the number of contextually false hits in a data set that contains millions of documents, the risk of developing an over-inclusive set of search terms grows into the very liability of added expenses in terms of cost, labour, and time.

In a traditional keyword driven linear review, the legal team undertakes a review of documents returned by keyword searches.   There is no process for applying the results of the legal team’s review to other documents. If Document 1 is marked as relevant/not relevant this has no automatic impact on Document 2 etc.

In contrast, TAR will continue to revise, refine, and improve its internal model based on the decisions made by the review team.  TAR uses the decisions made by the review team to find contextually and conceptually similar documents from the pool of potentially relevant documents. As further batches are reviewed (and subsequent batches will also include documents which TAR has marked as not relevant for QC purposes) the model continues to ‘learn’.

The premise behind predictive coding is to locate documents comparable to those that have been determined relevant or not relevant by the legal team. It is based on binary rating systems that help to prioritise documents based on their relevance. However, the computer does not impose its own judgments about the responsiveness of documents; instead, it seeks to match the decisions made by the legal team. This process can dramatically drill down the number of documents in extensive collections to include only those relevant to a specific case.

In practice TAR enables the review of a significantly lower number of documents than a set of documents retrieved by keywords alone. This is because TAR learns from the decision of reviewers, scores documents for prospective relevance and prioritises documents that are most likely to be relevant for first level review by lawyers.

The result is that the TAR will produce a more targeted set of documents for review by the legal team.  In contrast keyword searches will produce all documents which respond to a keyword regardless of relevance.

It is not the case that keywords can be used firstly to cull data volumes and then using TAR over the keyword hit documents. This deprives the TAR methodology of assessing the entire pool of documents, which is excluded if keywords alone are used; because keywords alone do not pick up contextually or conceptually similar documents.

A simple analogy is that predictive coding will use automatically all synonyms for a word whereas keyword searching is strictly limited to the word or phrase itself.

Our e-disclosure team has undertaken a review of recent e-disclosure matters to understand the impact of TAR when compared to keywords.[7] This analysis show that TAR applied over the entirety of a review pool meant that the number requiring review was approximately 40% of the number that keyword searching identified as requiring review.  In addition, TAR identified a significant number of relevant documents that had not been recognised by any keyword searches

A landmark study[8] in 1985 revealed that attorneys, using search terms and iterative search, supervising skilled paralegals believed they had found at least 75% of the relevant documents from a document collection when they had in fact found only 20%.

In March 2022, the University of Colorado Law Review published a paper titled ‘Robophobia’ authored by Professor Andrew Woods from the University of Arizona.[9] This article discusses the uses of algorithms and the application of computer learning in different areas of life.  In part of this, the article discusses the use of TAR in disclosure and evidence:

“Robophobia crops up in civil litigation as well. It has been shown that machines are better than humans—faster, cheaper, more thorough—at many aspects of document review and related discovery tasks, especially over large datasets.[10] Not only are machines more effective than humans at certain kinds of reviews but lawyers are especially bad at them.[11] Lawyers are good at the interpretive task of identifying whether a particular document is responsive or not, but they are much worse at accurately plucking the relevant documents from a large stack of irrelevant material.[12] So we might imagine that lawyers would benefit from systems where a machine identifies a potential set of documents and lawyers then do the “last yard” of review to determine which documents in the smaller set are, in fact, relevant.

But lawyers generally decline to trust artificial-intelligence tools to conduct document review, despite the evidence that they can work. Surveys of lawyers show a reluctance to rely on technology-assisted review when compared to having lawyers make relevance determinations about every single document.[13] A recent survey of practicing attorneys found that only 31.1 percent of respondents use Technology Assisted Review (TAR) in all or most of their cases.[14] This is so despite the obvious efficiency and accuracy benefits of TAR.[15]

This reluctance is somewhat hard to understand. First, lawyers regularly turn to computers to conduct keyword searches—and, in fact, there is evidence that lawyers tend to be overconfident in the responsiveness of these results.[16] Additionally, lawyerly reluctance to use AI might have once been explained by a fear that these determinations would not hold up in court.[17] But, today, “it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it.”[18] So a fear about judicial acceptance hardly explains attorneys’ widespread reluctance to use robots more thoroughly in document review.[19]”

Through the use of TAR, in particular predictive coding, parties can realise a significant decrease in disclosure costs. For example, in several case studies, legal teams have reported reviewing 64 to 93 percent fewer documents with predictive coding due to the defensible identification and exclusion of non-responsive materials, which is confirmed through quality control sampling [20]. In a survey of 11 predictive coding vendors, 4 reported an average cost reduction of 45 percent, while seven of the vendors reported savings as high as 70 percent.[21] Similarly, another study found that the time and cost it takes legal team to conduct document review could be cut by 80 percent. [22] Legal teams have also reported that the use of predictive coding has allowed them to meet difficult deadlines.[23].

Bibliography

[1] See paragraph 9.6(3) of PD57AD.

[2] Paragraph 6(6) of the explanatory note to Section 2 of the DRD.

[3] Pyrrho Investments Limited v MWB Property Limited (2016) EWHC 256 (Ch)

[4] Moore v. Publicis Groupe, 287 F.R.D. 182, 191 (S.D.N.Y. 2012)

[5] Brown v BCA [2016] EWHC 1464 (Ch)

[6] A Booelan search is a structured search process that allows the user to insert words or phrases such as AND, OR, NOT to limit, broaden and define the search results.

[7] XBundle Statistics drawn from client work product within the last twelve months, the subject matter of which is privileged and confidential.

[8] David C. Blair and M. E. Maron ‘An Evaluation of Retrieval Effectiveness for a Full-Text Document- Retrieval System’ (March 1985), Communications of ACM, 28,3. (link here – An evaluation of retrieval effectiveness for a full-text document-retrieval system (acm.org))

[9] Andrew R. Woods, ‘Robophobia’ (March 17, 2022) U.C.L.R 93, 1, Robophobia – University of Colorado Law Review

[10] Maura R. Grossman & Gordon V. Cormack, ‘Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient than Exhaustive Manual Review’ (2011) 17 Rich. J.L. & Tech. 1

[11] Sam Skolnik, ‘Lawyers Aren’t Taking Full Advantage of AI Tools’, (May 14, 2019), Bloomberg L. https://news.bloomberglaw.com/business-and-practice/lawyers-arent-taking-full-advantage-of-ai-tools-survey-shows (reporting results of a survey of 487 lawyers finding that lawyers have not well utilized useful new tools)

[12] Moore v. Publicis Groupe, [2012] 287 F.R.D. 182, 191 (“Computer-assisted review appears to be better than the available alternatives, and thus should be used in appropriate cases.”).

[13] Bob Ambrogi, ‘Latest ABA Technology Survey Provides Insights on E-Discovery Trends, Catalyst: E-Discovery Search Blog’ (Nov. 10, 2016), https://catalystsecure.com/blog/2016/11/latest-aba-technology-survey-provides-insights-on-e-discovery-trends (noting that “firms are failing to use advanced e-discovery technologies or even any e-discovery technology”)

[14] Doug Austin, ‘Announcing the State of the Industry Report 2021, eDiscovery Today’ (Jan. 5, 2021), https://ediscoverytoday.com/2021/01/05/announcing-the-2021-state-of-the-industry-report-ediscovery-trends/

[15] Ibid

[16] David C. Blair & M. E. Maron, ‘An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System’, (1985) 28 Commc’ns ACM 289

[17] Thomas E. Stevens & Wayne C. Matus, ‘Gaining a Comparative Advantage in the Process’, (Aug. 25, 2008) Nat’l L.J, https://www.law.com/nationallawjournal/almID/1202423952310/

[18] Rio Tinto PLC v. Vale S.A (S.D.N.Y. 2015), 306 F.R.D. 125, 127

[19] The Sedona Conference, ‘The Sedona Conference Best Practices Commentary on the Use of Search & Information Retrieval Methods in E-Discovery’, (2014) 15 Sedona Conf. J. 217, 235-236

[20] Scott M. Cohen, Elizabeth T. Timkovich, and John J. Rosenthal, ‘The Tested Effectiveness of Equivio > Relevance in Technology Assisted Review’ (Dec, 2011) Metro. Corp. Couns. 17, 8

[21] Anne Kershaw and Joseph Howie, ‘eDiscovery Institute Survey on Predictive Coding’ (October, 2010) eDiscovery Institute

[22] Jason R. Baron, Ralph C. Losey and Michael D. Berman, ‘Perspectives on Predictive Coding’ (2016) ABA

[23] Ibid, 208