Court Orders Random Sample of the Null Set in City of Rockford v. Mallinckrodt ARD Inc., No. 17 CV 50107, No. 18 CV 379 (N.D. Ill. Aug. 7, 2018)

In a set of cases involving one of our clients, U.S. Magistrate Judge, Iain D. Johnston sides with the Plaintiffs and orders a random sampling of the null set after production in order to determine the quality of the production.  Judge Johnston’s humor shines throughout the order as he references the Simpsons, scary clowns, and Donald Rumsfeld. These cases involve alleged breach of contract, antitrust violations, and racketeering by Defendant in pricing its medication, Acthar. The cases were consolidated for discovery purposes. The court commended the parties’ amicable approach to discovery, and the parties generally agreed on ESI protocol, including search terms and ways to resolve discovery disputes.

The only issue in dispute were the steps to be taken after production to validate the produced set. The Plaintiffs proposed that a random sample of the null set be taken and reviewed to determine what percentage of relevant documents were missed. The Defendants proposed that if “the requesting party reasonably believes that certain categories of requested documents exist that were not included in the production, the parties will meet and confer to discuss whether additional terms are necessary.”  Defendants also claimed that a random sampling of the null set is unreasonable under Rule 26(g) and “burdensome” or disproportionate, under Rule 26(b)(1).

In siding with the Plaintiffs, the court found that a random sampling of the null set was both reasonable under Rule 26(g) and proportionate under Rule 26(b)(1). As Judge Johnston notes, sampling of the null set is a standard and common practice during TAR review. DaSilva Moore v. Publicis Groupe, 287 F.R.D. 182, 201 (S.D.N.Y. 2012). That sampling provides a picture of how accurate the TAR process was so that the results can be verified. Saying “just as it is used in TAR, a random sample of the null set provides validation and quality assurance of the document production when performing key word searches,” Judge Johnston applies the same logic to the null set in a keyword search production. In fact, randomly sampling irrelevant documents following keyword search productions was a common practice prior to the use of TAR.

The court also found that random sampling of the null set was proportionate under Rule 26(b)(1). The Defendants argued that sampling would be burdensome and expensive. However, the court found that a random sample will not be voluminous given the context of these cases and the extraordinary amount in controversy.

WHAT IS THE NULL SET?

Judge Johnston, quoting Donald Rumsfeld, speaks of the “known knowns” and the “known unknowns.” When it comes to ESI, the “knowns” are the documents that are produced through keyword searching or Technology Assisted Review (TAR). The “unknowns” are the documents that weren’t produced by the chosen retrieval process. These unproduced documents are the null set. Sampling the null set is vital to validate the original retrieval process, whether it was TAR or keyword search, so that the parties can get a sense of how many relevant documents were missed. This allows the parties to validate their process or modify it if too many relevant documents were missed.