Are all search results equal?

                Ever since George Socha and other analysts called out the raw size of the eDiscovery market and the rapid growth of technology sales driven by corporations efforts to become compliant with the new Federal Rules of Civil Procedure (FRCP), software developers with dollar signs in their eyes have adapted existing products for the eDiscovery market. The problem is that many or most of them have no real understanding of the unique requirements and challenges of legal search. The Information Lifecycle Management (ILM), Knowledge Management (KM) and Enterprise Content Management (ECM) movements have left the market littered with interesting applications that the corporate market was not yet ready to pay for. Many of these products are now showing up dressed in an eDiscovery GUI and being sold as eDiscovery solutions without even rudimentary testing on the myriad types of corporate ESI.

                Just because a system is marketed for use in eDiscovery does not relieve the service provider or corporate legal department from the responsibility to conduct fundamental validation testing of the systems basic functions. Too many customers believe that the software company somehow warrants or guarantees the accuracy of search results, just because it was sold specifically for legal search. The entire industry has suffered from a bit of willful blindness for several years now. Blaming inaccurate results on the software or appliance will not get you much pity from the judiciary when you fail to produce large portions of custodians communications because of the myriad ways that Microsoft Exchange can retain addresses.

                At the recent Electronic Discovery Reference Model (EDRM) 2008-2009 project kickoff conference, I was pleased to participate in the formation of a new group focused on legal search. The primary goal of this group is to explore how to document, audit and defend the search process used. The recent Sedona Conference commentaries for Working Group 1 have also called out the need to validate search, culling, review and all other methods that comprise your discovery response process. This is exciting stuff and not just for us techno-junkies.

                Even when search engines perform exactly as designed, counsel and their technical support need to understand how the systems handle their unique blend of corporate ESI. The configurations of enterprise messaging and file systems change over time and that actually changes the characteristics and metadata available to search engines. Consider the major changes in how message envelopes are formatted between Exchange 2003 and Exchange 2007. Now consider the fact the most corporate major upgrades are spread over quarters and you realize that almost every large enterprise environment has multiple systems that are running in a mixed mode (i.e. more than one version of the system running simultaneously).

                All of this does not mean that it is impossible to conduct a reasonable, defensible search, preservation or collection. Instead it points out the need to understand your environment and to document the efforts made to test and analyze the capabilities and exceptions of potential eDiscovery technology before putting them to use. With more innovative CIO's looking to implement enterprise search and analytics, it is critical that the legal department collaborates on the system requirements and testing process.

This concept of test before you trust applies to service providers as well as applications. The Autonomy Book on Compliance, Risk, Legal and Regulatory Technologies gives excellent basic guidelines for key testing considerations in a Proof of Concept scenario. They expand on these basic concepts in a full whitepaper titled, "A Guide to a Successful PoC: The Dos and Don'ts for Selecting Unstructured Data Management Software." We need more awareness of the pitfalls around implementation and adoption of untested technology and process. Do not let your faith in your existing system keep you from giving it a pop quiz.