The combination of the 2000-2002 public corporate scandals and heavy sanctions like the Zubulake v. UBS Warburg LLC, et al., S.D.N.Y 02 CV 1234 (SAS) 7/20/04; 2004 U.S. Dist. LEXIS (S.D.N.Y, July 20, 2004) case resulted in panicked phone calls from Legal to IT asking, "Are we keeping everything?"We know that IT was not keeping EVERYTHING before, but suddenly tape backup systems were repurposed from one month disaster recovery systems to impromptu preservation systems. Even if the company has faced those growing stockpiles of tape and conquered that mountain, the revised FRCP are so new that most General Counsels have been reluctant to approve of the resumption of destruction policies without a full assessment and revision of existing paper-based policies.

 This leaves many companies, municipalities and other entities struggling with whether to tackle their historical collections before or after putting new systems and policies in place. If they are not careful, it can paralyze their new retention initiative and leave them at even greater risk for increased cost and even sanctions. It is important to remember that the judiciary is always more critical of your current efforts to satisfy your retention and production obligations as compared to your historical 'gaps'. In this light, we can see that the first priority is to put the process, technology and people in place to prevent ongoing data loss and control the costs as the company moves forward. You cannot change the past, but you can change your capabilities to meet current and future challenges.

Does that mean that you just ignore all the old ESI? Do you just ignore the terabytes of 'Litigation Landmines' (PST files) living on local and network drives until a litigation request demands them? That is actually a business cost versus risk decision that every entity above a certain size must make in light of their regulatory or litigation profile. It is definitely not 'best practice' to leave all that unstructured content out there, but the current business environment does not always allow us the luxury of true 'best practice' solutions without a clear short term ROI justification.

In recent blogs, I have talked about how these early enterprise analytic products have tried to transform themselves into e-Discovery 'solutions' with mixed results. It is sometimes the secondary analytical features of these products that can provide an unexpected ROI far beyond the discovery compliance driver that got the budget out of legal. Although opposing counsel and many courts are not yet ready to rely exclusively on conceptual search and other 'black box' (i.e. not easily understood) technologies to winnow the wheat from the ESI chaff, these systems can dramatically reduce the cost of review and provide essential QA for Boolean criteria.

More relevant to IT, HR and Compliance needs, enterprise systems can be leveraged to shape data management policies, enforce computer/messaging usage policies and monitor a growing concern, Data Loss Prevention (DLP). These peripheral usage scenarios have not been a big enough pain point to drive widespread adoptions and budget allocation outside of tightly regulated verticals, but they are happy to make use of the new enterprise applications and appliances funded by legal.

In fact, the fastest ROI on search or archiving platforms frequently comes from resolving potential litigation through Early Case Assessment searches. I recall answering a simple request from a new director regarding the communications over a multi-million dollar contract by the former director. The company had just finished loading all the old email from tapes into an archive to comply with a discovery request, so we had everything possible online. The savings from finding that one email clearly expressing the mutual understanding of a term saved the company twice the price of the archive system. Instant ROI that had nothing to do with the original purpose of the system.

The biggest potential benefit of conceptual filters and categorization engines is to shift the corporation's capability to a proactive stance, rather than a reactive response. Data Loss Prevention (DLP) is all about preventing critical trade secrets and confidential information from reaching the wrong eyes. The same technology that helps review teams catch potentially privileged email before they are produced can be used to stop your pricing lists and source code from leaving the domain. With the widest ESI type capability, it is not surprising to hear that Autonomy's IDOL platform is used by a majority of the DLP market leaders. The focus is on the messaging systems, but endpoint control and enterprise encryption are also starting to make sales, again driven by the complexity of collecting and preserving ESI from user desktops. So when legal is fighting for budget to address a discovery gap, it is critical to involve the other departments who have a vested interest in gaining these new capabilities and slowly moving the corporation towards the goal of true information management.

                Corporate counsel may then turn to the vendor that sold them an 'enterprise solution' and demand an expert to defend the search engine. Some of the vendors have made the serious investment to validate their technology through 3^rd party testing or by recruiting resident experts. The reality of our industry is that there have been few if any public challenges to the accuracy and completeness of software sold to the e-discovery market. Until recently, most of this software was purchased and run by service providers who had an interest in defending the product that they proposed and operated behind the magic curtains.

                The majority of applications now focused on the corporate e-Discovery spend were originally created for information management, knowledge management, process analytics, storage management, disaster recovery and other non-litigation business needs. Enterprise search engines were mostly born from the research into scaling internet search engines that started back in the early 1990's. The problem with web based search is that the systems are optimized for speed and relevance, rather than completeness.  A recent Autonomy release drew attention to the engine 'jump out' issue, where some systems actually stop or skip indexes when they think it unlikely to find a match.

                Not all applications that use search engines created for the internet are unsuitable for discovery search. The index engine 'schema' or configuration controls how it crawls through files to extract out the information needed to get search results as well as how it behaves while running a search. Many enterprise applications like email archives and document management systems utilize open source or OEM'd index engines like FAST, IDOL and Lucene. Most of these index engines are dependent upon outside software to recognize divergent types of ESI and extract the text in discrete words (a process called tokenization) with a few exceptions like Autonomy's IDOL platform. The configuration and options in a system can result in dropping out critical ESI from the index/search or it can bring the system to its knees by forcing it to try to index file types which contain no text or character sets that it cannot handle.

                The critical take away from these new cases is that you need to know the capabilities and limitations of your chosen search engine(s). Not just what the software provider tells you it can do, but actually how it performs on your ESI. Think about standing before a court and answering the question, "How do you know that your search got everything that matched your criteria?" Judges are not seeking some impossible level of absolute certainty, but they expect to hear about your reasonable effort to validate your tools.

                The legal ramifications of the case are better left to attorney interpretation, but U.S. Magistrate Judge Paul Grimm made several interesting statements that give insight into how the increasingly sophisticated judiciary is raising the bar on search. The first leads in with, "there is a growing body of literature that highlights the risks associated with conducting an unreliable or inadequate keyword search or relying exclusively on such searches for privilege review."

                So the bedrock of retrieval, the venerated keyword search, does not stand all on its own. In this case, the seventy-odd search terms were neither disclosed to, nor agreed upon by the opposing party. The terms were created by counsel and client, then executed by their hired gun. All the items that could not be indexed were manually reviewed for privilege, but it appears that no one reviewed the 'searchable' items that did not contain the list of search terms before being produced.

                Judge Grimm continues, "Common sense suggests that even a properly designed and executed keyword search may prove to be over-inclusive or under-inclusive, resulting in the identification of documents as privileged which are not, and non-privileged which, in fact, are. The only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents determined to be privileged and those determined not to be in order to arrive at a comfort level that the categories are neither over-inclusive nor under-inclusive."

                Upon receiving the production, the opposing party used a "readily-available desktop search tool." to find some of the formerly privileged items. It is unfortunate that the order does not specify the technology used by each side, but it is clear that quite a bit of effort was expended to manually wade through PDF and TIFF files that could have been searched with the right tools. Even hiring a 'computer forensic expert' did not save the Defendants' from their own decisions.

                Keywords are easy to understand and even easier to misuse. Beyond the science of 'precision and recall' there needs to be process, communication and metrics to reach the standard of reasonable due diligence, this 'comfort level' that Judge Grimm cites. The process needs to identify the known exceptions to your chosen technology. Have you asked or tested for the ability to search across different formats of email, files, databases, images, voice, video and languages? Do you even know the composition of your ESI collection and how that will affect any searches?

                Just because a system is marketed for use in eDiscovery does not relieve the service provider or corporate legal department from the responsibility to conduct fundamental validation testing of the systems basic functions. Too many customers believe that the software company somehow warrants or guarantees the accuracy of search results, just because it was sold specifically for legal search. The entire industry has suffered from a bit of willful blindness for several years now. Blaming inaccurate results on the software or appliance will not get you much pity from the judiciary when you fail to produce large portions of custodians communications because of the myriad ways that Microsoft Exchange can retain addresses.

                At the recent Electronic Discovery Reference Model (EDRM) 2008-2009 project kickoff conference, I was pleased to participate in the formation of a new group focused on legal search. The primary goal of this group is to explore how to document, audit and defend the search process used. The recent Sedona Conference commentaries for Working Group 1 have also called out the need to validate search, culling, review and all other methods that comprise your discovery response process. This is exciting stuff and not just for us techno-junkies.

                Even when search engines perform exactly as designed, counsel and their technical support need to understand how the systems handle their unique blend of corporate ESI. The configurations of enterprise messaging and file systems change over time and that actually changes the characteristics and metadata available to search engines. Consider the major changes in how message envelopes are formatted between Exchange 2003 and Exchange 2007. Now consider the fact the most corporate major upgrades are spread over quarters and you realize that almost every large enterprise environment has multiple systems that are running in a mixed mode (i.e. more than one version of the system running simultaneously).

                All of this does not mean that it is impossible to conduct a reasonable, defensible search, preservation or collection. Instead it points out the need to understand your environment and to document the efforts made to test and analyze the capabilities and exceptions of potential eDiscovery technology before putting them to use. With more innovative CIO's looking to implement enterprise search and analytics, it is critical that the legal department collaborates on the system requirements and testing process.

This concept of test before you trust applies to service providers as well as applications. The Autonomy Book on Compliance, Risk, Legal and Regulatory Technologies gives excellent basic guidelines for key testing considerations in a Proof of Concept scenario. They expand on these basic concepts in a full whitepaper titled, "A Guide to a Successful PoC: The Dos and Don'ts for Selecting Unstructured Data Management Software." We need more awareness of the pitfalls around implementation and adoption of untested technology and process. Do not let your faith in your existing system keep you from giving it a pop quiz.

A conceptual engine essentially lets the ESI talk for itself. Rather than a person creating a search from their preconceptions of what criteria will retrieve all items related to a given request, the systems analyze and present the items back as folders, dot clusters and other visual diagrams to help the user make sense of the complex relationships. All of this sounds like just what the attorney asked for. "Give me everything relating to this deal."

But what one should consider is that concepts are just groups of words or phrases that express similar ideas. They do not directly answer the typical request for production demanding 'any and all documents or communications that constitute, contain, embody, comprise, reflect, identify, state, refer to, deal with, comment on, respond to, describe, involve, mention, discus, record, support or are in any way pertinent to' the  topic at hand. That means that you cannot just plug in the text of the demand into a conceptual search and expect that you will retrieve everything required.

To make matters even more complicated, the secret sauce of conceptual engines is a lot like black magic to those who do not spend all of their time figuring out how to build a better search engine, meaning most of us. Trying to explain to the court how such an engine determined that all these documents were related and why the cluster was named that way can challenge the bravest of testifying experts and drown the veracity of your efforts in a sea of technical fog.

"So what good are concepts if I cannot search on them?", asks counsel. No one doubts the worth of conceptual analysis to find the key items produced by the opposing party, but how does that help the corporation find all their relevant email? The first step of incident response in the EDRM model is the Identification phase. The traditional approach is to interview the primary custodians associated with the matter and try to generate the preservation instructions. When dealing with enterprise archives like Autonomy's EAS, that means coming up with search criteria to place your legal holds. How do you know what criteria are related to the matter? Up to this point, counsel inside and out have been doing their best to define the criteria, but few have leverage a defensible, explainable process to support their criteria.

Maybe you have already collected the initial, key ESI that the custodians thought relevant to the matter. Or you have run a search using the truly unique terms, names and dates to perform your risk assessment. If you have a collection that you know is relevant, conceptual search engines like Autonomy's IDOL can show you the concepts within your collection to define or expand your actual hard Boolean search criteria. Now you can demonstrate the tools used to verify your due diligence and catch the variations and jargon used to talk about your topic.

Finding key terms and phrases buried inside of mountains of recorded phone conversations, voice mails and IM chats can devour discovery budgets and send counsel crying 'undue burden' to the court. There seem to be two dominant speech analytics methods: phonetic indexing (first brought to eDiscovery by Nexidia) and transcription or speech-to-text (long dominated by Autonomy's engine which supports both methods). Phonetic search renders sound wave forms into simplified strings of phonemes that can be indexed and searched. This makes the technology effectively content agnostic, but makes it challenging to integrate with text based search. Speech-to-text has been the foundation for automated conceptual search and improvements in speaker recognition technologies have also increased the value in what was effectively raw dialogue.

Knowing that one can search digitized conversations, the next question is can users effectively search everything within the enterprise system from unified federated search? There is little doubt that the archiving systems are aggressively pursuing acquisitions, partnerships and development to enable ingestion and indexing of every conceivable data stream. All of them started with email back in the late 1990's.   For example, Symantec doesn't have audio, but jumped ahead with early products to handle IM, file shares, Sharepoint through merger and acquisition.

Mergers inject complexity by requiring integration of technology, services and cultures. Having experienced the Symantec-Veritas integration first hand, it always surprises me to see pundits and bloggers jumping all over the expected personnel departures.  Moreover, other restructurings that occur in the wake of big M&A moves. Instead of looking at who departed, which products are End-of-Lifed and which partners jump ship, I think that we should look at who stays, fundamental IP integrations and new solution offerings to get a better idea of where the new joint entity is headed.

As a former customer of Zantaz's Introspect, I watched press releases closely after the Autonomy acquisition last July. I was surprised to see how quickly all of Zantaz's products were integrated to the IDOL search architecture. A two to three month integration tells me that the business units got the resources and investment needed to make necessary changes under the hood. The back channel continues to be positive.  For example, after speaking with two big Introspect customers at a conference last week, it was clear that Autonomy had greatly improved search speed and performance. The back end administration and database complexity still seems to be an issue with some customers, but systems that offer enterprise scale will require enterprise level investments in architecture, support and consistent project management to succeed. Perhaps that is at the root of the shift within the Zantaz channel to go up market and target larger sales?

If we are going to demand that large public corporations make all of their communication data streams accessible and manageable to respond to the new FRCP requirements, then we have to expect further consolidation in the maze of different applications used today. Discovery, ILM, Retention Management, Enterprise Content Management and all the other flavors of alphabet soup are just ways of saying that companies are responsible for administrating their information assets, including formerly transient forms such as audio, IM, PIN-2-PIN, etc.