Classification versus search – which is more important?

Read his thoughts on the Classification versus Search in the 60 second interview below:

Should companies utilize automated classification systems - or can the human never be replaced?

Ultimately you need both; what has changed is that the technology which supports automated classification has progressed to the extent that it can now do so much more for you, while at the same time minimising user intervention. If you can provide suitable business rules that work every time (which presumes that the user complies with business ways of working to support those business rules), then the process works. Auto-classification is difficult to set up in organisations which have, for instance, large fileplans and many different correspondence types. The human factor can never be replaced: people tend not to follow the rules in the first place, and our knowledge allows us to apply intuition that technology still can't.

How do you roll out common classification schemes across the whole of an organisation?

The way we have done it is to engage with a small number users, put together a straw man, check that it works in theory, pilot it and then roll it out gradually, making minor adjustments learning from problems or suggestions. However, getting the classification scheme right and ensuring the technology supports it is only half the battle! The people change aspect is at least half of the problem: people need to be sold the concept and the benefits, trained, encouraged (told they have to use the new classification scheme) and then monitored. Also, unless it has very senior commitment to the effort required and the new ways of working, it will never be 100% successful.

What is the best form of classification? Functional, thematic, expiratory, organisational or hybrids of the former?

People are most comfortable conceptually with a hierarchical structure because it's what they know and understand. Most academics and theorists would have us all go down functional route. Almost all the fileplans I've worked on have been functional followed by capability, output based structures lower down; occasionally in huge organisations (50,000+ users), it's been a hybrid - an element of hierarchical at level one then functional below. However, there are several factors that need to be considered: size of organisation, how often the structure of an organisation changes, technical constraints, numbers of users, length of time records need to be kept. Definitely not a one size fits all!

Given that the majority of companies have no mechanism in place to remove regulatory-expired records - should this be a priority for classification?

Expiration or disposition of records needs to be considered when designing a fileplan; the logical way to handle this is to have retention schedules that are built into folder structures. The follow on - archiving procedures - need to be carefully thought through, whether it be to TNA or another organisation. These points will only work if there is suitable education of the user community alongside the exploitation of technology. Note that some modern software allows disposition linked not only to hierarchy but also to document types, which can be especially helpful to regulatory issues.

What differentiates search engines and enterprise search tools?

Let's just consider the difference first of all. A search engine used to search the internet would typically have many thousands of servers configured to provide quick search results; also hyperlinks (those pointing to your site and for the use of ranking) help provide the right response, as long as what you were trying to find was a fairly common query. On the other hand, an enterprise search tool is unlikely to use hyperlinks in the same way, will not use as many servers for indexing and will be configured specifically to search your enterprise's data stores.

In either case, speed of search, relevance of hits, number of hits, ability to refine the search are the most important factors; both are achieved differently, depending on which search technology is being used. However, what will change is our need to search other media, e.g. voice, video, IMs, CCTV footage, internet and mobile phone traffic.

What are the best solutions out there for Enterprise search? And given that at least 10% of an organisations salary bill is wasted in fruitless searching - what sort of investment should companies be looking at to optimize this?

It's hard to provide a generic answer to this sort of question. What is "best" for one client won't be "best" for all. For example, in a regulatory application, completeness may outweigh other considerations; in a medical application you might favour precision of recall; and in yet other cases you'd major on affordability. One observation: as our storage requirements increase, search is without doubt the key to finding information. It is therefore worth investing for the future. Bigger is usually better is usually more expensive!

Are Enterprise solutions services 'one-size-fits-all' or would you recommend different engines depending on the size and activities of an organisation?

This debate has raged for several years and will doubtless continue to engage us. As products become more complex, our needs become more convoluted, the possibilities become more endless. The Enterprise ideal is certainly in ascendancy in some areas (e.g. financial solutions, ERP systems, Content Management) but it's horses for courses - not everyone can afford to implement a large ERP suite, not everyone needs a multi-faceted Financials product. The golden rule is still look at what you need, define your business requirements THEN look to see what is available and will meet your requirements.

How would you determine and measure a successful deployment of classification and search in an organisation?

Firstly, measuring any sort of benefits requires you to measure the baseline situation before implementation and then monitor the position subsequently. Of the two, search is probably easier to measure: people are familiar with search technologies, know what is possible, have certain expectations. A simple questionnaire, pre- and post-implementation, will reveal how well the technology meets users' needs. The success of a classification scheme is harder to measure because for many it is a new concept and a new technology; evidence points to it being harder to find and store information initially, unless a large degree of tailoring has been done to automate the record declaration process, taking out user intervention. It's only after several months' usage that people get used to storing information in the right place and with all the right metadata.

Given that Managers spend 25% of their time looking for information: Which is more important to get right - Classification or Search?

Both are important, but of the two, search is the more crucial. If you store information in the wrong place and have a good search engine you can find it; if you misfile information and have a lousy search engine, you're stuffed!

Justin Waters will be speaking in the Technical Theatre at Documation UK on Thursday 18th October, from 12.15 - 13.30.

Documation UK, in partnership with AIIM, it is the UK’s only dedicated end-to-end content, business process & information management event. Olympia, London from 17-18 October 2007 www.documation-uk.com