Corporate Search Engine Metrics

[adamjgb]

Hello. I wanted to get some feedback on what others are doing to measure their Enterprise Search Engine results. My company has placed a goal on the amount of successful searches made by our users, but the simple breakdown of what is and what isn't a successful search seems to have everyone with a different opinion. What is a good working definition of a successful search on an Intranet? For example, if I typed in Michael Jordan on my company's search engine and get no results, is that a successful search because I shouldn't have gotten any results?

I appreciate any and all responses to this thread.

[multintel]

What ever you do don't count HITS (it's an acronym...how idiots track success). I dividethe number of positive search results from false positives. False positives are those terms that get a result but get too many and cause the user to have to scroll and hunt for the right info. I also report just what you mentioned...irrelevent searches such as Michael Jordan. I did get a search for MJ last year. These are not tallied in with the failed search result numbers because users should not have been searching for them anyway.

[Dhall]

This is really hard to do from search logs because you can not guess as to the intent of the user. I have tried.

We have tried benchmarking using a set of standard searches/questions before and after search engine changes. By using a sample of users, we assess the improvments between the studies based on time on task and successful completion of a task across those same searches.

Of course, we only do this when we change something significant because pulling 20 users out of work for an hour plus the analysis time is costly if done without a good reason. A usability consultant can do the testing with you.

Good luck

[dorjemckinnon]

The intranet I look after was asked to provide the kind of metrics you're speaking about.
As the other posters have pointed out it is very hard to do appropriately, and very easy to do inappropriately e.g. HIT count

We've ended up giving the big dogs what they want Search lead to page, by tagging all the links presented in the search results with a URL parameter, that we can then quantify using the HTTP logs
We present the number of pages that were viewed as a result of a search.
Keeps the high ups happy - doesn't mean much really.

Failure
The very very useful thing that has come out of this is that we now check the search logs for situations where there have been ZERO results returned.
For each of these users we send them a link to the information they were looking for, if we can figure it out. Or we email them asking for more detail on what wanted or where they found what they were looking for.
In both cases we update the 'best bets' and synonyms links that show up in the search results - so that if the same search is done again there is something to help the searcher.

Success
This personal approach has been phenominally successful in improving staff confidence in the search tool, and have happy huggy feelings for the intranet.

Hope this helps
Dorje
For more intranet information those of us Down Under share information here via
http://onlinegroups.net/groups/kiwi_intranets