Jason Hull is a principal at OpenSource Connections, a Solr search engine consultancy that works with clients across all three layers of the search experience to improve the search user experience in a cost-effective and revenue-driving way. Jason leads client engagements with marketing functions, helping them to think about the business case behind search. Jason holds a BSc from the United States Military Academy at West Point and a MBA from the University of Virginia’s Darden Graduate School of Business. Prior to founding OpenSource Connections, Jason led an analytics team at Capital One responsible for call center demand forecasting and managed their internal call center IT infrastructure investment budget. Jason is a DZone MVB and is not an employee of DZone and has posted 14 posts at DZone. You can read more from them at their website. View Full User Profile

Analyzing Search Results for Fun and Profit

04.23.2012
| 4760 views |
  • submit to reddit

In a previous post, we looked at the first two components of the success of search: the density of highlighted search terms and the placement within the search engine results page. However, there are times when the science of a search engine does not match the art of marketing.

It is the role of a web manager and of marketing in general to understand the voice of the customer. They place themselves in the role of a customer and try to understand what a customer is trying to accomplish when performing a task. Additionally, the marketing and web management roles should have an understanding of the profitability of certain products versus other products and how generating sales in one versus the other will affect the bottom line of the company.

Oftentimes, a marketer will have a list of what results he or she will want to see given a search. It is possible to evaluate how the SERP compares to the marketer’s ideal list using a similar approach to the Results Utility Positioning calculation that we discussed previously. Conceptually, this is similar to the judgment list we discussed when covering how to index Chinese in Solr.

The Results Position Analysis (RPA) score is a factor of two attributes related to the SERP and the marketer’s judgment list. The first one has to do with actual matching. Does a result in the judgment list actually appear somewhere on the SERP? If so, this is positive. The second attribute has to do with the distance from the result on the SERP compared to the location in the judgment list. A result in the SERP which is tenth on the page doesn’t have much value when the judgment list has it first, and, similarly, a result that is first on the SERP doesn’t have much value if the judgment list has it showing up tenth.

We normalize results to a 0 – 100 score and add it to the RUP score to come up with a cumulative score. The RPA and the RUP should, to some extent, inform each other. Perhaps the marketer has an unrealistic ideal of what should be coming up. If someone searches for red shoes, blue elephants will never sell. On the other hand, if the marketer’s judgment list is fairly reasonable, a low RUP score should indicate failures of content or in the search algorithm to cause the quality of the SERP to match the judgment list.

Combined, RUP and RPA can be a very powerful measurement tool to improve the profitability of search within an e-commerce website. Site search is often ignored, particularly because there aren’t many known metrics to evaluate both the qualitative (RPA) and quantitative (RPA) qualities of the search engine results page.

If you need metrics on the viability of your search engine results pages, please contact me or e-mail us at talktous at opensourceconnections dot com.

Published at DZone with permission of Jason Hull, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)