February 2010 – Cover Story: Make Site Search More Personal

Subjectivity is the Key to Driving Site Search Success

By Dr. Scott Brave

If your site search is failing both you and your customers, you aren’t alone. Marketers, website managers and search professionals across all types of organizations struggle every day to produce sets of meaningful search results that truly help their user base–rather than inundating them.

Enterprise search has long been the de facto standard for websites and corporate knowledge databases, predominately employing text-matching algorithms. While statisticians might be impressed with the outcomes, users who initiate searches find themselves drowning in a sea of homogenous data.

Search professionals and merchandisers are in no better position, with product catalogs and content that is constantly growing and changing.

Site search has come a long way over the past few years, and some traditional vendors are even making impressive strides to improve relevance without having to rely as much on manual tuning. Even so, the single most important ingredient to produce relevant search results is still often lacking: subjectivity. In other words, typical search engines still have no reliable way of accurately interpreting users’ intent or determining what results are actually useful in your ever-growing sea of content.

“What exactly is so wrong with objectivity?” you might ask.

For one, the critical information needed to determine relevance is simply not in our content. The content that makes up our websites–whether HTML, flash, video, PDFs or some other type–seems like the best place to discover a match to a user’s search terms, but often it’s not. Frequently, there’s a mismatch in how a user phrases his or her query and the words content creators use–and that’s just for content that the engines are able to spider.

If we make an assumption that the content is in a format that’s spiderable and that it contains the user’s search terms, it still doesn’t mean that the documents that are returned in the results are truly relevant. The key is to understand what content is useful, when it’s useful and who it’s useful for. However, that critical information is in users’ heads, and it’s based on their experiences and needs. That’s why introducing subjectivity into search is so important.

The best approach for surfacing this subjective insight might seem to be to explicitly ask the users. This approach seems good on the surface, but upon further scrutiny it’s flawed from a standpoint of coverage, bias and inaccuracy. Some enterprise search providers are beginning to move in this direction, but have generally failed to recognize a fundamental rule: actions speak louder than words. As social science has taught us all along, if you really want to understand people, you need to watch what they do, not what they say.

A few players in the enterprise search space have even figured this out. That’s the good news. The bad news is that they neglect to look beyond search. What users are doing in search only skims the surface of what’s really going on. Anytime someone comes to your website, they are looking for something. Many times they will have done a search on Google before they came to your site. Their Google search terms are your first clue. Once they get to your site, not all users search. And even if they do, there are many steps and actions that they take after the search is performed. Every action, be it search, navigation, engaging with content–or even all of the above–is providing you with valuable clues about what’s useful, why it’s useful and who it’s useful to.

Consequences of Objective Search
Site search is one of the single most important features on websites, and a primary method for your users to find the products and content they need. The consensus among analysts is that while there have been great improvements in site search technology, the current options aren’t making the grade–on the user experience side or the search manager side. To really move the needle on the search experience, companies will have to look at new methods that fall outside of the traditional search technology realm. Independent analyst firm Gartner predicts that by year-end 2013, more than 30 percent of the most popular websites will use search technology to better target content by adapting to the user’s context.

Poor search experiences aren’t just an inconvenience for users, they are also costly. Marketing budget is spent attracting visitors to your site. Opportunity cost is sunk into the hours spent manually tuning search results and content. Customer service costs rise as users pick up the phone or send e-mails. Revenue is lost as prospects and customers abandon your site in frustration. This clearly isn’t lost on companies. A recent study by Internet Retailer reports that 35.9 percent of online retailers have plans to upgrade their site search.

Creating a Personalized Search Experience
Improving site search isn’t just about producing better search results. It’s about personalizing the search experience to each user based on their specific needs. It’s about making search subjective.

If continuing on with the status quo is no longer an option because of both the negative impact on customer experience and the high associated costs, then you need some concrete, objective points to consider when looking for a solution to the search problem:

Think not only about content, but also about context - Users come to sites with intent. That intent might be indicated in a Yahoo or Google search that brought them to your site. After reaching your site, they might then express intent or context in the pages they visit and engage with, the links they click and maybe the site searches they perform.

What else are they doing to express intent and even engagement on the site? Consider how long their mouse hovers over a piece of content. Are they engaging with multimedia content? Do they spend time comparison shopping, hopping back and forth from one document to the other? This expression of interest may span multiple searches and actions, and finding content that engages them and holds true value for the user’s specific intent or context may also take multiple steps.

By paying attention to the context of users’ visits, you can begin to fine-tune their search experiences and identify what web pages and content will be most relevant to them.

Make search social - In recent years, explicit user actions such as click-throughs, ratings and feedback were introduced to solve the issue of search relevancy. We’ve seen this on the likes of YouTube ratings, Yelp reviews and Amazon product commentary.

However, today’s advanced social search techniques have evolved to take into account how humans search for, find and consume information and products in the physical world. In this sense, the implicit feedback users are giving you as they navigate both the web and your site are much more important and telling than explicit feedback–on many levels. These implicit actions take into account all traffic, not just a small percentage. They also include information such as how a visitor arrives at a site and every action they take once they arrive, including navigation patterns, search behavior, browsing behavior and even interactions with non-spiderable content like video and PDFs.

Remember, actions speak louder than words or, in this case, search queries as well as navigation on the web and throughout your site. It’s amazing what a little listening will tell you about your visitors and what will interest them.

On a larger scale, you should also consider the macro data that can be collected and analyzed from all your website visitors. As this data is continuously distilled, virtual communities of like-minded visitors begin to emerge. Actions, patterns and tendencies associated with these communities form the basis of a collective prospective that you can use to deliver better search.

Automate the fine-tuning process - When companies do find flaws in their search results, the approach most take is to manually tune and tweak the search algorithms or the content. With this approach, companies can only focus on the most popular products and content, and they miss out on the highly profitable long tail. Simply put, it isn’t timely, scalable or all-inclusive.

Furthermore, search administrators are rarely subject matter experts and possess little knowledge about how visitors think about a company’s products and services, making it difficult to establish rules to actually help users. When subject matter experts are deployed to tune the content, it takes valuable time away from more high-value activities they could be focusing on, and still only addresses the most popular searches.

Companies need a solution that automates the fine-tuning of results in real-time, and allows the site to change the results as the behaviors of visitors themselves change, continuously optimizing the conversion potential of search.

Give online marketers “community plus control” - If we rethink how we approach fixing search and let our users or online community automatically surface the right content in the right context in real-time, we can really take our sites to the next level. Yet with that automation, companies still need the capacity for control.

Automation should do the heavy lifting for marketers, but it must have flexibility built into it. Online marketers and merchandisers will still need to create business rules to augment and sometimes even override what the community is doing.

Think outside of the box engine – Remember, search should be more than just a box. The engine is just one piece of the navigation framework that guides visitors to your products and content–no matter where the content lives or the format it takes. Perhaps it’s on the homepage, internal product pages, customer support areas, a microsite or elsewhere. It could be HTML, multimedia, a PDF or even some other format.

If companies begin to accept the limited nature of search engines and the concept of intent, they can do much more than fix search. They can start recommending products and content the second they see users expressing intent. When companies create these personalized experiences–be it personalized search results or personalized product and content recommendations across the site–they create a user experience that is engaging at every touch. And driving users to the best content every time they visit ultimately improves sales and reduces costs.

Dr. Scott Brave is co-founder and CTO of Baynote. He holds a Ph.D. in Human-Computer Interaction from Stanford University. Dr. Brave can be contacted at scott@baynote.com.