Using machine learning to improve eBay Help Search

How to access

  • Go here
  • Engage the search bar

My contribution

  • Designed the vision and roadmap for the product: Its use cases, behavior, core requirements
  • Defined success metrics and reporting
  • Set up a process to derive customer insights from large amounts of unstructured language data
  • Worked through all technical design aspects of the problem by collaborating with team members from architecture, engineering, data science
  • Worked through user interface design with eBay’s UX team

Like the work I’ve done on the HelpBot and natural language contact routing, this project was part of the broader vision to overhaul the help platform experience so that it is quicker and easier to find solutions. Each of the products I’ve delivered were intended to provide better help using machine learning at different stages of the help journey (discovery, self-service options, contact and routing, resolution). This product addressed key issues in the help discovery stage.

Users & The problem

User need: Accuracy, reassurance in results

Our interviews with new and occasional buyers and sellers revealed that their default action on the help platform was to search for a solution for their problem. They’ve been taught by Google that this is a fast way to get relevant information. However, on eBay’s help platform search, users exhibited a low click-through rate, and, often, the engine would return no results (typically for longer, more complex strings). When they didn’t get what they needed, these users would often contact eBay for issues that could have been self-served.

We also discovered that many of these users distrusted the result set because they weren’t sure whether the topics shown were related to their query. For example, if a user typed “item arrived damaged” even if the top result was “Returning an item” (which is the correct course of action for this problem) users didn’t click it because it was not obvious this was the solution.

Business need: Tuning efficiency, search improvement scalability, reduced contacts

eBay business analysts tuning the search engine also employed a very manual process of hand-mapping certain search queries to show certain arrangements of results. This was laborious, not easily scalable, and didn’t really move the needle on key metrics like search comprehension, click-through rate, and average click position (how far down on the page the result is that users select).

Solving these search problems

Machine learning (statistical modeling for query classification) has been applied to search for a long time because it helps with many of the problems mentioned above: It’s much more accurate over time and much less-labor intensive to teach an algorithm how to match results to search queries.

Since I was already working on the eBay HelpBot, which relies heavily on machine learning for natural language understanding, there were obvious cross-applications of our technology and training data that we could bring to bear on the search problem. We created a model to classify the user’s query into a result set that was much more accurate in predicting correct results and deployed it first in Australia, a smaller english-speaking test market, to measure its performance at scale.

To solve the problem of reassuring the user that the results shown for particular queries are accurate, we envisioned a system of rich snippets that would react to specific aspects of users’ language. For instance, if users typed that their item was damaged, we would show text inside of the search results indicating the correct course of action for damaged items, rather than merely showing a list of results with no context.

Impact and project status

Key metrics for this project:

  • Increase average click-through rate on search results
  • Reduce average click position
  • Increase search comprehension % of queries

I can’t go in to too much detail here as its an active project, but we’ve seen significant improvement in the above metrics and we’re scaling this search technology to other countries.