Honeycomb Natural Language Querying

Last year, I joined three of my colleagues to form a team with the goal of enabling AI-powered natural language querying (NLQ) in Honeycomb. Honeycomb is an observability platform for distributed services, and querying complex data is core to the product. Providing an interface to billions of data points presents an ongoing challenge: how to help our customers quickly and intuitively ask questions of their data as their systems become larger and more complex.

We had a hypothesis that LLM-powered natural language queries could provide useful results for common queries and help newer users get started with exploring their data. Instead of asking our customers to know or guess the structure of the data they needed to query, we wanted to make it possible to ask questions like “Can you show me latency for users with the highest cart totals, by user id?”

The team we formed was Honeycomb’s first machine learning team, and we had the ambitious goal of building, integrating, and launching our first iteration of natural language querying in one month. To make that timeframe work, design, engineering, and prompt engineering had to run in parallel.

My first design priority was to find a temporary place in the product to put a prompt interface behind a feature flag so that the team could quickly assess the accuracy of different prompts and models. My next priority was to create a set of low-fidelity prototypes to explore how to integrate the feature into the product, to design feedback loops, and to make sure our customers could get information in context about data privacy and handling before deciding to use the feature.

We launched Query Assistant as the first-of-its-kind fully executing natural language assistant for querying observability data in May of last year. Our built-in feedback loop showed that 20% of queriers were getting useful results—better results than we had aimed for. We were also able to see that new users who tried natural language querying became expert queriers faster than those who didn’t. Since then, we've continued to iterate on Query Assistant's prompt and capabilities based on what we've learned.

To read a case study about this project, please contact me.