Why has Glean's search relevance score dropped by 15% over the past month?

Introduction

Glean's search relevance score dropping by 15% over the past month is a critical issue that demands immediate attention. This metric is likely a key performance indicator for Glean's core search functionality, directly impacting user satisfaction and potentially affecting retention rates. I'll approach this problem systematically, starting with clarifying questions to gather context, then moving through hypothesis generation, validation, and solution development.

Step 1

Clarifying Questions (3 minute)

• Looking at the timing, I'm thinking there might have been a recent product update. Has there been any significant change to the search algorithm or indexing process in the last 1-2 months?

Why it matters: Recent changes could directly impact search relevance. Expected answer: Yes, there was an update to improve search speed. Impact on approach: If confirmed, we'd focus on the trade-off between speed and relevance.

• Considering user segments, I'm curious about the distribution of this drop. Is the 15% decrease uniform across all user types, or are certain segments more affected?

Why it matters: Helps identify if the issue is global or segment-specific. Expected answer: Enterprise users are more affected than individual users. Impact on approach: We'd prioritize investigating enterprise-specific search patterns and content.

• Regarding the metric itself, has there been any change in how search relevance is calculated or measured in the past month?

Why it matters: Ensures we're comparing apples to apples in our analysis. Expected answer: No changes to the metric calculation. Impact on approach: If changed, we'd need to reassess the validity of the comparison.

• Thinking about external factors, have there been any significant changes in the type or volume of content being indexed by Glean recently?

Why it matters: Content changes could affect search relevance without system changes. Expected answer: There's been a 20% increase in non-textual content indexing. Impact on approach: We'd investigate how well the current algorithm handles diverse content types.