Why has the average query response time for Glean's enterprise search increased from 200ms to 500ms in the past two weeks?

Introduction

The recent increase in Glean's enterprise search query response time from 200ms to 500ms over the past two weeks is a critical issue that demands immediate attention. This performance degradation directly impacts user experience and could potentially affect customer satisfaction and retention. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Given the sudden increase, I'm wondering about recent changes. Have there been any significant updates to the search algorithm or infrastructure in the past month?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a recent update. Impact on approach: If yes, we'd focus on the update's impact; if no, we'd look at gradual degradation factors.

• Considering user patterns, has there been a notable increase in search volume or complexity of queries recently?

Why it matters: Increased load or query complexity could explain the slowdown. Expected answer: Search volume has increased by 20%. Impact on approach: If confirmed, we'd investigate scaling solutions; if not, we'd look at other factors.

• Thinking about data sources, have we added any new content repositories or significantly expanded existing ones lately?

Why it matters: New or expanded data sources could strain the system. Expected answer: A new large repository was added last month. Impact on approach: If true, we'd focus on indexing and data integration; if not, we'd explore other system changes.

• Regarding system health, have we observed any unusual patterns in server load or network latency?

Why it matters: Infrastructure issues could be causing the slowdown. Expected answer: Some servers are showing higher than normal CPU usage. Impact on approach: If confirmed, we'd prioritize infrastructure optimization; if not, we'd focus more on application-level issues.