Why has Quora's question recommendation algorithm seen a 15% drop in click-through rates over the past month?

Introduction

Quora's question recommendation algorithm experiencing a 15% drop in click-through rates over the past month is a significant issue that requires immediate attention. This decline directly impacts user engagement and content discovery, which are crucial for Quora's ecosystem. I'll approach this problem systematically, focusing on identifying potential root causes, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a seasonal component. Has this 15% drop been compared to the same period last year?

Why it matters: Seasonal trends could explain the fluctuation and impact our approach. Expected answer: No significant seasonal pattern observed. Impact on approach: If seasonal, we'd focus on adjusting for cyclical trends; if not, we'd investigate recent changes.

• Considering user segments, I'm curious if this drop is uniform across all user groups. Are we seeing any differences in click-through rates between new and established users?

Why it matters: Different user behaviors could point to specific issues in the algorithm or UI. Expected answer: The drop is more pronounced among established users. Impact on approach: If established users are more affected, we'd investigate changes that might have disrupted their familiar experience.

• Thinking about recent updates, have there been any significant changes to the recommendation algorithm or UI in the past 1-2 months?

Why it matters: Recent changes could directly correlate with the performance drop. Expected answer: A minor update to the algorithm was implemented 6 weeks ago. Impact on approach: If changes coincide with the drop, we'd focus on analyzing and potentially rolling back those specific updates.

• Considering data integrity, has there been any change in how click-through rates are measured or reported in the last month?

Why it matters: Ensures we're comparing apples to apples and not chasing a non-existent problem. Expected answer: No changes in measurement or reporting methods. Impact on approach: If measurement has changed, we'd need to recalibrate our analysis based on the new methodology.