Why has the click-through rate on recommended streams on Twitch declined by 20% since the latest UI update?

Introduction

The recent 20% decline in click-through rate (CTR) on recommended streams on Twitch following the latest UI update is a concerning trend that requires immediate attention. This analysis will systematically investigate potential root causes, generate data-driven hypotheses, and propose a strategic plan to address the issue while considering both short-term fixes and long-term implications for Twitch's recommendation system.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking the UI update might be directly related. Can you confirm when exactly the UI update was rolled out and if it was a gradual or sudden release?

Why it matters: This helps establish a clear timeline and potential correlation. Expected answer: The update was rolled out two weeks ago over a 48-hour period. Impact on approach: A sudden drop aligning with the rollout would strengthen the UI change hypothesis.

• Considering user segments, I'm curious if the CTR decline is uniform across all user types. Have you noticed any differences in CTR changes between new and returning users, or between mobile and desktop users?

Why it matters: This could reveal if the issue is platform-specific or affects certain user groups more. Expected answer: The decline is more pronounced on desktop (25%) compared to mobile (15%). Impact on approach: This would focus our investigation more on the desktop UI changes.

• Regarding the recommendation algorithm, has there been any recent change in the backend systems that generate stream recommendations?

Why it matters: This helps distinguish between frontend UI issues and potential backend changes. Expected answer: No significant changes to the recommendation algorithm in the past month. Impact on approach: This would shift focus more towards the UI changes rather than backend issues.

• In terms of content, has there been any notable shift in the types of streams being recommended or any changes in popular streamers' activities?

Why it matters: This could reveal if the issue is content-related rather than UI-specific. Expected answer: No significant changes in content mix or streamer activities. Impact on approach: This would reinforce the focus on UI and user experience factors.