Why has user engagement with Databricks Notebooks decreased significantly since the latest UI update?

Introduction

The recent decline in user engagement with Databricks Notebooks following the latest UI update is a critical issue that demands immediate attention. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause, considering both short-term fixes and long-term strategic implications.

Our analysis will follow a structured framework, beginning with clarifying questions to establish context, ruling out external factors, understanding the product and user journey, breaking down the metric, gathering and prioritizing data, forming hypotheses, conducting root cause analysis, and finally, proposing validation methods and next steps.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking the UI update might be directly related to the engagement drop. Could you provide more details on when exactly the UI update was rolled out and when we started noticing the engagement decrease?

Why it matters: This helps establish a clear timeline and potential correlation. Expected answer: The UI update was rolled out X weeks ago, and engagement started dropping Y days later. Impact on approach: A close temporal relationship would strengthen the hypothesis that the UI change is the primary factor.

• I'm curious about the specific engagement metrics we're tracking. Are we looking at daily active users, time spent in notebooks, or some other measure of engagement?

Why it matters: Different engagement metrics might point to different root causes. Expected answer: We're primarily tracking daily active users and average session duration. Impact on approach: This would help focus our analysis on specific user behaviors affected by the change.

• Considering user segments, I'm wondering if the engagement drop is uniform across all user types or if it's more pronounced in certain groups. Do we have data on how different user segments (e.g., data scientists vs. engineers) have been affected?

Why it matters: This could reveal whether the issue is universal or specific to certain use cases. Expected answer: We've noticed a more significant drop among data scientists compared to engineers. Impact on approach: This would guide us to investigate features or workflows more commonly used by the affected segment.

• Given that this is a UI update, I'm thinking about potential changes in the user experience. Have we received any specific feedback or support tickets related to the new UI?

Why it matters: Direct user feedback can provide qualitative insights into the quantitative engagement drop. Expected answer: Yes, we've seen an increase in support tickets related to difficulty finding certain features. Impact on approach: This would help us focus on specific UI elements or workflows that might be causing friction.