What caused the sudden drop in active users for JFrog Xray's vulnerability scanning feature this month?

Introduction

The sudden drop in active users for JFrog Xray's vulnerability scanning feature this month presents a critical challenge that demands immediate attention and a thorough analysis. As we delve into this issue, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

Our analysis will follow a structured framework, beginning with clarifying questions to establish context, ruling out external factors, understanding the product and user journey, breaking down the metric, gathering and prioritizing data, forming hypotheses, conducting root cause analysis, and finally proposing validation methods and next steps.

Step 1

Clarifying Questions (3 minutes)

• I'm noticing the focus on "active users." Would you say this refers specifically to users who have run at least one vulnerability scan in the past 30 days?

Why it matters: Defining "active users" is crucial for accurate measurement and analysis. Expected answer: Yes, that's correct. Impact on approach: If confirmed, we'll focus on factors affecting scan frequency and completion.

• Given the sudden nature of the drop, I'm wondering about recent changes. Have there been any significant updates to Xray or related JFrog products in the past month?

Why it matters: Recent changes often correlate with sudden metric shifts. Expected answer: Yes, we released a minor update to Xray two weeks ago. Impact on approach: If confirmed, we'll prioritize investigating the impact of this update.

• Considering user segments, has this drop been observed across all customer types, or is it more pronounced in a particular segment?

Why it matters: Segmentation can reveal targeted issues or user-specific problems. Expected answer: The drop is more significant among enterprise customers. Impact on approach: If true, we'll focus on enterprise-specific features or use cases.

• Regarding the time frame, I'm curious about the pattern. Was this a gradual decline over the month or a sharp drop at a specific point?

Why it matters: The pattern can indicate whether it's a sudden technical issue or a growing user dissatisfaction. Expected answer: It was a sharp drop about two weeks into the month. Impact on approach: A sharp drop would lead us to investigate specific events or changes at that time.

• To ensure we're working with accurate data, has there been any change in how we measure or report active users for Xray?

Why it matters: Metric definition changes can cause apparent drops without actual usage changes. Expected answer: No, our measurement methodology has remained consistent. Impact on approach: If confirmed, we can rule out measurement issues and focus on actual usage factors.