How can we explain the sudden 35% decrease in completed online applications through NEOGOV's OHC onboarding module in the last two weeks?

Introduction

The sudden 35% decrease in completed online applications through NEOGOV's OHC onboarding module over the past two weeks is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term and long-term implications for our product and users.

I'll approach this problem by first clarifying the context, then ruling out external factors before diving deep into the product ecosystem, metric breakdown, and data analysis. From there, I'll form and validate hypotheses, conduct a root cause analysis, and propose a comprehensive resolution plan.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might have been a recent product update. Has there been any significant change to the OHC onboarding module in the last month?

Why it matters: Recent changes could directly impact user behavior. Expected answer: Yes, a UI refresh was implemented three weeks ago. Impact on approach: If confirmed, I'd focus on usability issues in the new interface.

• Given the specific 35% drop, I'm wondering about the normal fluctuation range. What's the typical week-to-week variance in completed applications?

Why it matters: Helps determine if this drop is truly anomalous. Expected answer: Usually within ±5%. Impact on approach: A 35% drop being far outside the norm would suggest a systemic issue rather than natural variation.

• Considering user segments, I'm curious if this affects all users equally. Have you noticed any differences in the drop-off rate among various user types or demographics?

Why it matters: Identifies if the issue is universal or specific to certain groups. Expected answer: The drop is more pronounced among first-time applicants. Impact on approach: I'd focus on the new user experience and onboarding process.

• Thinking about the application process, I'm wondering about completion rates at different stages. Has there been any change in where users are dropping off in the application funnel?

Why it matters: Pinpoints where in the process users are struggling. Expected answer: There's a significant increase in drop-offs at the document upload stage. Impact on approach: I'd investigate potential technical issues or usability problems specific to that step.

• Considering potential data anomalies, I'm curious about our tracking systems. Have there been any changes to how we measure or define a "completed application" in the last month?

Why it matters: Ensures the observed decrease isn't due to a measurement error. Expected answer: No changes to the measurement system or definition. Impact on approach: If confirmed, I'd focus on actual user behavior changes rather than data collection issues.