What caused the sudden spike in error rates for AppDynamics's Real User Monitoring feature last weekend?

Introduction

The sudden spike in error rates for AppDynamics's Real User Monitoring feature last weekend presents a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any significant update or change to the Real User Monitoring feature in the past week?

Why it matters: Recent changes often correlate with sudden performance issues. Expected answer: Yes, there was a minor update deployed last Thursday. Impact on approach: If confirmed, we'd focus on changes made in that update.

• Considering the nature of Real User Monitoring, I'm wondering about traffic patterns. Did we see any unusual spikes in user traffic or monitored applications during the weekend?

Why it matters: Abnormal traffic could strain the system and cause errors. Expected answer: Traffic was within normal ranges. Impact on approach: If traffic was normal, we'd look more closely at internal factors.

• Given that it's a "sudden" spike, I'm curious about the exact timeframe. Can you provide more details on when the error rates started increasing and if they've stabilized since?

Why it matters: The timing could indicate whether it's an ongoing issue or a one-time event. Expected answer: Error rates spiked Saturday morning and have remained elevated. Impact on approach: A sustained issue suggests a systemic problem rather than a temporary glitch.

• Thinking about the error rates, I'm interested in the magnitude. What's the normal error rate for this feature, and how much did it increase?

Why it matters: The scale of the increase helps prioritize the severity of the issue. Expected answer: Error rates typically hover around 0.1% but jumped to 5%. Impact on approach: A significant increase would warrant more urgent action and broader investigation.

• Considering potential external factors, I'm wondering if there have been any changes in our infrastructure or third-party services we rely on for this feature?

Why it matters: External dependencies can often be the source of unexpected issues. Expected answer: No known changes to infrastructure or third-party services. Impact on approach: If confirmed, we'd focus more on internal factors and our own codebase.