What caused the sudden spike in authentication failures for Ping Identity's PingID service last week?

Introduction

The sudden spike in authentication failures for Ping Identity's PingID service last week presents a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

Our analysis will follow a structured framework, beginning with clarifying questions to establish context, ruling out external factors, understanding the product and user journey, breaking down the relevant metrics, gathering and prioritizing data, forming hypotheses, conducting root cause analysis, and finally proposing validation methods and next steps.

Step 1

Clarifying Questions (3 minutes)

• I'm noticing the term "sudden spike." Would you say this increase occurred within a specific timeframe, such as 24-48 hours?

Why it matters: Understanding the timeframe helps narrow down potential causes and correlate with recent changes. Expected answer: Yes, the spike occurred within a 24-hour period. Impact on approach: A rapid increase suggests a specific trigger rather than a gradual trend.

• Given the nature of PingID, I'm assuming this affected multiple client organizations. Is that correct, or was it limited to a specific subset of customers?

Why it matters: This helps determine if the issue is systemic or isolated to certain user groups. Expected answer: It affected a significant portion of our client base. Impact on approach: A widespread issue points towards a core system problem rather than a client-specific configuration.

• I'm curious about the magnitude of this spike. Are we talking about a 2x increase in failures or something more dramatic like 10x?

Why it matters: The scale of the problem influences the urgency and type of response required. Expected answer: The failure rate increased by approximately 5x normal levels. Impact on approach: A significant increase like this would necessitate immediate action and likely points to a major system issue.

• Considering recent activities, were there any system updates, patches, or configuration changes deployed in the 48 hours preceding the spike?

Why it matters: Recent changes are often correlated with sudden performance issues. Expected answer: A minor security patch was applied 24 hours before the spike. Impact on approach: This would make the recent patch a primary suspect in our investigation.