What's causing the sudden increase in API response times for the Okta Identity Engine over the past 24 hours?

Introduction

The sudden increase in API response times for the Okta Identity Engine over the past 24 hours is a critical issue that demands immediate attention. As we delve into this product root cause analysis, we'll systematically investigate potential factors contributing to this performance degradation. Our approach will involve clarifying the situation, examining both internal and external factors, and developing data-driven hypotheses to identify the root cause.

Step 1

Clarifying Questions (3 minutes)

• I'm noticing the timeframe is quite specific. Has there been any recent deployment or system change within the last 48 hours?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, a minor update was pushed yesterday. Impact on approach: If confirmed, we'd prioritize investigating the recent deployment.

• Are we seeing this increase across all API endpoints or specific ones?

Why it matters: Helps narrow down the scope of the problem. Expected answer: It's affecting authentication-related endpoints primarily. Impact on approach: We'd focus on authentication services and related components.

• Have we observed any unusual traffic patterns or spikes in the last 24 hours?

Why it matters: Unexpected load can cause performance degradation. Expected answer: There's been a 20% increase in traffic from a specific region. Impact on approach: We'd investigate regional infrastructure and potential DDoS attacks.

• Are there any known issues with our database or caching layers?

Why it matters: Backend services often contribute to API slowdowns. Expected answer: No known issues, but we haven't thoroughly checked yet. Impact on approach: We'd prioritize database and cache performance analysis.

• Have we received any customer complaints or support tickets related to this issue?

Why it matters: User impact helps gauge the severity and nature of the problem. Expected answer: A few enterprise customers have reported slower logins. Impact on approach: We'd analyze these specific use cases and customer segments.