What caused the sudden spike in user authentication failures for AppDirect's single sign-on service yesterday afternoon?

Introduction

The sudden spike in user authentication failures for AppDirect's single sign-on (SSO) service yesterday afternoon is a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll systematically examine potential causes, gather relevant data, and develop a comprehensive plan to resolve the issue and prevent future occurrences.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any recent update to the SSO service or related systems?

Why it matters: Recent changes are often the culprit in sudden performance issues. Expected answer: Yes, there was a minor update to the authentication protocol. Impact on approach: If confirmed, we'd focus on the changes made in that update.

• Considering the scope, I'm curious about the extent of the issue. What percentage of authentication attempts were failing compared to the normal baseline?

Why it matters: This helps quantify the severity and potentially narrow down affected user segments. Expected answer: Failure rate increased from 0.1% to 5%. Impact on approach: A significant increase would suggest a systemic issue rather than an isolated incident.

• Given the nature of SSO, I'm wondering about the impact on different integrated applications. Were all applications affected equally, or did some see higher failure rates?

Why it matters: Uneven distribution could point to integration-specific problems. Expected answer: Certain high-traffic applications were more affected. Impact on approach: This would lead us to investigate those specific integrations more closely.

• Thinking about potential external factors, have there been any changes in network infrastructure or third-party services that our SSO relies on?

Why it matters: External dependencies can sometimes cause unexpected issues. Expected answer: No significant changes reported by our infrastructure team. Impact on approach: If confirmed, we'd shift focus to internal systems and recent code changes.