What caused the sudden spike in origin shield errors for Edgio's application acceleration customers yesterday?

Introduction

The sudden spike in origin shield errors for Edgio's application acceleration customers yesterday is a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll follow a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term implications for our product and customers.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this might be related to a recent deployment. Have there been any significant changes or updates to our origin shield infrastructure in the past 48 hours?

Why it matters: Recent changes often correlate with sudden performance issues. Expected answer: Yes, there was a minor update to the load balancing algorithm. Impact on approach: If confirmed, we'd focus on the recent changes as a primary area of investigation.

• Considering the scope, I'm wondering about the extent of the impact. What percentage of our application acceleration customers were affected by this error spike?

Why it matters: Understanding the scale helps prioritize the response and narrow down potential causes. Expected answer: Approximately 30% of customers experienced increased errors. Impact on approach: A widespread issue suggests a systemic problem rather than isolated incidents.

• Given the nature of origin shield errors, I'm curious about any patterns in the error messages. Are there specific error codes or messages that are more prevalent in this spike?

Why it matters: Error patterns can point to specific components or issues within the system. Expected answer: There's been a significant increase in timeout errors. Impact on approach: This would guide our investigation towards network or capacity issues.

• Thinking about external factors, I'm considering potential DDoS attacks. Have we observed any unusual traffic patterns or spikes concurrent with the error increase?

Why it matters: External attacks can cause sudden performance degradation. Expected answer: No significant traffic anomalies detected. Impact on approach: This would help rule out security-related causes and focus on internal factors.