Why has Kong's API Gateway seen a 20% drop in request throughput over the past week?

Introduction

Kong's API Gateway experiencing a 20% drop in request throughput over the past week is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term fixes and long-term implications for the product.

I'll approach this problem by first clarifying key details, ruling out external factors, and then diving deep into the product's functionality and metrics. From there, I'll generate and validate hypotheses, conduct root cause analysis, and propose a comprehensive resolution plan.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any significant update or change to the API Gateway in the last 1-2 weeks?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a minor update. Impact on approach: If yes, I'd focus on that update as a primary suspect.

• Considering the scale of the drop, I'm wondering about the distribution. Is this 20% drop uniform across all clients or concentrated in specific segments?

Why it matters: Helps narrow down if it's a global issue or specific to certain users or use cases. Expected answer: The drop is more pronounced for high-volume clients. Impact on approach: If concentrated, I'd investigate those specific segments more closely.

• Given the nature of API Gateways, I'm curious about the error rates. Have you noticed any increase in error rates or latency alongside this throughput drop?

Why it matters: Errors or latency often accompany throughput issues and can point to specific problems. Expected answer: There's been a slight increase in latency but no significant change in error rates. Impact on approach: This would guide me to look more closely at performance bottlenecks rather than outright failures.

• Thinking about external factors, has there been any change in the pattern of API calls from major clients that could explain this drop?

Why it matters: Client behavior changes can sometimes masquerade as product issues. Expected answer: No significant changes reported from major clients. Impact on approach: If no changes, I'd focus more on internal factors rather than client-side issues.