What caused the sudden increase in latency for Kong Ingress Controller deployments in the last 24 hours?

Introduction

The sudden increase in latency for Kong Ingress Controller deployments in the last 24 hours is a critical issue that demands immediate attention. As we dive into this analysis, we'll systematically identify potential causes, validate our hypotheses, and develop a comprehensive solution strategy. Our approach will balance short-term fixes with long-term improvements to ensure robust performance and prevent future occurrences.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any significant update or change to the Kong Ingress Controller or related systems in the past 48 hours?

Why it matters: Recent changes are often the culprit in sudden performance shifts. Expected answer: Yes, there was a minor update. Impact on approach: If yes, we'd focus on the changes made; if no, we'd look at external factors or gradual degradation.

• Considering the scope, I'm curious about the extent of the issue. Is this latency increase affecting all deployments uniformly, or are there variations across different clusters or regions?

Why it matters: Helps determine if it's a global issue or localized to specific environments. Expected answer: Varied impact across regions. Impact on approach: Uniform impact suggests a core issue, while variations point to environmental factors.

• Given the nature of ingress controllers, I'm wondering about traffic patterns. Have there been any unusual spikes in incoming traffic or changes in request patterns in the last 24 hours?

Why it matters: Unusual traffic can strain resources and cause latency. Expected answer: No significant traffic anomalies. Impact on approach: If yes, we'd investigate traffic management; if no, we'd focus on internal system issues.

• Thinking about system dependencies, has there been any change in the underlying Kubernetes clusters or related cloud infrastructure?

Why it matters: Ingress controllers are tightly coupled with the cluster environment. Expected answer: Minor cluster upgrades were performed. Impact on approach: Changes in the underlying infrastructure could directly impact the ingress controller's performance.