What's causing the sudden 30% increase in latency for bid requests in Criteo's European data centers?

Introduction

The sudden 30% increase in latency for bid requests in Criteo's European data centers is a critical issue that demands immediate attention. This performance degradation could significantly impact Criteo's ability to serve timely and relevant ads, potentially leading to revenue loss and decreased client satisfaction. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• I'm noticing the issue is specific to European data centers. Could you confirm if this latency increase is isolated to Europe or if there are any signs of similar issues in other regions?

Why it matters: This helps determine if the problem is localized or potentially global. Expected answer: The issue is primarily observed in European data centers. Impact on approach: If isolated, we'll focus on Europe-specific factors; if global, we'll consider broader system-wide issues.

• Given the sudden nature of the increase, I'm wondering about recent changes. Have there been any significant updates to the bidding system, infrastructure, or data processing pipelines in the last week?

Why it matters: Recent changes often correlate with sudden performance issues. Expected answer: A system update was deployed last week. Impact on approach: If confirmed, we'll prioritize investigating the recent changes and their potential impact.

• Considering the magnitude of the increase, I'm curious about the traffic patterns. Has there been any unusual spike in bid request volume or changes in traffic sources recently?

Why it matters: Unusual traffic patterns could explain performance degradation. Expected answer: Traffic volumes have remained relatively stable. Impact on approach: If traffic is stable, we'll focus more on internal system issues rather than capacity problems.

• Looking at the timing, I'm thinking about potential external factors. Have there been any reported issues with major ISPs or cloud service providers in Europe recently?

Why it matters: External infrastructure problems could cause localized performance issues. Expected answer: No major external infrastructure issues have been reported. Impact on approach: If confirmed, we'll prioritize internal factors over external ones in our investigation.