What caused the sudden 30% decrease in bid requests processed by Xandr's Prebid Server yesterday?

Introduction

The sudden 30% decrease in bid requests processed by Xandr's Prebid Server yesterday is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term and long-term implications for our ad tech ecosystem.

I'll approach this problem by first clarifying key details, ruling out external factors, and then diving deep into our product mechanics, metrics, and potential internal causes. My goal is to pinpoint the root cause and develop a comprehensive plan to resolve the issue and prevent future occurrences.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any significant code push or configuration change in the last 48 hours?

Why it matters: Recent changes often correlate with sudden performance shifts. Expected answer: Yes, there was a deployment yesterday morning. Impact on approach: If confirmed, I'd focus on analyzing the changes in that deployment.

• Considering the scale of the decrease, I'm wondering about the geographic distribution. Is this 30% decrease uniform across all regions, or are some areas more affected than others?

Why it matters: Helps identify if it's a global issue or localized to specific infrastructure. Expected answer: The decrease is more pronounced in North America and Europe. Impact on approach: I'd prioritize investigating region-specific factors if the impact is uneven.

• Given the nature of Prebid Server, I'm curious about the demand-side impact. Have we seen any corresponding changes in fill rates or CPMs from major DSPs?

Why it matters: Helps determine if the issue is isolated to our system or part of a broader market trend. Expected answer: Fill rates have remained stable, but CPMs have decreased slightly. Impact on approach: If DSP metrics are stable, I'd focus more on internal technical issues.

• Considering potential data anomalies, I'm wondering about our monitoring systems. Have there been any alerts or unusual patterns in our error logs or latency metrics?

Why it matters: Helps rule out measurement errors and points to potential system failures. Expected answer: There was a spike in timeout errors starting yesterday morning. Impact on approach: I'd prioritize investigating our timeout handling and network performance.