Why did Messenger message delivery success rate drop to 75%?

Introduction

The sudden drop in Messenger's message delivery success rate to 75% is a critical issue that demands immediate attention. This significant decline in performance could severely impact user experience, retention, and overall product health. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this might be a recent change. When exactly did we notice this drop in success rate?

Why it matters: Helps pinpoint potential causes related to recent changes or events. Expected answer: Within the last 24-48 hours. Impact on approach: If recent, focus on immediate changes; if gradual, consider longer-term factors.

• Considering user segments, I'm wondering if this affects all users equally. Are there any specific user groups or regions more impacted than others?

Why it matters: Identifies if the issue is global or localized, helping narrow down potential causes. Expected answer: Varies by region or user type. Impact on approach: If localized, investigate region-specific factors; if global, focus on core infrastructure.

• Given the nature of messaging, I'm curious about the types of messages affected. Is this impacting all message types (text, media, etc.) or specific formats?

Why it matters: Helps isolate whether the issue is content-specific or system-wide. Expected answer: Affects all message types. Impact on approach: If specific types, investigate content processing; if all, look at core delivery systems.

• Thinking about recent updates, have there been any significant changes to our infrastructure or code deployments in the past week?

Why it matters: Identifies potential internal causes related to recent changes. Expected answer: Yes, a major backend update was rolled out. Impact on approach: If yes, focus on rollback or hotfix; if no, investigate external factors or gradual degradation.