What caused the sudden spike in failed transactions for Viva Wallet's online payment gateway during the Black Friday sales period?

Introduction

The sudden spike in failed transactions for Viva Wallet's online payment gateway during the Black Friday sales period is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be a capacity issue. Can you provide details on the transaction volume during this period compared to normal operations?

Why it matters: Understanding the scale of the increase helps determine if our systems were overwhelmed. Expected answer: A significant increase, possibly 5-10x normal volume. Impact on approach: If confirmed, we'd focus on scaling solutions; if not, we'd look elsewhere.

• Given the nature of Black Friday, I suspect there might be changes in user behavior. Have we seen any unusual patterns in transaction types or amounts?

Why it matters: Atypical user behavior could strain specific parts of our system. Expected answer: Higher average transaction values and more frequent multi-item purchases. Impact on approach: If true, we'd investigate how our system handles complex transactions.

• Considering the critical nature of payments, I'm curious about recent system changes. Were any updates deployed to the payment gateway in the weeks leading up to Black Friday?

Why it matters: Recent changes could introduce unforeseen issues under high load. Expected answer: Possibly a minor update or security patch. Impact on approach: If confirmed, we'd scrutinize those changes for potential issues.

• With the increased traffic, I'm wondering about our fraud detection systems. Have we seen any changes in the rate of transactions flagged as potentially fraudulent?

Why it matters: Overzealous fraud detection could cause legitimate transactions to fail. Expected answer: A possible increase in flagged transactions due to unusual spending patterns. Impact on approach: If true, we'd look into optimizing our fraud detection algorithms for high-volume periods.