What factors are causing the increased latency in Amount's decisioning engine API responses observed in the last week?

Introduction

The increased latency in Amount's decisioning engine API responses over the past week is a critical issue that demands immediate attention. As we analyze this product problem, I'll employ a systematic framework to identify, validate, and address the root cause while considering both short-term fixes and long-term implications.

I'll approach this issue by first clarifying the context, then ruling out external factors before diving deep into the product ecosystem, metric analysis, and hypothesis generation. We'll then validate our findings and develop a comprehensive plan to resolve the latency issues.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to recent changes. Have there been any significant updates or deployments to the decisioning engine in the last two weeks?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a minor update. Impact on approach: If yes, we'd focus on change-related hypotheses; if no, we'd look at gradual degradation or external factors.

• Considering the nature of API responses, I'm curious about the scale. What's the current average latency compared to our baseline, and what percentage of requests are affected?

Why it matters: Helps quantify the issue and determine if it's systemic or isolated. Expected answer: 500ms increase on average, affecting 30% of requests. Impact on approach: Higher impact would prioritize immediate action; lower impact might allow for more thorough investigation.

• Given the critical nature of decisioning engines, I'm wondering about user impact. Have we seen any changes in user behavior or received customer complaints related to this latency?

Why it matters: Helps assess the urgency and business impact of the issue. Expected answer: Some high-volume customers have reported slower decisions. Impact on approach: Significant user impact would necessitate faster resolution and more communication.

• Thinking about potential data issues, has there been any change in the volume or nature of data being processed by the decisioning engine?

Why it matters: Data changes can significantly affect processing time. Expected answer: Data volume has increased by 15% in the last month. Impact on approach: If yes, we'd focus on data processing optimizations; if no, we'd look more at infrastructure or code-level issues.