How can Riskified balance the accuracy of its Policy Abuse detection with minimizing false positives that may frustrate legitimate customers?

Introduction

Balancing the accuracy of Riskified's Policy Abuse detection with minimizing false positives that may frustrate legitimate customers is a critical trade-off for the company's success. This scenario involves weighing the benefits of robust fraud prevention against the potential for customer dissatisfaction and lost revenue. I'll analyze this trade-off by examining the product, stakeholders, metrics, and potential solutions.

Step 1

Clarifying Questions (3 minutes)

• Context: I'm thinking about the current state of Riskified's Policy Abuse detection. Could you share some insights on its current accuracy rate and false positive rate?

Why it matters: Establishes a baseline for improvement Expected answer: Current accuracy around 95%, false positive rate around 5% Impact on approach: Higher accuracy might require more subtle improvements, while lower accuracy could justify more significant changes

• Business Context: Based on Riskified's business model, I assume reducing fraud is a top priority. How does this balance against customer satisfaction in our strategic goals?

Why it matters: Helps prioritize between fraud prevention and user experience Expected answer: Both are critical, with a slight edge towards fraud prevention Impact on approach: Would influence the acceptable trade-off between accuracy and false positives

• User Impact: I'm curious about the segments most affected by false positives. Do we see any patterns in terms of user demographics or transaction types?

Why it matters: Identifies potential areas for targeted improvements Expected answer: Higher-value transactions and new customers more likely to trigger false positives Impact on approach: Could lead to segment-specific solutions or risk thresholds

• Technical: Considering the complexity of fraud detection, what's our current capability to implement machine learning or AI-driven solutions?

Why it matters: Determines the feasibility of advanced technical solutions Expected answer: Some ML capabilities in place, room for expansion Impact on approach: Would influence the types of solutions we consider, from rule-based to advanced AI

• Timeline: Given the potential impact on revenue and customer satisfaction, how urgent is this issue? Are we looking at a quick fix or a longer-term strategy?

Why it matters: Helps frame the scope and depth of potential solutions Expected answer: Medium urgency, aiming for significant improvements within 6 months Impact on approach: Would determine whether to focus on quick wins or more comprehensive, long-term solutions