Why has RTB House's deep learning algorithm seen a sudden 20% decrease in conversion prediction accuracy for travel industry clients this quarter?

Introduction

RTB House's deep learning algorithm has experienced a sudden 20% decrease in conversion prediction accuracy for travel industry clients this quarter. This significant drop in performance requires a thorough investigation to identify the root cause and implement effective solutions. I'll approach this issue systematically, examining both internal and external factors that could contribute to this decline in accuracy.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a seasonal factor at play. Has this 20% decrease been compared to the same quarter last year?

Why it matters: Seasonal trends can significantly impact travel industry behavior. Expected answer: No, we haven't compared it to the same quarter last year. Impact on approach: If it's a seasonal trend, we'd need to adjust our algorithm to account for these fluctuations.

• Considering the specificity to travel clients, I'm wondering if there have been any major changes in the travel industry recently. Have there been any significant events or policy changes affecting travel patterns?

Why it matters: External factors could be disrupting typical travel behavior, affecting our prediction accuracy. Expected answer: There have been some changes in international travel restrictions. Impact on approach: We might need to incorporate real-time policy data into our algorithm.

• Given the sudden nature of the decrease, I'm curious about recent updates to the algorithm. Have there been any changes to the deep learning model or its training data in the past quarter?

Why it matters: Recent changes could have unintended consequences on model performance. Expected answer: Yes, we updated the model with new features last month. Impact on approach: We'd need to investigate the impact of these new features and potentially roll back if necessary.

• Thinking about data quality, I'm concerned about potential issues with our input sources. Have we seen any changes in the quality or quantity of data we're receiving from our travel industry partners?

Why it matters: Changes in input data can significantly affect model performance. Expected answer: We haven't noticed any significant changes, but we haven't specifically looked into it. Impact on approach: We'd need to audit our data sources and potentially reach out to partners for more information.