Why has the click-through rate on KnowBe4's simulated phishing emails decreased from 12% to 9% in the last 60 days?

Introduction

The recent decline in KnowBe4's simulated phishing email click-through rate from 12% to 9% over the past 60 days is a concerning trend that warrants thorough investigation. As we analyze this product issue, we'll follow a systematic framework to identify, validate, and address the root cause while considering both immediate and long-term implications.

Our approach will involve clarifying the context, ruling out external factors, understanding the product and user journey, breaking down the metric, gathering relevant data, forming hypotheses, conducting root cause analysis, and proposing validation methods and solutions. This structured process will ensure we comprehensively address the problem and develop effective strategies to reverse the trend.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might have been recent changes to the phishing simulation content. Have there been any updates to the email templates or scenarios used in the past 60 days?

Why it matters: Content changes could directly impact user engagement and click-through rates. Expected answer: Yes, there have been some updates to the templates. Impact on approach: If confirmed, we'd need to analyze the specific changes and their potential effects on user behavior.

• Considering user segments, I'm curious about the distribution of the decline. Has the decrease been uniform across all user groups, or are certain segments more affected?

Why it matters: Identifying specific affected groups could point to targeted issues or changes in user behavior. Expected answer: The decline varies across different user segments. Impact on approach: We'd need to focus on the most affected segments and analyze their characteristics.

• Given the nature of phishing simulations, I'm wondering about the frequency of campaigns. Has there been any change in how often these simulated emails are sent out?

Why it matters: Changes in frequency could lead to user fatigue or decreased engagement. Expected answer: The frequency has remained consistent. Impact on approach: If frequency has changed, we'd need to consider its impact on user engagement and learning retention.

• Thinking about the broader context, have there been any significant company-wide security initiatives or training programs implemented recently?

Why it matters: New security measures or training could influence how users interact with emails, including simulated phishing attempts. Expected answer: A new security awareness program was launched recently. Impact on approach: We'd need to evaluate how this program might be affecting user behavior and awareness levels.

• Considering the measurement system, I'm curious if there have been any changes to how click-through rates are tracked or calculated in the past 60 days?

Why it matters: Ensuring data consistency is crucial for accurate analysis. Expected answer: No changes to the measurement system. Impact on approach: If there were changes, we'd need to reassess the validity of the comparison between current and historical data.