How can we explain the unexpected 20% increase in false positive rates for Veracode's Dynamic Analysis scans in the last two weeks?

Introduction

The unexpected 20% increase in false positive rates for Veracode's Dynamic Analysis scans over the past two weeks is a critical issue that demands immediate attention. This surge in false positives could significantly impact our customers' trust and the overall effectiveness of our security scanning service. To address this problem, I'll employ a systematic approach to identify, validate, and resolve the root cause while considering both short-term fixes and long-term implications.

Step 1

Clarifying Questions (3 minutes)

• Given the sudden spike, I'm wondering about recent changes. Have there been any updates to the Dynamic Analysis engine or scanning algorithms in the past month?

Why it matters: Recent changes could directly correlate with the increase in false positives. Expected answer: Yes, there was a minor update to improve detection rates. Impact on approach: If confirmed, we'd focus on the update's impact and potential rollback.

• Considering the nature of false positives, I'm curious about the types of vulnerabilities being flagged. Has there been a change in the distribution of reported vulnerability types?

Why it matters: Certain vulnerability types may be more prone to false positives. Expected answer: There's been an increase in reported cross-site scripting (XSS) vulnerabilities. Impact on approach: We'd investigate the XSS detection logic and recent changes to it.

• Thinking about external factors, have there been any significant changes in the types of applications or technologies our customers are scanning?

Why it matters: New technologies or frameworks could trigger unexpected behavior in our scans. Expected answer: We've seen an uptick in modern single-page applications (SPAs) being scanned. Impact on approach: We'd examine how our engine handles SPAs and if recent changes affected this.

• Reflecting on our measurement process, has there been any change in how we're calculating or reporting false positive rates?

Why it matters: Ensures the observed increase is real and not a result of measurement changes. Expected answer: No changes to the calculation or reporting methods. Impact on approach: Confirms we're dealing with an actual increase, not a measurement artifact.