What caused the sudden spike in API errors for Axonius's Vulnerability Management feature last week?

Introduction

The sudden spike in API errors for Axonius's Vulnerability Management feature last week is a critical issue that demands immediate attention and thorough analysis. To address this problem, I'll employ a systematic approach to identify, validate, and resolve the root cause while considering both short-term fixes and long-term implications for the product.

My analysis will follow a structured framework, beginning with clarifying questions to gather essential context, followed by a comprehensive examination of potential causes, data analysis, hypothesis formation, and ultimately, a proposed resolution plan.

Step 1

Clarifying Questions (3 minutes)

• I'm noticing the issue is specific to the Vulnerability Management feature. Could you confirm if this spike in API errors is isolated to this feature or if it's affecting other parts of the Axonius platform as well?

Why it matters: This helps determine if it's a feature-specific problem or a broader system issue. Expected answer: The issue is primarily affecting the Vulnerability Management feature. Impact on approach: If isolated, we'll focus on feature-specific changes; if widespread, we'll consider system-wide factors.

• Considering the timing, I'm wondering about recent deployments. Have there been any significant updates or changes to the Vulnerability Management feature or its underlying infrastructure in the days leading up to the error spike?

Why it matters: Recent changes often correlate with sudden performance issues. Expected answer: A minor update was deployed to the feature two days before the spike. Impact on approach: This would guide us to scrutinize recent code changes and deployment processes.

• Looking at user behavior, has there been any unusual increase in usage patterns or load on the Vulnerability Management API in the past week?

Why it matters: Unexpected load can trigger latent issues or exceed system capacity. Expected answer: Usage has been relatively stable with a slight increase in API calls. Impact on approach: If usage patterns are normal, we'd focus more on internal system issues rather than external factors.

• Regarding error types, are we seeing a specific error message or a variety of different API errors?

Why it matters: The nature of the errors can point to specific components or issues within the system. Expected answer: There's a mix of timeout errors and data parsing failures. Impact on approach: This would help us narrow down potential bottlenecks or data integrity issues.