Why are we seeing a 25% decrease in document writes per second for MongoDB's time-series collections over the past 48 hours?

Introduction

The recent 25% decrease in document writes per second for MongoDB's time-series collections over the past 48 hours is a critical issue that demands immediate attention. This performance degradation could significantly impact our users' ability to store and analyze time-series data efficiently. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this might be related to a recent deployment or configuration change. Has there been any significant update to our MongoDB infrastructure or related systems in the past week?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a minor update to the database configuration. Impact on approach: If confirmed, we'd focus on rollback options and change management processes.

• Considering the specificity of the 25% decrease, I'm curious about our monitoring granularity. Are we seeing this decrease consistently across all time periods, or is it more pronounced during specific hours?

Why it matters: Helps identify if the issue is time-dependent or constant. Expected answer: The decrease is more significant during peak usage hours. Impact on approach: Time-dependent issues might point to capacity problems or specific workload types.

• Given that this affects time-series collections specifically, I'm wondering about any changes in data ingestion patterns. Have we onboarded any new clients or seen changes in existing clients' data volumes recently?

Why it matters: Unusual data patterns could strain the system. Expected answer: No significant changes in client base or data volumes. Impact on approach: If true, we'd focus more on internal system issues rather than external factors.

• Considering potential environmental factors, have there been any changes in our cloud infrastructure or network configuration that might affect database performance?

Why it matters: Infrastructure changes can have cascading effects on database performance. Expected answer: No major infrastructure changes reported. Impact on approach: If confirmed, we'd shift focus to database-specific issues rather than broader infrastructure problems.