Why has TiDB (Database Software)'s automatic failover mechanism failed to activate in 30% of recent high-availability cluster tests?

Introduction

TiDB's automatic failover mechanism failing to activate in 30% of recent high-availability cluster tests is a critical issue that demands immediate attention. This problem directly impacts the database software's reliability and could erode user trust if left unaddressed. I'll approach this analysis systematically, focusing on identifying potential root causes, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Given the specific 30% failure rate, I'm wondering about the test environment. Are these tests conducted in a controlled lab setting or in production-like environments?

Why it matters: The test environment could significantly impact the failure rate and help narrow down potential causes. Expected answer: Tests are conducted in production-like environments. Impact on approach: If in production-like environments, we'd need to consider real-world factors more heavily.

• Considering the nature of high-availability clusters, I'm curious about the cluster configuration. Has there been any recent change in the cluster setup or node count?

Why it matters: Changes in cluster configuration could directly affect the failover mechanism. Expected answer: No recent changes to cluster configuration. Impact on approach: If there were changes, we'd focus on those; if not, we'd look at other factors.

• Thinking about the failover process, I'm wondering about the specific failure scenarios being tested. Are these tests simulating various types of failures, or is it a specific failure mode?

Why it matters: Different failure modes could trigger different responses from the failover mechanism. Expected answer: Tests cover various failure scenarios. Impact on approach: If varied, we'd need to analyze patterns across scenarios; if specific, we'd focus on that particular failure mode.

• Noticing the 30% failure rate, I'm curious about the timeline. Has this issue emerged recently, or has it been observed over an extended period?

Why it matters: The timeline could indicate whether this is a new problem or a longstanding issue that's been overlooked. Expected answer: The issue has been observed in the past month. Impact on approach: A recent emergence would point us towards recent changes or environmental factors.