What factors are contributing to the unexpected 30% increase in power consumption for Tenstorrent's Wormhole accelerator cards during high-intensity machine learning tasks?

Introduction

The unexpected 30% increase in power consumption for Tenstorrent's Wormhole accelerator cards during high-intensity machine learning tasks is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term and long-term implications for the product and its users.

I'll approach this problem by first clarifying the context, then ruling out external factors before diving deep into the product's user journey, metric breakdown, and data analysis. From there, I'll form and validate hypotheses, conduct a root cause analysis, and propose a comprehensive resolution plan.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this might be related to recent software updates. Have there been any significant firmware or driver updates for the Wormhole accelerator cards in the past month?

Why it matters: Software changes can significantly impact power consumption. Expected answer: Yes, a major firmware update was released three weeks ago. Impact on approach: If confirmed, we'd prioritize investigating the firmware changes.

• Considering the specificity of "high-intensity machine learning tasks," I'm wondering about the workload characteristics. Can you provide more details on the types of ML tasks showing increased power consumption?

Why it matters: Different ML tasks stress hardware components differently. Expected answer: The issue is primarily observed in large language model training. Impact on approach: This would narrow our focus to optimizations for specific ML workloads.

• Given the magnitude of the increase, I'm curious about the measurement methodology. Has there been any change in how power consumption is measured or reported for these cards?

Why it matters: Ensures we're comparing apples to apples in our data analysis. Expected answer: No changes in measurement methodology. Impact on approach: If confirmed, we can rule out measurement errors and focus on actual consumption issues.

• Thinking about the hardware lifecycle, I'm wondering about the age of the affected cards. Is this issue consistent across all Wormhole cards or more prevalent in newer/older batches?

Why it matters: Could indicate a manufacturing or degradation issue. Expected answer: The issue affects all cards regardless of age. Impact on approach: This would steer us away from hardware batch-specific investigations.