What caused the sudden 30% increase in compilation time for large language models on Graphcore's Poplar SDK?

Introduction

The sudden 30% increase in compilation time for large language models on Graphcore's Poplar SDK presents a critical challenge that demands immediate attention. This issue directly impacts developer productivity and could potentially affect Graphcore's competitive position in the AI hardware market. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this might be related to a recent SDK update. Has there been a major release or update to the Poplar SDK in the past few weeks?

Why it matters: Recent changes often correlate with performance shifts. Expected answer: Yes, there was an update. Impact on approach: If yes, we'd focus on changes in that update; if no, we'd look at other factors.

• I'm curious about the consistency of this increase. Is the 30% slowdown uniform across all model sizes and types, or does it vary?

Why it matters: Helps narrow down if it's a general issue or specific to certain use cases. Expected answer: It varies depending on model size. Impact on approach: If consistent, we'd look at core SDK components; if varied, we'd investigate specific model characteristics.

• Considering user segments, are all customers experiencing this issue, or is it limited to specific types of users or workloads?

Why it matters: Identifies if it's a universal problem or tied to certain use cases. Expected answer: Primarily affecting users with very large models. Impact on approach: If limited, we'd focus on those specific use cases; if universal, we'd look at core SDK components.

• I'm wondering about the measurement methodology. Has there been any change in how compilation time is measured or reported recently?

Why it matters: Ensures the observed change is real and not a measurement artifact. Expected answer: No changes in measurement. Impact on approach: If changed, we'd scrutinize the new measurement process; if not, we'd focus on actual performance issues.