Why has SambaNova Systems's DataScale SN30 system seen a 15% decrease in inference speed for large language models over the past month?

Introduction

SambaNova Systems's DataScale SN30 system has experienced a 15% decrease in inference speed for large language models over the past month. This performance degradation is concerning and requires immediate attention. I'll approach this issue systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might have been a recent software update. Has there been any system or software changes in the past month?

Why it matters: Software updates can significantly impact performance. Expected answer: Yes, there was a recent update. Impact on approach: If confirmed, we'd focus on the update's contents and rollback options.

• Considering the specificity of the decrease, I'm wondering about our measurement accuracy. Has there been any change in how we measure or define inference speed?

Why it matters: Ensures we're addressing a real issue, not a measurement anomaly. Expected answer: No changes in measurement methodology. Impact on approach: If changed, we'd need to reassess our baseline metrics.

• Given the focus on large language models, I'm curious about workload changes. Have there been any significant changes in the types or sizes of models being run?

Why it matters: Different models can stress the system in various ways. Expected answer: No major changes in model types or sizes. Impact on approach: If changed, we'd investigate the new models' specific requirements.

• Thinking about hardware, I'm considering potential physical issues. Have there been any environmental changes or hardware alerts in the data centers?

Why it matters: Physical factors can impact performance. Expected answer: No significant environmental or hardware changes reported. Impact on approach: If issues found, we'd prioritize hardware diagnostics and repairs.