What metrics would you use to evaluate Edmunds's car comparison feature?

Introduction

Use the rate of initiated comparisons that lead to one preselected, qualified next step as the primary metric. Pair it with render reliability, decision usefulness, lead quality, and data accuracy. Treat purchase rate as secondary unless Edmunds can validate consented, sufficiently complete attribution.

Verified product context and interview assumptions

• Verified product context: Edmunds's official comparison page says shoppers can compare up to four vehicles across price, features, fuel economy, trim levels, and options. It also includes ratings, ownership costs, warranties, and other vehicle details. Source: Edmunds Car Comparison
• External data-quality reference: NHTSA's vPIC API provides manufacturer-submitted vehicle and specification data. It could be one validation source for suitable standardized fields, but this does not imply that Edmunds uses vPIC or that vPIC covers Edmunds's pricing and editorial data. Source: NHTSA Vehicle API
• Interview premise: Assume Edmunds can instrument on-site comparison events and downstream on-site actions with appropriate consent.
• Unknowns to clarify: Current strategic goal, baseline funnel, data sources, identity coverage, lead definitions, and the share of purchases that occur outside Edmunds.

Step 1

Clarifying Questions (3 minutes)

• **Outcome and scope:** Is the feature meant to narrow a shortlist, generate a qualified inventory or dealer action, or support research, and are we evaluating new vehicles, used vehicles, or both?

Why it matters: The primary event must match the intended outcome, while the baseline should be segmented by journey and device so a blended average does not hide a weak experience. Ask for: Product strategy, event definitions, and the current funnel by mobile, desktop, new, and used paths.

• **Attribution:** Which outcomes can Edmunds observe directly after a comparison, such as viewing inventory, saving a vehicle, or submitting a lead?

Why it matters: An off-site purchase should not be presented as directly observed unless a consented and validated data link exists. Ask for: Identity match rate, attribution window, consent rules, and missing-outcome rate.

• **Data quality:** How are vehicle fields sourced, refreshed, reconciled, and corrected when sources disagree?

Why it matters: A comparison can be fast and engaging while still failing its core purpose if the data is stale or wrong. Ask for: Field coverage, freshness, correction, and dispute baselines by make, model year, and field, plus abandonment by funnel stage.