← jasperhaynes.com
← Back to Blog

North Star Metrics: The AI Advisor That Keeps You Honest

• metrics, North Star, OKR, AI, product management, analytics

You shipped a major feature last quarter. Usage went up 15%. NPS improved by 3 points. Revenue grew 8%.

Your team celebrates. You put it in the quarterly review. Leadership is happy.

But here's the uncomfortable question: did your feature actually cause any of that?

Maybe usage went up because of a marketing campaign that ran the same month. Maybe NPS improved because you fixed a critical bug the week before. Maybe revenue grew because it was Q4 and enterprise deals always close in Q4.

Correlation isn't causation. And without proper analysis, you're building a narrative around coincidences.

The Metrics Problem in Product Management

Problem 1: Too Many Metrics, No Clear Priority

The average product team tracks 30-50 metrics. Monthly active users, daily active users, session duration, feature adoption rates, NPS, CSAT, CES, retention curves, conversion funnels, revenue per user...

When everything is a metric, nothing is a priority. Teams optimize for whatever metric their stakeholder cares about most, creating fragmented effort across conflicting objectives.

Problem 2: Feature Impact Is Assumed, Not Measured

Let's be honest about how most product teams measure feature impact:

  1. Ship feature
  2. Check if the graph went up
  3. If up → feature was successful
  4. If flat → "we need to give it more time"
  5. If down → blame external factors

This isn't measurement. It's storytelling.

Real impact measurement requires:

Most product teams do zero of these.

Problem 3: Forecasting Is Wishful Thinking

"If we build this feature, we'll increase retention by 5%."

Where does that 5% come from? Usually from a PM's experience, pattern matching from other products, or pure optimism. Rarely from rigorous analysis.

Without predictive models, feature forecasting is just goal-setting dressed in data clothing.

The Metrics & North Star Advisor

The Metrics & North Star Advisor, planned for the Jasper Toolkit, brings rigor to product metrics without requiring a data science team.

Finding Your North Star

Not every product has a clear North Star metric. The Advisor helps you identify one:

Step 1: Map the value chain

The system maps your product's value chain:

User signs up → Completes onboarding → Uses core feature → 
Gets value → Returns regularly → Expands usage → Refers others

Step 2: Identify the value moment

Which step in the chain most closely represents your product delivering value? That's your North Star metric.

For example:

Step 3: Validate the metric

The Advisor checks whether your candidate North Star metric correlates with:

If it doesn't correlate with these outcomes, it's a vanity metric, not a North Star.

Feature Impact Tracking

Every feature you ship gets automated impact analysis:

Pre-launch:

Post-launch (Week 1):

Post-launch (Month 1):

Ongoing:

Causal Analysis Engine

The system uses several techniques to determine true feature impact:

Difference-in-Differences: Compare the metric change for users who got the feature vs. users who didn't, accounting for underlying trends. This isolates the feature's contribution from background market changes.

Cohort Analysis: Track the same cohort of users over time, comparing those who adopted the feature early vs. those who adopted later or not at all.

Switchback Testing: For features that can be toggled, periodically enable/disable to measure the real-time impact on metrics.

The result: instead of "NPS went up after we shipped X," you get "Feature X caused a 2.1-point NPS increase (95% confidence interval: 1.3 - 2.9 points) for enterprise users. The effect was not statistically significant for SMB users."

That's actionable intelligence.

Predictive Forecasting

Based on historical data, the system forecasts future metric movements:

Current trajectory forecast: "At current velocity, your North Star metric will reach 85% of the Q2 target by end of quarter. Probability of meeting target: 62%."

Feature impact forecast: "If Feature Y launches on schedule and achieves expected adoption, it will contribute an estimated 8% improvement to the North Star metric, increasing target achievement probability to 78%."

Risk forecast: "Churn risk in Segment A is increasing. If not addressed, the North Star metric will decline 3-5% by end of Q2, reducing target achievement probability to 45%."

Metric Health Dashboard

The dashboard surfaces what matters at a glance:

Making Better Decisions with Data

Here's how this changes your weekly rhythm:

Instead of: "I think users will love this feature because of my experience." You say: "The data shows this feature will improve our North Star by 3-5% based on feedback volume, user request patterns, and impact of similar features."

Instead of: "We shipped it and the numbers look good." You say: "Causal analysis shows this feature drove a 2.1-point NPS increase for enterprise users. SMB impact was neutral. Recommendation: double down on enterprise use cases."

Instead of: "We're on track to hit our OKR target." You say: "Our probability of hitting the OKR target is 62%. We can increase it to 78% by accelerating Feature Y, or to 85% by both accelerating Y and addressing the churn risk in Segment A."

This isn't about replacing intuition with data. It's about validating intuition with evidence — and being honest when the evidence disagrees.

Getting Started Without AI

Even without AI tooling, you can bring more rigor to product metrics:

1. Choose One North Star Metric

Just one. Write it on the whiteboard. Make every feature decision reference it. If a feature doesn't clearly connect to the North Star, question whether it belongs on your roadmap.

2. Measure Before AND After

For every feature launch, record baseline metrics one week before. Then compare to one week, one month, and three months after. Simple before/after analysis catches most big impacts.

3. Find a Control Group

Not everyone uses every feature on day one. Compare early adopters to non-adopters. The difference is your feature's signal (imperfect, but far better than "the graph went up").

4. Track Your Prediction Accuracy

Before each feature launch, predict the metric impact. After launch, compare to actuals. After a year, you'll know whether your predictions are systematically optimistic, pessimistic, or random. That self-awareness alone is worth the exercise.

5. Separate Leading from Lagging

Track leading indicators (feature adoption, activation, engagement) alongside lagging indicators (revenue, churn, NPS). Leading indicators give you a 2-4 week head start on problems — enough time to course-correct.

The Bottom Line

Product management without causal metrics is driving a car by looking in the rearview mirror. You can see where you've been, but you can't see the curve ahead.

The Metrics & North Star Advisor puts a GPS in your dashboard. You still choose the route — but now you can see where you're actually heading.

The Metrics & North Star Advisor is in development as part of the Jasper Toolkit. Follow our blog for launch updates.