← Back to Blog

The AI PM's First Week: A Validation-Focused Onboarding Checklist

• AI product management, onboarding, product strategy, validation framework, AI metrics, first week, product leadership, ML products, technical product management

The AI PM's First Week: A Validation-Focused Onboarding Checklist

I've watched dozens of AI product managers stumble in their first month—not because they lacked technical chops or product intuition, but because they optimized for the wrong thing. They tried to absorb everything: every Confluence page, every Slack thread, every architectural diagram. Meanwhile, the critical assumptions underlying their product remained unexamined until it was politically difficult to challenge them.

Your first week isn't about comprehension. It's about validation.

The distinction matters because AI products fail differently than traditional software. A conventional SaaS product might fail because of poor market fit or execution. An AI product can fail because someone made an optimistic assumption about model performance in month three, and by month nine, you've built an entire go-to-market strategy around a capability that doesn't reliably exist.

Here's the framework I use—and teach other AI PMs—to validate the assumptions that actually matter before organizational momentum makes them impossible to question.

Day 1-2: Validate the Value Hypothesis

The User Problem Isn't What's in the PRD

Start by ignoring the product requirements document. Seriously. The PRD represents what someone believed six months ago, filtered through committee revision and political compromise. Instead, talk to three users in your first 48 hours.

Not stakeholders. Not your engineering lead's interpretation of users. Actual users.

Ask them this specific question: "What would you be doing right now if our product didn't exist?" Their answer reveals whether you're a painkiller or a vitamin. For AI products, this distinction is existential because the computational costs of inference mean you can't afford to be a nice-to-have.

When I joined an AI-powered content moderation product, the positioning was "we help platforms scale trust and safety." But when I asked moderators what they'd do without us, they said: "We'd just hire more people." That's a cost-reduction play, not a capability unlock. It completely reframed our pricing strategy and roadmap prioritization.

Validate the Metrics That Matter

Every AI product I've audited has a dashboard. Most are measuring the wrong things.

Your first week validation: identify the single metric that, if it degraded by 20%, would cause users to churn. Not the metric in your OKRs. Not the metric your CEO mentions in board meetings. The metric users actually care about.

For a recommendation engine, it's rarely accuracy. It's often diversity of recommendations or time-to-value. For a code completion tool, it's not suggestions per minute—it's acceptance rate weighted by context complexity.

Document the gap between what you measure and what matters. This becomes your mandate for instrumentation improvements.

Day 3-4: Validate the Technical Feasibility

The Model Isn't Production-Ready (Even If It's in Production)

Schedule a working session with your ML engineer or research scientist. Not a presentation—a working session where they show you the actual model evaluation notebook.

Here's what you're validating:

1. Evaluation dataset representativeness: Ask to see the test set distribution compared to production traffic from the last 30 days. In every AI product I've shipped, there's been drift. Sometimes catastrophic drift. If your test set is six months old and your product domain is evolving, your performance metrics are fiction.

2. Edge case coverage: Ask: "Show me the worst-performing slice of our evaluation data." If they can't immediately pull this up, your evaluation infrastructure isn't production-grade. AI products don't fail uniformly—they fail on specific demographic groups, content types, or input patterns. Knowing where failure concentrates tells you where to invest in data collection and where to set user expectations.

3. Latency distribution, not averages: The p50 latency is irrelevant. Ask for p95 and p99. AI model inference has high variance, and users remember the slow experiences. If your p99 is 3x your p50, you have an architectural problem masquerading as a statistical outlier.

Validate the Data Flywheel

AI products should improve with usage. Most don't because the data flywheel is broken.

Map the actual flow: User interaction → Data capture → Labeling → Retraining → Deployment. Ask how long this cycle takes end-to-end. If the answer is "we haven't done it yet" or "about six months," you don't have a learning system—you have a static model with an expiration date.

When I audited one AI assistant product, I discovered they'd collected 2 million user interactions but had never retrained the model because the data pipeline required manual CSV exports and format conversion. They were sitting on a goldmine with no mining equipment.

Your validation task: identify the single biggest bottleneck in the data flywheel and put "fix this" on the roadmap for your first quarter.

Day 5: Validate the Business Model

Unit Economics Tell the Truth

AI products have a cost structure that scales differently than traditional software. Every inference costs money—compute, memory, sometimes API calls to foundation models.

Calculate your unit economics:

I've seen AI products with 60% gross margins at 1,000 users that would have negative margins at 100,000 users because inference costs scaled linearly while pricing didn't.

If you're using a foundation model API (OpenAI, Anthropic, etc.), model your exposure to pricing changes. When GPT-4 pricing dropped 50% in 2024, some products saw margin expansion. Others had contractual commitments that prevented them from capturing the benefit. Know which category you're in.

Validate Pricing Alignment

Your pricing model should align with how users perceive value. For AI products, this is tricky because the cost driver (inference volume) often doesn't match the value driver (quality of output).

Ask your sales team or customer success: "What feature request do we get most often?" If it's "we need more API calls" or "increase our rate limit," your pricing is constraining usage of a product people love—that's a good problem. If it's "we need better accuracy" or "reduce false positives," your pricing is disconnected from value delivery—that's an existential problem.

Day 6-7: Validate the Competitive Position

The Moat Isn't Where You Think

AI products are defensible through data, distribution, or specialized infrastructure—rarely through models alone. Foundation models are increasingly commoditized, and fine-tuning is accessible to anyone with a GPU and a weekend.

Your validation exercise: list your product's defensibility factors, then pressure-test each one.

If you think your moat is proprietary data: How much data would a competitor need to replicate your performance? Could they acquire it in 6 months through partnerships or synthetic generation? When I pressure-tested one "data moat," we realized a competitor could reach 90% of our performance with 10% of our data volume—not a moat, just a head start.

If you think your moat is model performance: Run your model against the latest GPT-4 or Claude on your evaluation set. If the foundation model scores within 10% of your specialized model, your moat is evaporating. This isn't hypothetical—I've seen three "proprietary AI" products get commoditized by foundation model improvements in a single quarter.

If you think your moat is distribution: Validate that your AI capability is the reason users stay, not just the wrapper around it. Talk to five customers and ask: "If we were acquired by [competitor], would you switch?" Their hesitation tells you everything.

Map the Competitive Landscape's Trajectory

Don't just analyze current competitors—model where they'll be in 12 months.

For each major competitor, identify:

When I mapped this for a document understanding product, I realized our primary competitor was losing money on every API call but had raised $50M. They could subsidize pricing for 18 months, which completely changed our go-to-market strategy from "better performance" to "enterprise security and compliance."

The Validation Synthesis: Your Week One Output

By day seven, you should produce a single document—not a comprehensive onboarding summary, but a validation report structured around assumptions and evidence.

Format it like this:

Critical Assumption #1: [The assumption as currently believed] Validation Status: ✓ Confirmed / ⚠ Partially validated / ✗ Contradicted Evidence: [What you learned] Implication: [What this means for roadmap/strategy]

For example:

Critical Assumption: Users value response speed over response quality Validation Status: ✗ Contradicted Evidence: User interviews revealed 4 of 5 users would wait 2-3x longer for higher quality. Current p95 latency of 1.2s is well within acceptable range. Implication: Deprioritize latency optimization sprint planned for Q2. Reallocate engineering to quality improvements (model fine-tuning, better prompt engineering).

This document becomes your mandate for change. It's politically defensible because it's evidence-based and time-bound—you're not contradicting the team's judgment, you're updating beliefs with new information.

What Not to Do in Week One

Avoid these common traps:

Don't propose solutions yet: Your job this week is diagnosis, not prescription. The moment you propose a solution, you trigger defensive reactions. Validate first, propose later.

Don't get lost in the technical weeds: You don't need to understand transformer architecture or backpropagation in week one. You need to understand whether the technical approach can deliver the promised user value.

Don't skip the uncomfortable conversations: The most important validations require asking questions that might expose problems. Ask them anyway. It's easier to surface issues in week one than month six.

Don't optimize for looking smart: Optimize for being useful. The smartest thing you can do is identify a critical assumption that's wrong before it's expensive to fix.

The Compounding Returns of Validation

Here's why this validation-focused approach matters: every week you operate on false assumptions compounds the error.

If you assume users value feature A over feature B, and you're wrong, you don't just waste the sprint building A—you waste the customer conversations positioning A, the marketing content explaining A, the sales training on A, and the opportunity cost of not building B.

In AI products, this compounding is accelerated because technical decisions create lock-in. Choose the wrong model architecture, and you're stuck with it until you can justify a rewrite. Build the wrong data pipeline, and you're collecting the wrong data for months.

Your first week is when you have maximum permission to question everything. Use it.

Your Week Two Mandate

If you've done week one right, you'll start week two with:

  1. A prioritized list of assumptions that need deeper validation
  2. A clear understanding of which metrics actually matter
  3. Relationships with the three people who can tell you the truth (usually: a candid engineer, a customer-facing team member, and a power user)
  4. A documented gap between current state and required state

You won't have all the answers. That's fine. You'll have the right questions, which is more valuable.

The AI PM who spends week one reading documentation emerges with knowledge. The AI PM who spends week one validating assumptions emerges with conviction.

Conviction is what you need to make the hard calls that define whether your AI product becomes essential or gets commoditized.

Start validating.