AI Product Management

AI Suitability Assessment Core Method

Use when: evaluating whether AI is the right approach for a product problem.

Before committing to an AI solution, score the problem on five dimensions:

• Pattern complexity — is the pattern too complex for rules but learnable from data? Simple if/then logic doesn't need AI.
• Data availability — do you have enough labeled data to train or fine-tune a model? (For LLM features, do you have enough examples to write effective prompts?).
• Error tolerance — can the user recover from wrong outputs? Autocomplete suggestions: high tolerance. Financial calculations: zero tolerance.
• Explanation need — do users need to understand why the AI made a decision? Regulated industries often require explainability that black-box models can't provide.
• Maintenance capacity — can your team monitor model performance, retrain when accuracy drifts, and handle edge cases over time?

If a problem scores low on data availability and error tolerance but high on explanation need, AI is probably the wrong approach. Use a rules engine or decision tree instead.

Evaluation Framework Design Core Method

Use when: defining what "good enough" means for an AI-powered feature.

Build an evaluation framework with three layers:

• Automated metrics — quantitative measures you can compute at scale (accuracy, precision, recall, F1 for classification; BLEU, ROUGE for generation; latency, cost per inference).
• Human evaluation — qualitative assessment by human raters on dimensions like relevance, helpfulness, safety, and tone. Design a rating rubric (1-5 scale with clear anchor descriptions) and evaluate a sample of 100-200 outputs regularly.
• User-facing metrics — how AI quality translates to product outcomes (click-through rate on recommendations, task completion rate with AI assistance, user satisfaction scores).

Set minimum quality thresholds for launch: "We launch when automated metrics exceed X AND human evaluation scores average Y AND user-facing metrics show no degradation from the non-AI baseline." Review these thresholds quarterly as user expectations and competitive benchmarks evolve.

Prompt Engineering as Product Spec Technique

Use when: building features powered by LLMs where the prompt IS the product logic.

For LLM-powered features, the prompt is the spec. It defines the system's behavior, constraints, tone, and output format. Treat prompts with the same rigor as traditional specs: version them, review them, test them against edge cases, and track their performance over time. Build a prompt evaluation suite: a set of 50-100 test inputs with expected outputs. Run every prompt change through the suite before deploying. Track prompt performance metrics: output quality score, latency, token cost, and failure rate (outputs that violate constraints or produce nonsensical results).

AI Failure Mode Analysis Core Method

Use when: designing the user experience around AI uncertainty.

Map every way the AI can fail:

• Wrong output — the AI gives an incorrect answer with high confidence.
• Hallucination — the AI generates plausible-sounding information that doesn't exist.
• Bias — the AI produces systematically unfair or skewed outputs for certain groups.
• Refusal — the AI declines to answer when it should engage.
• Latency — the AI takes too long, degrading the user experience.
• Cost spike — unexpected usage patterns cause inference costs to explode.

For each failure mode, design a mitigation: confidence scores displayed to users, citation requirements, bias testing in the evaluation suite, fallback to non-AI paths, streaming for latency, and cost guardrails (rate limits, cheaper model fallbacks).

Human-in-the-Loop Workflow Core Method

Use when: AI accuracy isn't high enough for fully autonomous operation.

HITL puts a human reviewer in the AI pipeline for cases where the model isn't confident or the stakes are high. Design the workflow:

• Confidence threshold — below what confidence score does the output get routed to a human?
• Review interface — what does the reviewer see? (AI output, confidence score, relevant context, approve/edit/reject actions.).
• Feedback loop — how do human corrections improve the model over time? (Corrections become training data for the next model version.).
• Escalation path — what happens when the human reviewer is also uncertain?

The PM decision is where to set the confidence threshold. Too low (route everything to humans) and you've built an expensive human workflow with AI decoration. Too high (route nothing to humans) and errors reach users unchecked. Start conservative (low threshold, more human review), measure error rates, and raise the threshold as model performance improves.

Model Performance Monitoring Technique

Use when: an AI feature is in production and you need to detect quality degradation.

AI models degrade over time as the real world drifts from training data. Monitor:

• Input drift — are users sending inputs the model wasn't trained on?
• Output quality — sample and human-evaluate a percentage of outputs weekly.
• User feedback signals — thumbs up/down, "regenerate" clicks, manual overrides of AI suggestions.
• Business metrics — are conversion rates, engagement, or satisfaction changing? Set alerts for statistically significant drops in any monitored metric. When alerts fire, investigate: is it a model issue, a data issue, or a user behavior change?

Responsible AI Checklist Tool

Use when: launching any AI-powered feature.

Before launching, verify:

• Bias testing — have you tested outputs across demographic groups for systematic unfairness?
• Privacy — does the model process or store PII? Is user data used for training? Have users consented?
• Transparency — do users know they're interacting with AI? Can they opt out?
• Safety — can the model produce harmful, illegal, or dangerous outputs? What guardrails prevent this?
• Accountability — who is responsible when the AI makes a consequential error? Is there a human appeal path?

Trust as Product Quality Framework

Use when: defining quality standards for AI-powered features — beyond accuracy to the full trust experience.

For AI products, trust is not a feeling — it's a measurable product quality built through four design properties. Transparency: Can users see what the AI is doing, what data it uses, and when it's uncertain? Transparency requirements scale with stakes — a music suggestion needs a one-line reason, a medical recommendation needs detailed justification. Control: Can users steer, correct, undo, or override AI behavior? Define control granularity per feature — some AI actions need a simple accept/reject, others need parameter adjustment. Consistency: Does the AI behave predictably in similar situations? Inconsistency erodes trust faster than occasional errors because users can't form a reliable mental model. Failure recovery: When the AI is wrong, how quickly can users recover? Design for clear error identification, easy correction paths, and visible system improvement. Set product requirements against all four qualities, not just accuracy.

AI Agent Product Considerations Technique

Use when: deciding whether to build for AI agent interaction, or when agents are becoming a significant interaction channel for your product.

AI agents increasingly act on behalf of users — booking travel, comparing products, completing workflows. PMs must decide when to optimize for agent interaction. When to build for agents: When your product has structured, repeatable tasks (booking, purchasing, data retrieval) that agents can complete reliably. When a significant portion of your users already delegate tasks to assistants. When your product's API or structured data could serve agent consumption. Agent task completion metrics: Measure what percentage of common tasks an agent can complete without human intervention, where handoffs fail, and how agent-driven usage compares to direct usage in conversion and satisfaction. Product implications: Agent-friendly products need clean APIs, structured data, predictable state transitions, and clear error responses. This often aligns with accessibility improvements — well-structured products serve both agents and assistive technologies better.