ab-test-analysis
Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.
适合你，如果你经常做A/B测试并需要统计依据来决策。
npx oh-my-skill add phuryn/pm-skills/ab-test-analysiscurl -fsSL https://oh-my-skill.com/install.sh | bash -s -- phuryn/pm-skills/ab-test-analysisnpx oh-my-skill verify phuryn/pm-skills/ab-test-analysis怎么用
商店整理自技能原文 · 版本 18468a9 · 表述以原文为准装上后，Claude 会分析 A/B 测试数据，计算统计显著性、样本量、置信区间，并给出“发布/延长/停止/调查”的建议。
当你提供 A/B 测试结果数据（如 CSV、Excel 文件），或描述实验假设、指标、时长等信息时触发。
技能原文 SKILL.md
A/B Test Analysis
Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
Context
You are analyzing A/B test results for $ARGUMENTS.
If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
Instructions
- Understand the experiment:
- What was the hypothesis?
- What was changed (the variant)?
- What is the primary metric? Any guardrail metrics?
- How long did the test run?
- What is the traffic split?
- Validate the test setup:
- Sample size: Is the sample large enough for the expected effect size?
- Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
- Flag if the test is underpowered (<80% power)
- Duration: Did the test run for at least 1-2 full business cycles?
- Randomization: Any evidence of sample ratio mismatch (SRM)?
- Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
- Calculate statistical significance:
- Conversion rate for control and variant
- Relative lift: (variant - control) / control × 100
- p-value: Using a two-tailed z-test or chi-squared test
- Confidence interval: 95% CI for the difference
- Statistical significance: Is p < 0.05?
- Practical significance: Is the lift meaningful for the business?
If the user provides raw data, generate and run a Python script to calculate these.
- Check guardrail metrics:
- Did any guardrail metrics (revenue, engagement, page load time) degrade?
- A winning primary metric with degraded guardrails may not be a true win
- Interpret results:
| Outcome | Recommendation | |---|---| | Significant positive lift, no guardrail issues | Ship it — roll out to 100% | | Significant positive lift, guardrail concerns | Investigate — understand trade-offs before shipping | | Not significant, positive trend | Extend the test — need more data or larger effect | | Not significant, flat | Stop the test — no meaningful difference detected | | Significant negative lift | Don't ship — revert to control, analyze why |
- Provide the analysis summary: ``` ## A/B Test Results: [Test Name]
Hypothesis: [What we expected] Duration: [X days] | Sample: [N control / M variant]
| Metric | Control | Variant | Lift | p-value | Significant? | |---|---|---|---|---|---| | [Primary] | X% | Y% | +Z% | 0.0X | Yes/No | | [Guardrail] | ... | ... | ... | ... | ... |
Recommendation: [Ship / Extend / Stop / Investigate] Reasoning: [Why] Next steps: [What to do] ```
Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.