Capability template for analytics and reporting
Comparison claim check
A manager holding a claim such as "version B converts better" and two groups of numbers gets back the method used, the size of the difference, how uncertain it is, the assumptions checked, and plain wording for what the data does and does not support. The manager decides what to do.
Listed under
Add it to a workflow in the Fully Powered Planner, beside the teams and roles that will run it. Book a fit call
The outcome
Someone says the new page, the new offer or the new process did better, and shows two columns of numbers. Before anyone changes anything, the manager gets one page that answers four questions. How big is the difference? How sure can we be that it is more than luck? Were the groups fair to compare? And what can honestly be said about it in a meeting?
The page names the method used and why it fits, gives the difference in plain units and as a range, and checks the things that quietly break a comparison: how the groups were formed, whether the split matches the plan, whether results were watched and the test stopped when it looked good, and whether several comparisons were tried until one worked (if that number is not given, the result cannot be called strong). It then states one of four support levels and the wording the data allows.
The capability never says a change caused a result unless the groups were assigned at random, and it never turns a thin result into a confident one. The manager decides whether to act.
Who starts the work, and when
An analyst, a marketing or product manager, or an owner starts it when a claim about a difference is about to be believed or acted on: a test has finished, a pilot has run for a month, or someone has pasted two sets of results into a message. The person who owns the decision reads the result.
What they supply
- The claim, in the words it was made, and which group is said to be better on which outcome.
- The data: either a short summary per group (how many, and how many succeeded, or the average and spread) or one row per record with the group and the outcome.
- How the groups were formed: assigned at random by a system, chosen by people, or simply different periods or places.
- What counts as success, over what period, and whether each row is a person, a visit or an order.
- How many comparisons were looked at on this outcome, and whether anyone watched the results and stopped early.
- Optionally, the smallest difference that would matter to the business.
What comes out
One page in a fixed order: the support level, the claim, what was compared, the method, the result with its range, seven assumption checks each marked as passed, flagged or not checkable, what the data supports, what it does not support, questions for the manager, and a sentence that can go into a report. Where a section has nothing to say it says so.
An invented example
Illustrative example with invented data. An invented online shop, "Shop X", has tried a new checkout page. Everything below, including the company, the message and the numbers, is made up to show the shape of the work.
What the person supplies
The claim, from a message to the owner: "The new checkout page (version B) converts better than the old one (version A). Can we switch everyone over?"
The data, a summary with the default headers group, n, successes:
- A, 4020, 482
- B, 3980, 537
n is visitors who reached the checkout. successes is visitors who completed an order.
Notes from the person:
- Outcome: a visitor who reached the checkout placed an order in the same visit. Counted per browser, so one person on two devices can count twice.
- Groups were formed by the shop's testing tool, which assigned each new visitor to A or B at random on the first visit. The planned split was half and half.
- The test ran for the same 14 days for both versions, which had been set in advance. Nobody looked at results before the end.
- One comparison was planned: conversion, B against A.
- A sale ran in the second week for everyone.
- Smallest difference that would matter: not supplied.
What comes out
COMPARISON CLAIM CHECK: new checkout page (B) converts better than the old (A)
Support level: Supports the claim, within the limits below.
The claim: Version B has a higher share of checkout visitors who place an order than version A.
What was compared: Group A: 4,020 visitors, 482 orders. Group B: 3,980 visitors, 537 orders. Outcome: an order in the same visit, counted per browser. Period: the same 14 days for both (dates not supplied). Groups formed by: random assignment on first visit by the testing tool.
Method used: Difference between two rates, with a 95% interval and a z-test using the pooled rate. It fits because the outcome is yes or no and each group has at least 10 orders and 10 non-orders.
Result: Version A 11.99% against version B 13.49%. Difference (B minus A): 1.50 percentage points, a relative difference of 12.5%. 95% interval: 0.04 to 2.96 points. p-value 0.044, compared with 0.05 for one comparison. Effect size: Cohen's h 0.045, below the conventional "small" of 0.2. In business terms: about 15 more orders per 1,000 visitors (interval 0.4 to 29.6). Smallest meaningful difference: not supplied.
Assumption checks:
- Groups formed: Pass. Random assignment, so a gap can be read as an effect of the version.
- Split against plan: Pass. 4,020 and 3,980 against a planned half and half; chi-square 0.20, p 0.655.
- Size and shape: Pass. Orders 482 and 537, non-orders 3,538 and 3,443, all at least 10.
- Each unit counted once: Flag. Counting is per browser, so some people may be counted twice, and a person on two devices could be in both groups; how many is not known.
- Early looking and repeated comparisons: Pass. Fixed 14 days, nobody looked before the end, and one comparison was made.
- Same conditions: Pass. Both versions ran over the same days, including the week-two sale.
- Missing or extreme values: Cannot check. A summary cannot show missing outcomes and none were mentioned; there are no numeric values to check. This does not lower the support level.
What the data supports: In these 14 days, 13.49% of visitors on version B placed an order against 11.99% on version A, a gap of 1.50 points. Visitors were assigned at random, so the gap can be read as an effect of the page and not of who saw it. The margin is narrow: the lower end of the interval, 0.04 points, is under a tenth of the observed gap, so the true gain could be close to nothing or about 3 points.
What it does not support:
- That B will do as well in other weeks, on other traffic or without a sale running.
- Why B does better, or which change on the page is responsible.
- Any forecast of yearly orders or revenue; revenue per order was not compared.
- A gain for any group inside the test, such as mobile visitors; no breakdown was run.
- A precise size: the interval is wide compared with the gap.
Questions for you:
- Whether to switch everyone over is yours to decide; this check does not say. Is a gain that could be as small as 0.4 orders per 1,000 visitors enough to justify the switch, or do you want to run longer to narrow the range?
- Can the testing tool count by person as well as by browser, so the double-counting flag can be cleared?
Suggested wording for a report: "In a 14-day test with visitors assigned at random, the new checkout page had a higher order rate than the old one: 13.49% against 11.99%, a gap of 1.50 points (95% interval 0.04 to 2.96). The gain could be very small; the test does not show why it happened or that it will hold in other periods."
Every figure was calculated from the two rows supplied, using Python's standard library. Nothing has been changed or sent, and whether to act is yours to decide.
Where it fits
- Analytics and reporting, where a claim should be checked before it travels.
- Marketing and content, for page, subject-line and offer comparisons.
- E-commerce and retail, where checkout, pricing and promotion changes are tried on live visitors.
- Software and technology, for product experiments and release comparisons.
- Marketing and advertising, where one campaign variant is said to beat another.
- Growth & scaling, when many changes are being tried quickly and each needs an honest reading.
- Established businesses, when a pilot in one branch or team is the basis for a wider change.
It is not a review of why deals were won or lost (see win-loss review) or a grouping of what customers say (customer feedback themes). It compares two groups on one outcome.
What has to change for your company
Decide the usual confidence level and keep it for every claim. Write down the smallest difference that would justify the cost of a change in each area you test. Fix the unit of counting for your main outcomes: a person, a visit or an order. Decide who must read the page before a result is quoted outside the team. Name the tool the assistant is allowed to calculate with, so every figure can be rerun.
What has been checked
Fully Powered AI wrote this capability template. It was editorially reviewed on 10 November 2026. It was run twice on its invented example in Claude on 10 November 2026, both times by the session that wrote and revised it, so neither run was blind; the record of the second run is published beside it. It has not yet been tested in a client environment.
Testing in your environment means running it on comparisons whose answers you already know: a real difference, no difference, a split that does not match the plan, and a result that was peeked at. The analyst should recompute every figure before the page is relied on.
The full capability template
Everything above is the outline. The complete capability template is the working version: the roles, the inputs, the procedure, the exceptions and stop conditions, the output format, the reviewer checklist, the acceptance tests, the instructions for an AI assistant and what to customize for your company. It comes with a download and the link to its folder in Fully Powered’s public repository.
Free with an account. Sign in by email: no password and no card.
Next step
Put this capability to work for your team: on your plan, or built with you in a sprint.
Build this in a sprint
In a sprint, Fully Powered adapts this capability template to your rules, proves it on your own work and puts it in the hands of the roles that own it.
Two ways to begin: