How to read a study: eight questions before you believe the headline

This page does not list studies and does not say what has been proven about any ingredient. It does something else: it gives you the eight questions used to interrogate a study, so you can do it yourself — including on pages that are selling something.

This one included. If you find a reference to research anywhere on this site, the same eight questions apply to it.

1. Who was studied

A result in 20 male athletes aged 25 does not transfer automatically to a woman of 55. If you are outside the studied group, the number is not about you — not necessarily wrong, but unknown.

The commonest case: the study is in people with a marked deficiency, and the conclusion is sold to people without one. A missing substance, given to someone it is missing in, does something; the same substance given to someone it is not missing in is a different question and usually a different result.

2. How many people, and for how long

Twelve people for two weeks is a pilot. It has a purpose — it decides whether a larger trial is worth running — but it is not grounds for a claim.

In a small group, chance does a lot of the work. So the same result in 12 people and in 1200 people is not the same result.

Duration matters the same way. If the substance is sold for daily use and was studied for four weeks, month six has no data.

3. What dose

This is the question with the most direct use to you, because it can be checked on the pack.

A study uses a specific amount. If the product contains a fifth of it, the study does not describe that product — it describes an amount the product does not deliver. This is the most frequent gap between an advertisement and its source, and it is pure arithmetic.

How to make the comparison:

• find the dose in the study — usually mg per day
• find the label amount for the same substance, per daily dose
• for an extract, account for standardisation: 500 mg of a 10:1 extract is not 500 mg of the plant, and 500 mg of an extract at 2% marker contains 10 mg of marker

Our own amounts, for every ingredient, are in the ingredient reference. We do not copy them here, so that there is one place that changes when a formula changes.

4. Compared with what

"A 30% improvement" compared with what: with placebo, with nothing, or with the starting point?

Against the starting point is the weakest comparison and the most used. People who start taking something usually change other things at the same time, and many measures move on their own.

The placebo group exists for exactly this reason.

5. In vitro, mice, or people

Three different levels, cited identically in advertising.

In a test tube, a substance is shown to do something to cells at a concentration never reached in human blood. In mice, it is shown to do something in an organism that is not human. In people, it is shown to do the thing you care about.

If the text does not say which of the three it is, that is itself information.

6. Relative versus absolute effect

One number, said two ways:

• "Halves the risk"
• "Risk falls from 2 in 1000 to 1 in 1000"

Both describe the same result. The first sounds like news; the second sounds like what it is.

When you see a percentage with no baseline, half the sentence is missing.

7. What "statistically significant" means

It means the result is hard to explain by chance alone. It does not mean the result is large, or that it matters to you.

With a large enough group, a very small difference becomes statistically significant. So "significant" is always followed by the question: how big is the difference in real units.

8. Who paid, and what was not published

Manufacturer funding does not make a study untrue, but it is information and is normally declared in the paper itself.

The bigger issue is what you cannot see: studies with negative results get published less often. That is why one positive study weighs less than a review that went looking for the negative ones too.

What this page does not say

It does not list studies and does not claim what has been proven about any ingredient. That is deliberate: citing someone else's paper does not make an unauthorised claim authorised, and under Regulation (EC) 1924/2006 the responsibility sits with whoever communicates the claim, not with whoever wrote it first.

Nor is it a statistics course. These eight questions are the ones that most often change how a message reads — not the complete list.

Food supplements do not treat or prevent disease and do not replace a varied diet.