Lesson test: How statistics mislead

Sign in

Libre University uses your GitHub account. Signing in is only needed to sit a final test, so the score is kept on your profile.

How statistics mislead

1.[3p]
Why can no multiple testing correction repair the garden of forking paths?

Only one test was actually run, so there is nothing to correct, yet the choice depended on the data

Correct

The corrections only apply to more than twenty tests

The answer is: Only one test was actually run, so there is nothing to correct, yet the choice depended on the data

The $p$ -value is computed from the wrong distribution

The answer is: Only one test was actually run, so there is nothing to correct, yet the choice depended on the data

The problem is confined to small samples

The answer is: Only one test was actually run, so there is nothing to correct, yet the choice depended on the data

2.[3p]
Testing at five interim points and stopping at the first $p < 0.05$ gives what true type I error rate, as a percentage?

CorrectNot quite: 14.2

3.[2p]
A researcher tests 20 hypotheses. What Bonferroni threshold keeps the overall error rate at 5 per cent?

CorrectNot quite: 0.0025

4.[3p]
A trial with no real effect examines 12 subgroups at the 5 per cent level. What is the chance of at least one significant subgroup, to two decimal places?

CorrectNot quite: 0.46

5.[2p]
A food raises the risk of a cancer with lifetime incidence 6 per 100 by 20 per cent. What is the new incidence per 100, to one decimal place?

CorrectNot quite: 7.2

6.[2p]
A 50 per cent relative risk reduction always corresponds to a large absolute benefit.

True

The answer is: False

False

Correct

7.[3p]
Match each practice to the repair for it.

Analysis chosen after seeing the data
Repeated interim looks
Only significant results published
Many hypotheses in one study

pre-registration
a group sequential design
control the false discovery rate
trial registration

Show the answer

Analysis chosen after seeing the data: pre-registration Repeated interim looks: a group sequential design Only significant results published: trial registration Many hypotheses in one study: control the false discovery rate

8.[3p]
Which chart devices exaggerate a difference?

Select all that apply

A vertical axis that does not start at zero on a bar chart

Correct

Encoding a quantity as the area of a circle

Correct

Two differently scaled vertical axes on one plot

Correct

Plotting the raw data points alongside the fitted line

The answer is: A vertical axis that does not start at zero on a bar chart, Encoding a quantity as the area of a circle, Two differently scaled vertical axes on one plot

9.[2p]
Of 97 psychology studies that originally reported significant effects, how many replications reached significance in the 2015 Open Science Collaboration project?

Correct

The answer is: 35

1.[3p] Why can no multiple testing correction repair the garden of forking paths?

2.[3p] Testing at five interim points and stopping at the first p<0.05 gives what true type I error rate, as a percentage?

3.[2p] A researcher tests 20 hypotheses. What Bonferroni threshold keeps the overall error rate at 5 per cent?

4.[3p] A trial with no real effect examines 12 subgroups at the 5 per cent level. What is the chance of at least one significant subgroup, to two decimal places?

5.[2p] A food raises the risk of a cancer with lifetime incidence 6 per 100 by 20 per cent. What is the new incidence per 100, to one decimal place?

6.[2p] A 50 per cent relative risk reduction always corresponds to a large absolute benefit.

7.[3p] Match each practice to the repair for it.

8.[3p] Which chart devices exaggerate a difference?

9.[2p] Of 97 psychology studies that originally reported significant effects, how many replications reached significance in the 2015 Open Science Collaboration project?

1.[3p]
Why can no multiple testing correction repair the garden of forking paths?

2.[3p]
Testing at five interim points and stopping at the first $p < 0.05$ gives what true type I error rate, as a percentage?

3.[2p]
A researcher tests 20 hypotheses. What Bonferroni threshold keeps the overall error rate at 5 per cent?

4.[3p]
A trial with no real effect examines 12 subgroups at the 5 per cent level. What is the chance of at least one significant subgroup, to two decimal places?

5.[2p]
A food raises the risk of a cancer with lifetime incidence 6 per 100 by 20 per cent. What is the new incidence per 100, to one decimal place?

6.[2p]
A 50 per cent relative risk reduction always corresponds to a large absolute benefit.

7.[3p]
Match each practice to the repair for it.

8.[3p]
Which chart devices exaggerate a difference?

9.[2p]
Of 97 psychology studies that originally reported significant effects, how many replications reached significance in the 2015 Open Science Collaboration project?