Sequential Testing with Information Criteria and Evidence Ratios

Ladislas Nalborczyk
Univ. Grenoble Alpes, CNRS, LPNC (France) • Ghent University (Belgium)

The power problem

How many participants/observations should I obtain for my next experiment, in order to show the effect I want to show ?

Usual algorithm (a priori NHST-PA): trying to guess the population effect size, plugging numbers (effect size and requested power) in the magic hat (the power function), and planning for a specific sample size.

plot of chunk unnamed-chunk-1

Sequential testing

But if I knew what (effect size) I am looking for, I would not bother making an experiment, right ?

Another approach consists in recruiting participants until we reach a predefined level of evidence. This is called sequential testing.

But this requires that we have a way to quantify evidence

Sequential Bayes Factor

Example of the Sequential Bayes Factor procedure as presented in Schönbrodt, Wagenmakers, Zehetleitner, & Perugini (2017) and Schönbrodt & Wagenmakers (2017).