Advances in anytime-valid hypothesis testing and online multiple testing

In traditional statistics, all aspects of the study design, such as sample size, hypotheses to test, and tests to use, must be determined before data collection. In modern data science, however, decisions often have to be made in real time, and data-dependent changes to the study design are desirable. In this thesis, we develop new methods and refine existing ones that allow for such dynamic and flexible decision-making.

In particular, we consider the following three hypothesis testing settings: (1) Anytime-valid hypothesis testing, (2) Sequential Monte Carlo testing and (3) Online multiple testing. While these have been distinct branches of the literature, we draw connections between the settings and demonstrate how these insights enrich the existing methodology. This thesis consists of eight articles, of which one can be attributed to setting (1), two to setting (2) and five to setting (3).

Our contribution to anytime-valid hypothesis testing (1) is a general approach for improving sequential tests by avoiding that the used test martingale overshoots at the critical values. We illustrate its application with the well-known sequential probability ratio test.

With regard to (2), we show that sequential Monte Carlo testing can be interpreted as anytime-valid hypothesis testing of the specific null hypothesis that the data are exchangeable. We use this insight to construct a general framework for Monte Carlo testing that allows to stop sampling at any time and yields improvements over existing sequential Monte Carlo tests. In a follow up paper, we extend our methodology to multiple testing.

We present new results and procedures for various error rates and sub-settings in online multiple testing~(3). Of particular note is the development of an online closure principle, which extends the classical closure principle to the online case and allows all online procedures with familywise error rate control and true discovery guarantee to be constructed using intersection tests. This reveals a close connection to anytime-valid testing, which is used to construct new online multiple testing procedures.