Statistical testing under distributional shifts
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Statistical testing under distributional shifts. / Thams, Nikolaj; Saengkyongam, Sorawit; Pfister, Niklas; Peters, Jonas.
In: Journal of the Royal Statistical Society, Series B (Statistical Methodology), Vol. 85, No. 3, 2023, p. 597-663.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Statistical testing under distributional shifts
AU - Thams, Nikolaj
AU - Saengkyongam, Sorawit
AU - Pfister, Niklas
AU - Peters, Jonas
PY - 2023
Y1 - 2023
N2 - We introduce statistical testing under distributional shifts. We are interested in the hypothesis for a target distribution , but observe data from a different distribution . We assume that is related to through a known shift τ and formally introduce hypothesis testing in this setting. We propose a general testing procedure that first resamples from the observed data to construct an auxiliary data set (similarly to sampling importance resampling) and then applies an existing test in the target domain. We prove that if the size of the resample is of order and the resampling weights are well behaved, this procedure inherits the pointwise asymptotic level and power from the target test. If the map τ is estimated from data, we maintain the above guarantees under mild conditions on the estimation. Our results extend to finite sample level, uniform asymptotic level, a different resampling scheme, and statistical inference different from testing. Testing under distributional shifts allows us to tackle a diverse set of problems. We argue that it may prove useful in contextual bandit problems and covariate shift, show how it reduces conditional to unconditional indep
AB - We introduce statistical testing under distributional shifts. We are interested in the hypothesis for a target distribution , but observe data from a different distribution . We assume that is related to through a known shift τ and formally introduce hypothesis testing in this setting. We propose a general testing procedure that first resamples from the observed data to construct an auxiliary data set (similarly to sampling importance resampling) and then applies an existing test in the target domain. We prove that if the size of the resample is of order and the resampling weights are well behaved, this procedure inherits the pointwise asymptotic level and power from the target test. If the map τ is estimated from data, we maintain the above guarantees under mild conditions on the estimation. Our results extend to finite sample level, uniform asymptotic level, a different resampling scheme, and statistical inference different from testing. Testing under distributional shifts allows us to tackle a diverse set of problems. We argue that it may prove useful in contextual bandit problems and covariate shift, show how it reduces conditional to unconditional indep
U2 - 10.1093/jrsssb/qkad018
DO - 10.1093/jrsssb/qkad018
M3 - Journal article
VL - 85
SP - 597
EP - 663
JO - Journal of the Royal Statistical Society, Series B (Statistical Methodology)
JF - Journal of the Royal Statistical Society, Series B (Statistical Methodology)
SN - 1369-7412
IS - 3
ER -
ID: 370486535