Invariant Models for Causal Transfer Learning

Research output: Contribution to journalJournal articlepeer-review

Standard

Invariant Models for Causal Transfer Learning. / Rojas-Carulla, Mateo; Schoelkopf, Bernhard; Turner, Richard; Peters, Jonas.

In: Journal of Machine Learning Research, Vol. 19, No. 1, 2018, p. 1-34.

Research output: Contribution to journalJournal articlepeer-review

Harvard

Rojas-Carulla, M, Schoelkopf, B, Turner, R & Peters, J 2018, 'Invariant Models for Causal Transfer Learning', Journal of Machine Learning Research, vol. 19, no. 1, pp. 1-34.

APA

Rojas-Carulla, M., Schoelkopf, B., Turner, R., & Peters, J. (2018). Invariant Models for Causal Transfer Learning. Journal of Machine Learning Research, 19(1), 1-34.

Vancouver

Rojas-Carulla M, Schoelkopf B, Turner R, Peters J. Invariant Models for Causal Transfer Learning. Journal of Machine Learning Research. 2018;19(1):1-34.

Author

Rojas-Carulla, Mateo ; Schoelkopf, Bernhard ; Turner, Richard ; Peters, Jonas. / Invariant Models for Causal Transfer Learning. In: Journal of Machine Learning Research. 2018 ; Vol. 19, No. 1. pp. 1-34.

Bibtex

@article{511606930ba34b6788d1f97ddcd71cbd,
title = "Invariant Models for Causal Transfer Learning",
abstract = "Methods of transfer learning try to combine knowledge from several related tasks (or domains) to improve performance on a test task. Inspired by causal methodology, we relax the usual covariate shift assumption and assume that it holds true for a subset of predictor variables: the conditional distribution of the target variable given this subset of predictors is invariant over all tasks. We show how this assumption can be motivated from ideas in the field of causality. We focus on the problem of Domain Generalization, in which no examples from the test task are observed. We prove that in an adversarial setting using this subset for prediction is optimal in Domain Generalization; we further provide examples, in which the tasks are sufficiently diverse and the estimator therefore outperforms pooling the data, even on average. If examples from the test task are available, we also provide a method to transfer knowledge from the training tasks and exploit all available features for prediction. However, we provide no guarantees for this method. We introduce a practical method which allows for automatic inference of the above subset and provide corresponding code. We present results on synthetic data sets and a gene deletion data set.",
keywords = "Transfer learning, Multi-task learning, Causality, Domain adaptation, Domain generalization",
author = "Mateo Rojas-Carulla and Bernhard Schoelkopf and Richard Turner and Jonas Peters",
year = "2018",
language = "English",
volume = "19",
pages = "1--34",
journal = "Journal of Machine Learning Research",
issn = "1533-7928",
publisher = "MIT Press",
number = "1",

}

RIS

TY - JOUR

T1 - Invariant Models for Causal Transfer Learning

AU - Rojas-Carulla, Mateo

AU - Schoelkopf, Bernhard

AU - Turner, Richard

AU - Peters, Jonas

PY - 2018

Y1 - 2018

N2 - Methods of transfer learning try to combine knowledge from several related tasks (or domains) to improve performance on a test task. Inspired by causal methodology, we relax the usual covariate shift assumption and assume that it holds true for a subset of predictor variables: the conditional distribution of the target variable given this subset of predictors is invariant over all tasks. We show how this assumption can be motivated from ideas in the field of causality. We focus on the problem of Domain Generalization, in which no examples from the test task are observed. We prove that in an adversarial setting using this subset for prediction is optimal in Domain Generalization; we further provide examples, in which the tasks are sufficiently diverse and the estimator therefore outperforms pooling the data, even on average. If examples from the test task are available, we also provide a method to transfer knowledge from the training tasks and exploit all available features for prediction. However, we provide no guarantees for this method. We introduce a practical method which allows for automatic inference of the above subset and provide corresponding code. We present results on synthetic data sets and a gene deletion data set.

AB - Methods of transfer learning try to combine knowledge from several related tasks (or domains) to improve performance on a test task. Inspired by causal methodology, we relax the usual covariate shift assumption and assume that it holds true for a subset of predictor variables: the conditional distribution of the target variable given this subset of predictors is invariant over all tasks. We show how this assumption can be motivated from ideas in the field of causality. We focus on the problem of Domain Generalization, in which no examples from the test task are observed. We prove that in an adversarial setting using this subset for prediction is optimal in Domain Generalization; we further provide examples, in which the tasks are sufficiently diverse and the estimator therefore outperforms pooling the data, even on average. If examples from the test task are available, we also provide a method to transfer knowledge from the training tasks and exploit all available features for prediction. However, we provide no guarantees for this method. We introduce a practical method which allows for automatic inference of the above subset and provide corresponding code. We present results on synthetic data sets and a gene deletion data set.

KW - Transfer learning

KW - Multi-task learning

KW - Causality

KW - Domain adaptation

KW - Domain generalization

M3 - Journal article

VL - 19

SP - 1

EP - 34

JO - Journal of Machine Learning Research

JF - Journal of Machine Learning Research

SN - 1533-7928

IS - 1

ER -

ID: 203245100