Online / 5 & 6 February 2022



A set of Out-of-Distribution Generalization Benchmarks for Sequential Prediction Tasks

In the last decade, the field of AI has seen a great surge in capabilities in the field of machine learning techniques. Nowadays, models with up to billions of parameters are trained for wide arrays of downstream tasks and obtain performance that defy what a lot considered possible 20 years ago. However, reliance of machine learning models on the spurious correlations often prevents them from learning the intrinsic and invariant features of data which leads to their failure in generalizing to Out-Of-Distribution (OOD) data. Understanding and overcoming these failures has led to a research program on Out-Of-Distribution (OOD) generalization.

The field has been extensively explored in the static computer vision tasks (Domainbed, WILDS), but has not been explored for sequential prediction tasks that are important modalities for multiple areas of applied machine learning, e.g. medical, finance, communication. We propose a set of new open source out-of-distribution generalization datasets for sequential prediction tasks spanning multiple modalities that acts as benchmarks for potential algorithms that promotes invariant learning. Along with the datasets, we provide a fair and systematic open source platform for evaluating performance of existing and potential algorithms on these datasets. We also provide a leaderboard which currently consists of the performance of popular algorithms in the field of OOD generalization.


Photo of Jean-Christophe Gagnon-Audet Jean-Christophe Gagnon-Audet