We introduce relevant subtask learning, a new learning problem which is a variant of multi-task learning. The goal is to build a classifier for a task-of-interest for which we have too few training samples. We additionally have "supplementary data" collected from other tasks, but it is uncertain which of these other samples are relevant, that is, which samples are classified in the same way as in the task-of-interest. The research problem is how to use the "supplementary data" from the other tasks to improve the classifier in the task-of-interest. We show how to solve the problem, and demonstrate the solution with logistic regression classifiers. The key idea is to model all tasks as mixtures of relevant and irrelevant samples, and model the irrelevant part with a sufficiently flexible model such that it does not distort the model of relevant data. We give two learning algorithms for the method---a simple maximum likelihood optimization algorithm and a more advanced variational Bayes inference algorithm; in both cases we show that the method works better than a comparable multi-task learning model and naive methods.

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

J. Peltonen and S. Kaski belong to Helsinki Institute for Information Technology HIIT and the Adaptive Informatics Research Centre, a Centre of Excellence of the Academy of Finland. They were supported by the Academy of Finland, decision numbers 207467 and 123983. Y. Yaslan was supported by the Center for International Mobility CIMO. This work was also supported in part by the IST Programme of the European Community, PASCAL2 Network of Excellence. This publication only reflects the authors’ views. We thank Y. Xue for providing SMTL code.