Fast Learning Guarantees for Weakly Supervised Learning

We study generalization properties of weakly supervised learning. That is, learning where only a few true labels are present for a task of interest but many more “weak” labels are available. In particular, we show that embeddings trained using weak labels only can be fine-tuned on the downstream task of interest at the fast learning rate of O(1/n) where n denotes the number of labeled data points for the downstream task. This acceleration sheds light on the sample efficiency of pre-trained embeddings and can happen even if by itself true labeled data on the task of interest admits only the slower O(1/ \sqrt{n}) rate. The amount of acceleration depends continuously on the number of weak labels available, and on the relation between the two tasks. Our theoretical results are reflected empirically and illustrate how pre-training with weak labels improves sample efficiency.