self-supervision (self-supervised)

A training setup where the correct label at every position already exists in the raw data itself, so no human needs to add it.

Self-supervision is a training setup where the target label the model is trained to predict is derived automatically from the raw data itself, rather than added by a human annotator. In pretraining, the "label" at every position is simply the next token that actually appears in the text -- so an enormous corpus of ordinary text becomes an enormous supervised dataset with zero manual labeling.

Examples

In pretraining, the raw sentence "the cat sat on the mat" supplies its own training signal: predict "cat" after "the," predict "sat" after "the cat," and so on -- no human ever labeled anything.