Prefix-Tuning: Optimizing Continuous Prompts for Generation

Thursday, June 17, 2021, 11:00 am - 12:00 pm PDTiCal
Where: event is open to the public.
NL Seminar
Xiang Lisa Li (Stanford University)
Video Recording:

Reminder: ​Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you're highly encouraged to use your USC account to sign into Zoom. If you're an outside visitor, please inform beforehand so we'll be aware of your attendance and let you in.

Fine-tuning is the de facto way of leveraging large pretrained language models for downstream tasks. However, fine-tuning modifies all the language model parameters and therefore necessitates storing a full copy for each task. In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which we call the prefix. Prefix-tuning draws inspiration from prompting for language models, allowing subsequent tokens to attend to this prefix as if it were “virtual tokens”. We apply prefix-tuning to GPT-2 for table-to-text generation and to BART for summarization. We show that by modifying only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting, outperforms fine-tuning in low-data settings, and extrapolates better to examples with topics that are unseen during training. 

Bio: Xiang Lisa Li is a first-year PhD student in computer science at Stanford University, advised by Percy Liang and Tatsunori Hashimoto. She works on controllable text generation/decoding and efficient adaptation of pre-trained language models. Lisa is supported by a Stanford Graduate Fellowship and is the recipient of an EMNLP Best Paper award.

Recording approved by Speaker

The recording will be posted on USC/ISI's YouTube site within 1-2 business days from the date of the presentation.

« Return to Upcoming Events