156页！NLPCC 2020《预训练语言模型回顾》讲义下载

声明：本文转载自 哈工大讯飞联合实验室 公众号

哈工大讯飞联合实验室（HFL）资深级研究员、研究主管崔一鸣受邀在NLPCC 2020会议做题为《Revisiting Pre-trained Models for Natural Language Processing》的讲习班报告（Tutorial），介绍了预训练语言模型的发展历程以及近期的研究热点。本期推送文末提供了报告的下载方式。

NLPCC 2020 Tutorials:

http://tcci.ccf.org.cn/conference/2020/tutorials.php

报告信息

Title: Revisiting Pre-Trained Models for Natural Language Processing

Abstract : Pre-Trained Language Models (PLM) have become fundamental elements in recent research of natural language processing. In this tutorial, we will revisit the technical progress of the text representations, i.e., from one-hot embedding to the recent PLMs. We will describe several popular PLMs (such as BERT, XLNet, RoBERTa, ALBERT, ELECTRA, etc.) with their technical details and utilizations. On the other hand, we will also introduce various efforts on Chinese PLMs. At the end of this talk, we will analyze the shortcomings of the recent PLMs and envision the directions of future research.

报告目录

  • Introduction

  • Traditional Approaches for Text Representation

    • one-hot, word2vec, GloVe

  • Contextualized Language Models

    • CoVe, ELMo

  • Deep Contextualized Language Models

    • GPT, BERT, XLNet, RoBERTa, ALBERT, ELECTRA

  • Chinese Pre-trained Language Models

    • Chinese BERT-wwm, ERNIE, NEZHA, ZEN, MacBERT

  • Recent Research on PLM

    • Trending: GPT-2, GPT-3, T5

    • Distillation: DistilBERT, TinyBERT, MobileBERT, TextBrewer

    • Multi-lingual: mBERT, XLM, XLM-R

  • Summary

讲义部分内容截图

讲义下载

赞 (0)

相关推荐