Joint Models for NLP

Motivation

Joint model

  1. Reduce error propagation
  2. Allow information exchange between task

Challenge

  1. Joint learning
  2. Search

Solutions

![](../asset/Joint Models for NLP/20190117094814.png)

Statistical Models

###Graph-Based Methods

Traditional solution
• Score each candidate, select the highest-scored output
• Search-space typically exponential

Joint Label Structure

Joint Learning, Joint Search

Joint Segmentation and POS tagging

Joint Parsing and NER

Reranking

Separate Learning, Joint Search

Joint Segmentation and POS Tagging

![](../asset/Joint Models for NLP/20190117101631.png)

Joint Parsing and SRL

Rerank k-best parse trees from a probabilistic parser using an SRL system.

Joint Modeling (Multi task)

Separate Learning, Joint Search

Joint Entity and Sentiment

Optimize the joint objective function which is defined as a linear combination of the potentials from different predictors with a parameter λ to balance the contribution of these two components: opinion entity identification and opinion relation extraction.

![](../asset/Joint Models for NLP/20190117102354.png)

Joint Modeling (Single task)

Joint Learning, Joint Search

Joint Segmentation and POS Tagging

The decoding algorithm for the joint word segmentor and POS tagger, agendas[i] stores the best sequences that end at i

Joint Entity Relation Extraction

Beam Search

###Transition-Based Methods

Transition-based Dependency Parsing

![](../asset/Joint Models for NLP/20190117103723.png)

A Learning+Search Framework

• Advantages
• Low computation complexity
• Arbitrary non-local features
• Learning-guided-search

• State-of-the-art accuracies and speeds
• Constituent parsing
• Dependency parsing
• Word Segmentation
• CCG parsing

• Enable joint models
• Address complex search space and use joint features, which have been difficult for traditional models

Joint Segmentation, Tagging and Normalization

Joint Segmentation, POS-tagging and Constituent Parsing

Traditional: word-based Chinese parsing

This: character-based Chinese parsing

Chinese words have syntactic structures.

Joint POS tagging and Dependency Parsing

Joint Entity and Relation Extraction

Deep Learning Models

Neural Transition-based Models

Joint Learning, Joint Search

Joint Entity and Relation Extraction

![](../asset/Joint Models for NLP/20190117113620.png)

Joint Parsing and SRL

Joint Word Segmentation, POS Tagging and Dependency Parsing

![](../asset/Joint Models for NLP/20190117140629.png)

![](../asset/Joint Models for NLP/20190117140757.png)

Joint Extraction of Entities and Relations

Neural Graph-based Models (Multi-task Learning)

Joint Learning, Separate Search

![](../asset/Joint Models for NLP/20190117141801.png)

Cross Task

Identifying beneficial task relations

![](../asset/Joint Models for NLP/20190117142724.png)

Cross Domain

Cross Lingual

Cross Standard