Advanced Natural Language Processing


Course organization

Instructor: Tatjana Scheffler
TA: Jens Johannsmeier
Time: Tuesdays & Fridays, 10 a.m.-12
Place: Golm, building 28, room 104 (Tue)
Golm, building 12, room 001 (Fri)
Modules: CSBM1 (M.Sc. Cognitive Systems)
Moodle: Please register on the course's Moodle site.


Grading policy

Passing the course

To be admitted to the module exam, you need to pass the course. For this, we will grade the best two assignments out of each half of the semester (i.e., the best 2 from the first 3 + the best 2 from the second 3). At least 250 points in total (out of 400) in these 4 assignments are needed to pass the course.

Module grade

The grade will be based on a collaborative final project, to be completed during the semester break. There are four graded deliverables for this project:

  1. a planning paper (individual)
  2. a project presentation (group)
  3. the implemented project (group)
  4. a project report (individual)

The grade will be composed equally from these four parts. Details will be discussed in class.

Course description

This class is the graduate-level introduction to computational linguistics, a first-year class in the MSc Cognitive Systems. The purpose of this class is to introduce the important concepts, models and methods used in natural language processing (NLP). After the successful completion of this course, students should be able to (i) read and understand the scientific literature in the area of computational linguistics and (ii) start implementing their own NLP projects.

We will cover the following topics:


Readings in J/M are marked for the third edition unless marked otherwise.

Date Topic Readings Assignments
T 10/17 Introduction
F 10/20 Review: Probability Theory Harald Goldstein, A short introduction to probability and related concepts
And/Or: Manning/Schütze, chapter 2.1
Optional: Kevin Murphy, Binomial and multinomial distributions
T 10/24 N-grams Jurafsky/Martin, chapters 4.1-4.2
A video about Zipf's law
A1 released
F 10/27 practical concerns (Jens) lab session for discussion of the assignments, Python, NLTK, etc.
T 31/10 holiday - no class
F 11/3 Smoothing language models Jurafsky/Martin, chapters 4.5-4.7
Opt.: Manning/Schütze, chapters 6.2-6.3
T 11/7 Part of speech tagging Jurafsky/Martin, chapters 9.1-9.5, 10.1-10.5 A1 due, A2 released
F 11/10 HMM Training Jurafsky/Martin, rest of chapters 9+10
HMM spreadsheet, Eisner's ice cream example
Eisner's paper explaining how to work with the spreadsheet
T 11/14 discussion of A1 be prepared to present your solutions!
F 11/17 tutorial session (Jens)
T 11/21 Context free grammars, CKY parsing Jurafsky/Martin, chapter 12 (-12.4),
J/M chapter 11 as background
further reading: Santorini/Kroch, Online syntax textbook
CKY animation
A2 due, A3 released
F 11/24 discussion of A2 (Jens) be prepared to present your solutions!
T 11/28 PCFGs Jurafsky/Martin, chapters 13.1-13.5, 13.7
F 12/1 Training PCFGs Jurafsky/Martin, chapter 13
Manning/Schütze, chapter 11
Michael Collins, The inside-outside algorithm.
T 12/5 Advanced PCFG models Mark Johnson (1998), PCFG Models of Linguistic Tree Representations (esp. on parent annotations)
Michael Collins, Lexicalized PCFGs
Dan Klein/Chris Manning (2003), Accurate unlexicalized parsing
A3 due, A4 released
F 12/8 discussion of A3 be prepared to present your solutions!
T 12/12 Dependency parsing McDonald/Pereira/Ribarov/Hajic (2005), Non-projective Dependency Parsing using Spanning Tree Algorithms
Joakim Nivre (2008), Algorithms for Deterministic Incremental Dependency Parsing
F 12/15 tutorial session (Jens)
T 12/19 Statistical machine translation: Alignments Jurafsky/Martin (2nd ed.), ch. 25 (through 25.6)
Adam Lopez, Word Alignment and the Expectation-Maximization Algorithm tutorial (try here)
MT Talks
A4 due

no class (winter break)

no class (winter break)
F 1/5 Phrase-based machine translation Jurafsky/Martin (2nd ed.), ch. 25 A5 released
T 1/9 discussion of A4 be prepared to present your solutions!
F 1/12 Syntax-based machine translation David Chiang, Hierarchical phrase-based translation. Computational Linguistics, 2007.
T 1/16 Semantic parsing Zettlemoyer/Collins, Learning to Map Sentences to Logical Form. Structured Classification with Probabilistic Categorial Grammars. 2005
Wong/Mooney, Learning for Semantic Parsing with Statistical Machine Translation. HLT-NAACL, 2006
Opt.: Mark Steedman, A very short introduction to CCG. 1996
F 1/19 Lexical semantics Jurafsky/Martin, chs. 15 + 16
Further reading: Mitchell/Lapata (2008), Vector-based models of semantic composition; Baroni/Zamparelli (2010), Nouns are vectors, adjectives are matrices
A5 due, A6 released
T 1/23 Speech recognition Jurafsky/Martin (2nd ed.), ch. 9
Links for further reading:
"Human parity" speech recognition
Language log on human parity speech recognition
Google speech API web demo
S. Germesin on using the speech API remotely
F 1/26 Speech synthesis Jurafsky/Martin (2nd ed.), ch. 8
Links for further reading/system samples:
WaveNet: Generating raw audio
Deep Learning in Speech Synthesis
Expressive and emotional synthetic speech
T 1/30 LDA David Blei, Probabilistic topic models. 2012
Links for further reading:
Steyvers/Griffiths, Probabilistic Topic Models. 2007.
Darling, A Theoretical and Practical Implementation Tutorial on Topic Modeling and Gibbs Sampling.
F 2/2 Intro to deep neural networks Richard Socher, Recursive Deep Learning for Natural Language Processing and Computer Vision (Chapter 2). 2014
Further reading (for intuitions):,
A6 due
T 2/6 presentations of final projects all
F 2/9 final discussion be prepared to present your solutions of A5 + A6!


Most computational linguists own both of these books. We will assign weekly readings, so you should ensure you get your own copy or have access to the copies that are available in the university library.

Last modified: Tue Jan 30 11:42:58 CET 2018