Homework 1
Release: Week 5 | Due: Week 8
Details and submission format will be announced in class and on the course page.
Intelligent Speech Technology
This course introduces core theory and modern practice in speech technology, from signal processing and ASR to speaker modeling, speech synthesis, audio and music processing, responsible AI, and speech agents.
Course Overview
Instructor
Location
SuJiao Building, Room C201
Meeting Time
Friday, 2:00 - 3:50 PM
Teaching Format
Lectures + demos + project-oriented practice
As one of the most natural forms of human communication, speech is central to the next generation of AI systems. This course provides a structured path from fundamentals to state-of-the-art, covering both classical and neural approaches. We discuss practical design choices used in modern systems and how to critically read and implement research ideas.
Learning Objectives
- Build a clear understanding of speech signal processing foundations.
- Master core methods in ASR, speaker modeling, synthesis, voice conversion, and separation.
- Apply machine learning and deep learning methods to speech tasks.
- Develop and evaluate practical speech technology pipelines.
- Read, analyze, and present recent research papers in the field.
- Understand audio and music processing, responsible AI, and large-model-based speech systems.
Prerequisites
- Basic linear algebra and calculus.
- Programming experience in Python.
- Introductory machine learning knowledge.
- Basic deep learning familiarity (PyTorch or TensorFlow).
Weekly Schedule
| Week | Date | Topic | Materials | Notes |
|---|---|---|---|---|
| 1 | 2026.8.28 | Course Introduction & Overview of Speech Technology | Demos | - |
| 2 | 2026.9.4 | Foundations of Speech Signal Processing | - | - |
| 3 | 2026.9.11 | Foundations of Automatic Speech Recognition | WER Demo | - |
| 4 | 2026.9.18 | Advanced Automatic Speech Recognition | - | - |
| 5 | 2026.9.25 | Speaker Modeling | - | Assignment 1 Released |
| 6 | 2026.10.2 | National Holiday | - | No class meeting |
| 7 | 2026.10.9 | Foundations of Speech Synthesis | - | - |
| 8 | 2026.10.16 | Advanced Speech Synthesis | - | - |
| 9 | 2026.10.23 | Voice Conversion | - | - |
| 10 | 2026.10.30 | Speech Separation | - | - |
| 11 | 2026.11.6 | Self-Supervised Learning for Speech | - | - |
| 12 | 2026.11.13 | Speech Processing with Large Language Models | - | - |
| 13 | 2026.11.20 | Audio and Music Processing | - | Assignment 2 Released |
| 14 | 2026.11.27 | Responsible Speech AI: Safety, Ethics, and Health | - | - |
| 15 | 2026.12.4 | Speech Agents and Full-Duplex Dialogue Systems | - | - |
| 16 | 2026.12.11 | Applications of Speech Processing in Industry | - | - |
| 17 | 2026.12.18 | Course Review & Project Clinic | - | - |
| 18 | 2026.12.25 | Final Project Presentation | - | Last class |
| 20 | 2027.1.8 | - | - | Final Project Due |
Course Materials
Recommended Textbooks
- Jurafsky D, Martin J H. Speech and Language Processing, 2025 edition.
- Aalto Speech Processing Book.
- Huang X, Acero A, Hon H-W. Spoken Language Processing, Prentice Hall, 2011.
- 韩纪庆、张磊、郑铁然，《语音信号处理》，清华大学出版社。
- 洪青阳、李琳，《语音识别：原理与应用》，电子工业出版社。
Recommended Reading
Recent papers from top conferences and journals: ICASSP, Interspeech, ACL, T-ASLP, and related venues.
Software & Tooling
- Python 3.8+ for assignments and projects.
- PyTorch / TensorFlow for deep learning experiments.
- Librosa for audio and feature-level processing.
- WeNet / WeSpeaker / WeSep and related open-source speech toolkits.
- ESPNet / SpeechBrain / Kaldi for broader ecosystem exposure.
Assessment
15%
Attendance
Tracked from Week 4 to Week 18
20%
Homework
Two assignments
65%
Final Project
One term project and presentation
Assignments & Deliverables
Homework 2
Release: Week 13 | Due: To be announced
This assignment focuses on advanced topics covered in the second half of the course.
Final Project
Presentation: Week 18 | Final Submission: Week 20 (2027.1.8)
Students are expected to propose, implement, and evaluate a speech-related project with a clear technical report.
Course Policies
The syllabus and schedule are subject to refinement as the semester progresses. Any updates to lectures, deadlines, or materials will be announced in advance.