Homework 1
Release: Week 5 | Due: Week 8
Details and submission format will be announced in class and on the course page.
Intelligent Speech Technology
This course introduces core theory and modern practice in speech technology, from signal processing and ASR to speaker modeling, speech synthesis, audio and music processing, responsible AI, and speech agents.
Demos & Video Gallery
A curated set of demos and short videos for this course, organized by topic (speech fundamentals, ASR, dialogue, TTS, singing & music). Use it throughout the semester.
Course Overview
Instructor
Location
Nanyong Building West, Room 447（南雍楼西447）
Meeting Time
Friday, 2:00 - 3:50 PM
Teaching Format
Lectures + demos + project-oriented practice
As one of the most natural forms of human communication, speech is central to the next generation of AI systems. This course provides a structured path from fundamentals to state-of-the-art, covering both classical and neural approaches. We discuss practical design choices used in modern systems and how to critically read and implement research ideas.
Learning Objectives
- Build a clear understanding of speech signal processing foundations.
- Master core methods in ASR, speaker modeling, synthesis, voice conversion, and separation.
- Apply machine learning and deep learning methods to speech tasks.
- Develop and evaluate practical speech technology pipelines.
- Read, analyze, and present recent research papers in the field.
- Understand audio and music processing, responsible AI, and large-model-based speech systems.
Prerequisites
- Basic linear algebra and calculus.
- Programming experience in Python.
- Introductory machine learning knowledge.
- Basic deep learning familiarity (PyTorch or TensorFlow).
Weekly Schedule
| Week | Date | Topic | Materials | Notes |
|---|---|---|---|---|
| 1 | 2026.8.28 | Course Introduction & Overview of Speech Technology | Slides | - |
| 2 | 2026.9.4 | Overview of Speech Applications & Tasks | Slides | - |
| 3 | 2026.9.11 | Signal Processing Basics & Project Introduction | Slides Project | Several project topics are recommended; final topics need not be limited to these. |
| 4 | 2026.9.18 | Foundations of Automatic Speech Recognition | Slides Supplementary Material ASR Videos WER Demo | Assignment 1 Released |
| 5 | 2026.9.25 | Mid-Autumn Festival | - | No class meeting |
| 6 | 2026.10.2 | National Holiday | - | No class meeting |
| 7 | 2026.10.9 | Advanced ASR: End-to-End & Pre-training / LLMs | Slides (E2E) Slides (Pre-training / LLM) Demo | - |
| 8 | 2026.10.16 | Speaker Modeling | Slides Reading Demo (GMM–UBM) Demo (Softmax / Margin) | Guest talk: Prof. Ming Li (CUHK-Shenzhen) — Human Centered Intelligent Speech Processing |
| 9 | 2026.10.23 | Foundations of Speech Synthesis | - | - |
| 10 | 2026.10.30 | Advanced Speech Synthesis | - | - |
| 11 | 2026.11.6 | Voice Conversion | - | - |
| 12 | 2026.11.13 | Speech Separation | - | - |
| 13 | 2026.11.20 | Self-Supervised Learning for Speech | - | - |
| 14 | 2026.11.27 | Speech Processing with Large Language Models | - | - |
| 15 | 2026.12.4 | Audio and Music Processing | - | - |
| 16 | 2026.12.11 | Responsible Speech AI: Safety, Ethics, and Health | - | - |
| 17 | 2026.12.18 | Speech Agents and Full-Duplex Dialogue Systems | - | - |
| 18 | 2026.12.25 | Course Review & Final Project Presentation | - | Last class |
| 20 | 2027.1.8 | - | - | Final Project Due |
Course Materials
Demos & Interactive Tools
- Demo & Video Gallery — topic-grouped videos for fundamentals, ASR, dialogue, TTS, and singing.
- WER Demo — interactive Word Error Rate calculator for ASR evaluation.
Recommended Textbooks
- Jurafsky D, Martin J H. Speech and Language Processing, 2025 edition.
- Aalto Speech Processing Book.
- Huang X, Acero A, Hon H-W. Spoken Language Processing, Prentice Hall, 2011.
- 韩纪庆、张磊、郑铁然，《语音信号处理》，清华大学出版社。
- 洪青阳、李琳，《语音识别：原理与应用》，电子工业出版社。
Recommended Reading
Recent papers from top conferences and journals: ICASSP, Interspeech, ACL, T-ASLP, and related venues.
Software & Tooling
- Python 3.8+ for assignments and projects.
- PyTorch / TensorFlow for deep learning experiments.
- Librosa for audio and feature-level processing.
- WeNet / WeSpeaker / WeSep and related open-source speech toolkits.
- ESPNet / SpeechBrain / Kaldi for broader ecosystem exposure.
Assessment
15%
Attendance
Tracked from Week 4 to Week 18
20%
Homework
Two assignments
65%
Final Project
One term project and presentation
Assignments & Deliverables
Homework 2
Release: Week 13 | Due: To be announced
This assignment focuses on advanced topics covered in the second half of the course.
Final Project
Presentation: Week 18 | Final Submission: Week 20 (2027.1.8)
Students are expected to propose, implement, and evaluate a speech-related project with a clear technical report.
Course Policies
The syllabus and schedule are subject to refinement as the semester progresses. Any updates to lectures, deadlines, or materials will be announced in advance.