We will conduct online teaching for this course in Fall 2020.
This course is about the measure, representation and communication of information. Since information theory is considered as subset of communication theory, there are two fundamental questions in communication theory that can be answered by information theory: What is ultimate data compression and what is the ultimate transmission rate of communication. Accordingly, lectures will focus on fundamental topics of information theory including information measures, lossless data compression, losssy data compression, channel coding, relations to and manifestations in other areas (e.g., blockchains, machine learning,etc.)
This course is about the measure, representation and communication of information. Since information theory is considered as subset of communication theory, there are two fundamental questions in communication theory that can be answered by information theory: What is ultimate data compression and what is the ultimate transmission rate of communication. Accordingly, lectures will focus on fundamental topics of information theory including information measures, lossless data compression, losssy data compression, channel coding, relations to and manifestations in other areas (e.g., blockchains, machine learning,etc.)
The aim of this course is to provide fundamental knowledge of information theory to apply to a range of relevant real situations.
| Goals and objectives | Course Outcomes | |
|---|---|---|
| 1. | Understand the fundamentals of information theory so that the beauty and utility of information science can be exposed |
A-1
|
| 2. | Understand the fundamental limits in communication system |
A-1
|
| 3. | Apply the equipped knowledge of information theory to various disciplines in information science |
A-1
|
| Class schedule | HW assignments (Including preparation and review of the class.) | Amount of Time Required | |
|---|---|---|---|
| 1. | Introduction and probability review Measures of information (I): Entropy, divergence, mutual information, Fano inequality |
Read chapter 2 in the textbook | 190分 |
| 2. | Measures of information (II): the Asymptotic Equipartition Property (AEP) and typical sets: introduction to stochastic processes | Read chapter 3 and 4 in the textbook | 190分 |
| 3. | Lossless data compression (I): variable length compression and fixed length, shannon's first theorem, Shannon's Code, Kraft-McMilan inequality | Read chapter 5 and 6 in the textbook | 190分 |
| 4. | Lossless data compression (II): Huffman Codes, Shannon-Fano-Elias Codes, basics of universal data compression | Read chapter 5, 6 and 13 in the textbook | 190分 |
| 5. | Channel Coding (I): channel capacity and examples. Gaussian channels | Read chapter 7, 8 and 9 in the textbook | 190分 |
| 6. | Channel Coding (II): Coding with feedback: zero and non zero error capacities, Schalkwijk-Kailath and variable length codes | Read chapter 7, 8 and 9 in the textbook | 190分 |
| 7. | Mid-term exam and discussions on the solutions | Preparation for mid-term exam | 190分 |
| 8. | Binary hypothesis testing: large deviation theory, Chernoff-stein lemma | Read chapter 11 in the textbook | 190分 |
| 9. | Lossy data compression (I): rate-distortion function, direct and converse parts of main result | Read chapter 10 in the textbook | 190分 |
| 10. | Lossy data compression (II): Joint source-channel coding and the separation theorem | Investigate the concepts of joint source-channel coding and theorem of separation. | 190分 |
| 11. | Network information theory (I): Gaussian multiple-user channels, multiple-access channel | Read chapter 15 in the textbook | 190分 |
| 12. | Network information theory (II): Encoding of correlated sources, broadcast channel, relay channel. | Read chapter 15 in the textbook | 190分 |
| 13. | From theory to practice: machine learning, Image compression application, blockchain application | Investigate concepts in machine learning, image compression and blockchain technology | 190分 |
| 14. | Final exam and discussions on the solutions | Preparation for final exam | 190分 |
| Total. | - | - | 2660分 |
| Mid-term | Final | Total. | |
|---|---|---|---|
| 1. | 10% | 25% | 35% |
| 2. | 10% | 25% | 35% |
| 3. | 10% | 20% | 30% |
| Total. | 30% | 70% | - |
Midterm exam (30%) and Final Exam (70%) are the criteria of the grade. More than 60% of the total score is needed for getting
the course credit.
<Note>
Students are marked absent from the class if they are late regardless of the delay time.
If students are absent from more than one third of the total number of classes, the credit of this course cannot be given to them.
Even though students are absent from the class whatever the reason, e.g. sickness, delay of public transportation systems, forgetting to bring the student ID card, it is counted as absence.
<Note>
Students are marked absent from the class if they are late regardless of the delay time.
If students are absent from more than one third of the total number of classes, the credit of this course cannot be given to them.
Even though students are absent from the class whatever the reason, e.g. sickness, delay of public transportation systems, forgetting to bring the student ID card, it is counted as absence.
Elements of information theory - 2nd edition, Thomas M. Cover, Joy A. Thomas, 2006
- Weekdays: From 10:00 - 16:30 by email or face-to-face discussion at 4F-Research building office. Making an appointment beforehand is highly recommended.
- Course that cultivates an ability for utilizing knowledge
| Work experience | Work experience and relevance to the course content if applicatable |
|---|---|
| N/A | N/A |
Last modified : Fri Jul 31 04:03:10 JST 2020