Dokuz Eylül University Information Package / Courses Catalog

Description of Individual Course Units

Course Unit Code	Course Unit Title	Type Of Course	D	U	L	ECTS
CSC 5004	INFORMATION AND ENTROPY	ELECTIVE	3	0	0	7

Offered By

Graduate School of Natural and Applied Sciences

Level of Course Unit

Second Cycle Programmes (Master's Degree)

Course Coordinator

PROFESSOR DOCTOR ÖZLEM EGE ORUÇ

Offered to

Ph.D. in Computer Science (English)
Computer Science

Course Objective

The course will study how information is measured in terms of probability and entropy, and the relationships among conditional and joint entropies.
How concepts of randomness and uncertainty are related to information. Ensembles, random variables, marginal and conditional probabilities. How the metrics of information are grounded in the rules of probability.
Marginal entropy, joint entropy, conditional entropy, and the Chain Rule for entropy. Mutual information between ensembles of random variables. Why entropy is the fundamental measure of information content. At the end of the course students should be able to
1. calculate the information content of a random variable from its probability distribution
2. relate the joint, conditional, and marginal entropies of variables in terms of their coupled probabilities.

Learning Outcomes of the Course Unit

1	An understanding of fundamental ideas of information and entropy,
2	Have a good understanding of entropy in information theory.
3	An understanding of advanced structure of entropy,
4	Produce entropy and chain rule in the computer programs,
5	Develop the students analytical abilities and ability to present and criticize arguments.

Mode of Delivery

Face -to- Face

Prerequisites and Co-requisites

None

Recomended Optional Programme Components

None

Course Contents

Week	Subject	Description
1	1. Information Sources 1.1. Introduction 1.2. Probability Spaces and Random Variables 1.3. Random Processes and Dynamical Systems 1.4. Distributions
2	2. Entropy and Information 2.1. Introduction 2.2. Entropy and Entropy Rate 2.3. Basic Properties of Entropy 2.4. Entropy Rate
3	3.Joint Entropy and Conditional Entropy 3.1. Introduction 3.2. Joint Entropy 3.3. Conditional Entropy and Information
4	4.Relative Entropy 4.1. Introduction 4.2. Divergence 4.3. Relative Entropy
5	5. Mutual Information 5.1. Introduction 5.2. Relationship Between Entropy and Mutual Information 5.3. Chain Rules for Entropy, Relative Entropy, and Mutual Information
6	6. Entropy Rates of a Stochastic Process 6.1. Introduction 6.2. Markov Chains 6.3. Entropy Rate 6.4. Example: Entropy Rate of a Random Walk on a Weighted Graph
7	7. Data Compression 7.1. Introduction 7.2. Examples of Codes (Ex.Huffman Kodlama, Aritmetik Kodlama, Lempel Ziv Kodu, etc.)
8	Mid-term examination
9	9. Differential Entropy 9.1. Introduction 9.2. Relation of Differential Entropy to Discrete Entropy 9.3. Joint and Conditional Differential Entropy 9.4. Properties of Differential Entropy, Relative Entropy, and Mutual Information
10	10. Information Theory and Statistics 10.1. Method of Types 10.2. Law of Large Numbers 10.3. Conditional Limit Theorem 10.4. Hypothesis Testing
11	11. Maximum Entropy 11.1. Maximum Entropy Distributions 11.2. Examples
12	12. Inductive learning 12.1. Rule extraction 12.2. Examples
13	13.Applications The Entropy of English
14	Project Evaluation

Recomended or Required Reading

Textbook(s):
Cover, T. M., Thomas, J. A., Elements of Information Theory, Second Edition, John Wiley& Sons, 2006.

Supplementary Book(s):
MacKay David, J.C., Information Theory, Inference, and Learning Algorithms, U.K., 2004

Planned Learning Activities and Teaching Methods

The course consists of lecture and homework.

Assessment Methods

SORTING NUMBER	SHORT CODE	LONG CODE	FORMULA
1	MTE	MIDTERM EXAM
2	ASG	ASSIGNMENT
3	FIN	FINAL EXAM
4	FCG	FINAL COURSE GRADE	MTE * 0.40 +ASG * 0.10 +FIN * 0.50
5	RST	RESIT
6	FCGR	FINAL COURSE GRADE (RESIT)	MTE * 0.40 +ASG * 0.10 +RST * 0.50

*** Resit Exam is Not Administered in Institutions Where Resit is not Applicable.

Further Notes About Assessment Methods

None

Assessment Criteria

Evaluation of exams, and homework.

Language of Instruction

Turkish

Course Policies and Rules

To be announced.

Contact Details for the Lecturer(s)

Assoc.Prof.Dr.Emel KURUOĞLU KANDEMIR
e-posta: emel.kuruoglu@deu.edu.tr
Tel: 0232 301 95 10

Office Hours

Will be announced.

Work Placement(s)

None

Workload Calculation

Activities	Number	Time (hours)	Total Work Load (hours)
Lectures	14	3	42
Preparations before/after weekly lectures	12	1	12
Preparation for midterm exam	1	48	48
Preparation for final exam	1	48	48
Preparing assignments	4	7	28
Final	1	2	2
Midterm	1	2	2
TOTAL WORKLOAD (hours)			182

Contribution of Learning Outcomes to Programme Outcomes

PO/LO	PO.1	PO.2	PO.3
LO.1	3	4	4
LO.2	3	4	4
LO.3	3	4	4
LO.4	3	4	4
LO.5	3	4	4

COURSE UNIT TITLE

DEGREE PROGRAMMES

Description of Individual Course Units

Offered By

Level of Course Unit

Course Coordinator

Offered to

Course Objective

Learning Outcomes of the Course Unit

Mode of Delivery

Prerequisites and Co-requisites

Recomended Optional Programme Components

Course Contents

Recomended or Required Reading

Planned Learning Activities and Teaching Methods

Assessment Methods

Further Notes About Assessment Methods

Assessment Criteria

Language of Instruction

Course Policies and Rules

Contact Details for the Lecturer(s)

Office Hours

Work Placement(s)

Workload Calculation

Contribution of Learning Outcomes to Programme Outcomes

CONTACT INFORMATION