CS5480 Deep Learning
Jan-Apr 2018

Instructor:


CONTENT:  

Deep learning is a sub-area of machine learning, considered to be the reincarnation of neural networks. However, with the increased availability of vast amounts of data and compute capability, it has evolved to a field of its own in the last few years with numerous applications in computer vision, speech understanding and natural language processing. While the fundamental methods to train deep neural networks has remained the same for three decades now, a wide variety of architectures for different application settings and a large group of methods for training them more effectively and efficiently have emerged over these years. This course will describe all these various facets of deep learning, including its applications to different domains. Like machine learning, deep learning has mathematical foundations, and this course will cover these foundations as and when possible/required. Students can expect to achieve the following objectives at the end of the course:

  • Be familiar with the fundamentals of deep learning methods, as well as its applications to various kinds of data such as images, videos, graphs.

  • Be familiar with the programming frameworks commonly used for deep learning today

  • Be able to apply the learned concepts and methods to a real-world problem

  • Learn to appreciate the mathematical rigor behind these methods where possible


PRE-REQUISITES:

  • Prior completion of Machine Learning course at the level of CS6510 (or equivalent)

  • Familiarity with basic Probability Theory, Linear Algebra, Calculus (especially Matrix Calculus)

  • Programming proficiency in Python

.

COURSE PORTAL:

We will use Google Classroom for sharing materials and discussions in the course. The Classroom link will be shared with the students registered for the course.


COURSE SCHEDULE (TENTATIVE):


Week

Topics


Jan 4

Course introduction, Introduction to deep learning 


Jan 8, Jan 11

Feedforward neural networks, Backpropagation 


Jan 15, Jan 18

Regularization methods, Optimization methods


Jan 22, Jan 25

Computational graphs, Computing gradients


Jan 29, Feb 1

Convolution and Feature Extraction in Images


Feb 5, Feb 8

Convolutional Neural Networks (CNNs), Visualizing and Understanding CNNs

Exam 1

Feb 12, Feb 15

Variants (Detection and Segmentation), Siamese CNNs


Feb 19, Feb 22

Ultra Deep Architectures, ResNets, Highway Networks, Capsule Nets


Feb 26, Mar 1

Mid-sem Break


Mar 5, Mar 8

CNNs on Graphs/Non-Euclidean Domains, Spectral Networks


Mar 12, Mar 15

Sequence learning with NNs, Recurrent Neural Networks, LSTM and GRU


Mar 19, Mar 22

Neural Networks with External Memory, Memory Networks, Neural Turing Machines

Exam 2

Mar 26, Mar 29

Deep Unsupervised Learning, Autoencoders

Mahavir Jayanti Holiday


Apr 2, Apr 5

Deep Generative Models, Variational Autoencoders


Apr 9, Apr 12

Generative Adversarial Networks (GANs), Wasserstein GANs


Apr 16, Apr 19

Reinforcement Learning, Deep Reinforcement Learning, AlphaGo


Apr 23, Apr 26

Buffer and additional lectures (optimization, theory of deep learning)


Apr 30 - May 4

End-sem Exam


* We will have guest lectures as and when possible during the lectures from renowned researchers in deep learning

Lecture Slides:


GRADING CRITERIA (TENTATIVE):

  • 10%: Quizzes (Surprise)/Class Participation

  • 25%: Endsem Exam

  • 20%: Other Exams

  • 20%: Assignments/Homework

  • 25%: Project


Other administrivia:

  • 7 grace days (upto 4 can be used for a single submission) to start with – use them wisely. May not apply to some deadlines, which will be pointed out.


REFERENCES:

Deep learning is a rapidly evolving field, and we will hence use multiple sources of references, including books, blogs and articles, each of which will be pointed out at the end of each topic. However, there are three recent books that ground a lot of the fundamentals. In particular, the book by Goodfellow, Bengio and Courville is highly recommended, not only for the quality of its discussions, but also given that it has widest coverage of topics. It is also the most up-to-date and will be followed in most of the lectures.

Main references:

Deep Learning By Ian Goodfellow and Yoshua Bengio and Aaron Courville, MIT Press, 2016, pdf

Neural Networks and Deep Learning, By Michael Nielsen, Online book, 2016

Learning Deep Architectures for AI (slightly dated) By Yoshua Bengio, NOW Publishers, 2009


Tools

We recommend the use of PyTorch or TensorFlow for projects.

Other useful references:

  • Bishop, Christopher. Neural Networks for Pattern Recognition. New York, NY: Oxford University Press, 1995. ISBN: 9780198538646.

  • Bishop, Christopher M. Pattern Recognition and Machine Learning. Springer, 2006. ISBN 978-0-387-31073-2

  • Duda, Richard, Peter Hart, and David Stork. Pattern Classification. 2nd ed. New York, NY: Wiley-Interscience, 2000. ISBN: 9780471056690.

  • Mitchell, Tom. Machine Learning. New York, NY: McGraw-Hill, 1997. ISBN: 9780070428072.

  • Stanford Winter Quarter 2016, 2017 class  ,


Last modified: Saturday, 31 March 2018, 9:17 AM