Neural Nets (596)

Type: Lecture with Exercises
ECTS: 6 points
Lectures: Fridays, 08:30-12:00
Venue: BIN 2.A.01
Lecturers: Prof. Rolf Pfeifer
Target audience: Recommended for MSc students. The course is interdisciplinary; it is also targeted at students from other fields than computer science, e.g. economics, biology, natural sciences, and psychology.
Language: English
Assessment: Written exam. Date: Friday, 14.06.2013, 8:00 - 9:30am
Assistants: Nico Schmidt, Matthias Weyland

Systematic introduction to neural networks, biological foundations; important network classes and learning algorithms; supervised models (perceptrons, adalines, multi-layer perceptrons), support-vector machines, echo-state networks, non-supervised networks (competitive, Kohonen, Hebb), recurrent networks (Hopfield, CTRNNs - continuous-time recurrent neural networks), spiking neural networks, spike-time dependent plasticity, applications. Special consideration will be given to neural networks embedded in adaptive systems having to interact with the real world, such as embodied systems (in particular robots). Cooperation of neural control, morphology, materials, and environment. Evolutionary approaches to designing autonomous systems; interaction of learning and evolution. Network theory applied to brain networks; motifs.

Additional case studies will be discussed to deepen the understanding of neural networks, e.g. Neural interfacing - coupling neural systems with technology (in particular robotic devices), neural imaging studies, Distributed Adaptive Control (DAC), neural gas and DRNNs - Dynamically Rearranging Neural Networks (neuro-modulator-based networks), neural network models of memory.

This is an elementary, interdisciplinary introduction to neural networks, suited not only for computer scientists, but also for economists, biologists, psychologists, etc.

If you wish, you can do the exercises in groups of two - please hand in only one task sheet. You can also do them on your own if you prefer.

Important

The page will be subjected to changes over time. Please stay up-to-date by checking it periodically.

Final Exam

List of topics relevant for exam (PDF, 69 KB)

List of topics relevant for the final exam.

The final exam will be Friday, 14 June, 2013, from 8.00 to 9.30am in the lecture room 2.A.01.

In order to attend the final exam, you must achieve 50% out of the total possible points of all the exercises (Task sheets 1-4) together (not each).

It will not be an open-book exam, so you are not allowed to use your books and notes. However, you don't have to learn any formulas by heart, as we will provide you with a sheet containing all formulas (but you do have to know which one applies to which type of Neural Net). No laptops or cell phones.

Please bring along:

  • simple pocket calculator (no cell phones or other advanced programmable devices)
  • your student ID card (required)

We wish you a lot of success.


Date Topic Lecture Exercises
22 February Introduction, Linear Algebra For an intuitive, 50min introduction to artificial neural networks, please consult this video which was recorded in the context of the ShanghAI Lectures. All the points raised in this video will be taken up again and discussed in more detail later in the class. Please also consult the pdf of the slides for this lecture.
Neural networks require relatively little prior knowledge in mathematics, just some linear algebra and a bit of elementary calculus. For those who are not confident about their linear algebra skills, we will provide an introductory tutorial - including a set of exercises during the first lecture on 22 February, starting approx at 10.00h. If you are confident that you already master basic linear algebra, you don't need to attend this tutorial.
1 march Supervised models Perceptron, Adaline, delta-rule
8 March No lecture Special event: Robots on Tour www.robotsontour.com
15 March Supervised models Back-propagation: examples, properties; Error surfaces, Momentum term, Other improvements; N-fold cross-validation, VC dimension
Task sheet 1 (PDF, 232 KB)
(Due date: 12 April)
Sample solutions (PDF, 179 KB)
22 March Supervised models Cascade correlation, Suport vector machines (SVMs)
29 March No lecture
5 April No lecture
12 April Supervised models Cascade correlation, Suport vector machines (SVMs) hand in Task Sheet 1
Task sheet 2 (ZIP, 133 KB)
(Due date: 26 April)
Task sheet 2 sample solutions (PDF, 129 KB)
19 April Recurrent neural networks Hopfield nets, Stocastic Models, CTRNNs (Continuous Time Recurrent Neural Network)
26 April Hybrid models Guest lecture: Naveen Kuppuswamy: Reservoir computing
Slides (PDF, 4 MB)
 hand in Task Sheet 2;
TaskSheet 3 (Due date: 10 May) (PDF, 227 KB)

Videos of Lectures by Andrew Ng on SVMs: 1, 2, 3
TaskSheet 3 solutions (PDF, 240 KB)
3 May Unsupervised models Nico Schmidt on Hebbian Learning, PCA, Oja's rule, Sanger's rule
10 May Biologically more plausible models Guest lecture: Pascal Kaufmann: Basic neurophysiology, Spiking neurons, Cyborgs, Lamprey experiment, Brain imaging
Slides (PDF, 4 MB)
TaskSheet 4 (Due date: 24 May) (PDF, 1 MB)

Simulator (ZIP, 315 KB)
kohonen.testpattern (unzip first) (ZIP, 944 bytes)
Task Sheet 4 solution (PDF, 1 MB)
17 May Unsupervised models Competitive learning, SOM, Kohonen-algorithm, Extended Kohonen map (robot arm), Adaptive light compass. it is OK to hand in Task Sheet 3 on 17 May!
24 May Application of recurrent networks Morphological Computation, Evolutionary Robotics, Co-evolution of morphology and control
Morph. Comp. slides (PDF, 22 MB)
Co-Evolution slides (PDF, 7 MB)
 hand in Task Sheet 4
31 May Wrap-up Wrap-up session, questions, final discussion
14 Jun Exam


  • J. Hertz, A. Krogh, R. Palmer, "Introduction to the theory of neural computation", Addison-Wesley Publishing Company. A "classic"; a bit mathematical, but sound, written by physicists. Recommended as a complement to the lecture script. It covers most but not all topics of the class (e.g. Support Vector Machines, spiking neurons, etc.).
  • S. Haykin, "Neural Networks: A comprehensive foundation", Prentice Hall. Very comprehensive, covers most of the topics of the class. Can also be used as an introductory textbook and as a complement to the class. It also introduces quite a few topics that go beyond the class.
  • N. Cristianini, J. Shawe-Taylor, "An Introduction to Support Vector Machines and other kernel-based learning methods", Cambridge University Press. Nice introduction to kernel-based learning machines. Mainly for the mathematically minded student. Support Vector Machines will be covered in class and are included in the book by Haykin.
  • R. Rojas, "Neural Networks - A Systematic Introduction", Springer-Verlag, 1996. A nicely and comprehensively written overview of the field with robotics application examples.
    The book can be downloaded for free here:
    http://www.inf.fu-berlin.de/inst/ag-ki/rojas_home/pmwiki/pmwiki.php?n=Books.NeuralNetworksBook

Materials that are useful for the understanding of the course:

Some links on the internet that are useful for the understanding of the course