Lecture SS 25 Advanced Topics in Numerical Methods in Science and Technology
Control Systems and Reinforcement Learning
Registration
Please register in ecampus for the course.
Content
The course is intended to explain the science behind reinforcement learning and optimal control in a way that is accessible to students with a background in analysis and linear algebra.
A focus of the course is on algorithm design to obtain the fastest possible speed of convergence for learning algorithms, along with insight into why reinforcement learning sometimes fails. Advanced stochastic process theory is avoided at the start by substituting random exploration with more intuitive deterministic probing for learning. Once these ideas are understood, it is not difficult to later in the course consider techniques rooted in stochastic control.
In reinforcement learning, we consider a system in interaction with some a priori (at least partially) unknown environment, which learns “from experience’, i.e. the underlying dynamical system is not perfectly known, but its effects have to be approximated during learning. Reinforcement learning is in its basic form very general, it is studied in many other disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, multi-agent systems, swarm intelligence, statistics, and genetic algorithms.
Prerequisites
Knowledge and understanding of analysis, numerical mathematics and statistics as taught in the first year of the Bachelor in Bonn is required.
Knowledge about ordinary differential equations is expected, including the (forward) Euler method as the elementary first-order numerical procedure to compute approximate solutions. Nevertheless, since we are mainly using results from these domains, and not extending them, students that have not been exposed to numerics of ordinary differential equations likely should be able to follow the course.
Numerical linear algebra, in the sense of computationally working with matrices, is expected.
Selected Literature:
- Meyn, S. Control Systems and Reinforcement Learning, Cambridge University Press, 2022. pre-publication draft
- Sutton, R., & Barto, A. Reinforcement Learning, MIT Press, 2018. second edition.
- Bertsekas, D. A Course in Reinforcement Learning, Athena Scientific, 2025. online resources