Towards Principled Sequential Decision-Making

Zoom link for remote attendees: password 123123

Abstract: Sequential decision-making studies how intelligent agents ought to make decisions in a dynamic environment to achieve their objectives. Its applications span diverse fields, from controlling robots to uncovering faster matrix multiplication algorithms and fine-tuning large language models (LLMs). In this talk, I will delve into my research on the theoretical foundations of sequential decision-making.

Firstly, I will talk about reinforcement learning with function approximation, a widely employed approach for addressing decision-making problems featuring enormous state spaces. Diverging from previous theories largely limited to linear-like scenarios, I will demonstrate that the classical Fitted Q-Iteration algorithm (the prototype of DQN), when combined with the idea of global optimism, is provably sample-efficient for a diverse array of problems involving generic nonlinear function approximation. In the second part, I will focus on partially observable decision-making in the framework of POMDP, a problem that has long been considered intractable within the theory community due to numerous hardness results. Contrary to this belief, I will reveal a rich class of POMDPs that are of practical interest and can be solved within polynomial samples using a variant of the classical maximum likelihood estimation algorithm. Finally, I will turn to multi-agent decision-making in the framework of Markov Game, where agents must learn to strategically cooperate or compete. I will introduce a fully decentralized algorithm capable of learning equilibria strategies with nearly minimax-optimal sample efficiency.

Bio: Qinghua Liu is a Ph.D. candidate in the Department of Electrical and Computer Engineering at Princeton University, advised by Chi Jin. He works on the theoretical foundations of sequential decision-making. His research has developed simple and generic algorithms that provably address fundamental challenges in decision-making, including but not limited to large state spaces, partial observability and multi-agency, all while providing reliability guarantees. His research has been recognized with the Princeton SEAS Award and a Best Paper Award at the ICLR 2022 MARL workshop.