What’s the Place for Planning?

Abstract: Our research began with a fundamental question in computational neuroscience: how do biological systems learn effectively across multiple time scales? Our initial approach using transformer-based RL (Q-Zero) led us to develop D2AC, a diffusion-based policy optimization method. However, this work revealed a more fundamental issue: standard RL agents optimize for population-level success at the cost of individual survival, taking unnecessary risks that no biological entity would consider rational. Our latest work, ‘Of Mice and Machines,’ systematically analyzes this survival-value disparity between artificial and biological learning, where mice consistently prioritize individual survival while RL agents sacrifice it for marginally better aggregate performance. We’ve developed new mathematical frameworks that modify the core RL objective function with variance-based components, making artificial agents value their ‘lives’ more like biological entities do. Our results show that these biologically-inspired modifications not only produce more natural behavior but also improve performance on navigation tasks. The anticipated outcomes include both a deeper understanding of biological risk-assessment principles and more effective artificial learning algorithms that better mirror natural learning processes. This funding also allowed the completion of a publication on converting static plans into time-varying movement commands, and several upcoming preprints on movement representation in mouse sensorimotor areas and dynamical inference in noisy nonlinear systems. The Northwestern PIs leading the effort are: Assistant Professor Bradly Stadie, Dept of Statistics and Data Science, Professor Malcolm A. MacIver, Dept of Mechanical Engineering, Dept of Biomedical Engineering, Professor Daniel A. Dombeck, Dept of Neurobiology. NU Trainees being funded: German Espinosa (Postdoc); Chris Angeloni (Postdoc). The University of Chicago PI is Assistant Professor Matthew Kaufman, Department of Organismal Biology and Anatomy;Zulfar Ghulam-Jelani, PhD student.