Incorporating Inverse Reinforcement Learning in Urban Agent-Based Modeling: (Re-)Introducing Agent Heterogeneity

Increasing the realism of agent-based models requires the definition of behavioral mechanisms and rules. Knowledge regarding parameter values and even functional forms for defining this behavior is not always available, leading to arbitrariness in models. Machine learning approaches expose the dynamics underlying empirical data and can assist in overcoming this obstacle. Extending previous work, this article contributes by developing an approach that systematically considers heterogeneity among agents. The procedure replaces hand-crafted utility functions for deriving daily agent routines with reward functions learned using a profile-based training approach. This considers the unique characteristics of agents when generating trajectories. A further contribution is the development of a comprehensive feature matrix and the introduction of a trajectory termination procedure that uses an empirically-derived discount factor to verify that trajectory lengths mimic real-world patterns. We use a global positioning system trajectory dataset from the city of Jerusalem as training data. For implementation, we utilize an epidemiological urban agent-based model for a subsection of the city, calibrated to Covid-19. Results show a clear differentiation between the inverse reinforcement learning-augmented and nonaugmented models. The differences between the various profile-based implementations while more nuanced, also provide significant insights. They emphasize first, the importance of the spatial structure within the model instead of treating the city as unitary system and second, the nonlinearity of spatial utility. Avenues for realizing the full potential of this integration are identified. These include scaling up the simulation model, utilizing geometrical deep learning instead of a linear approach and defining a dynamic learning procedure for understanding mobility-related reactions to morbidity and recreating real-world results.