This project addresses critical challenges in applying reinforcement learning (RL) to real-world urban environments, which have yet to achieve the same level of success as RL applications in controlled settings such as games or virtual simulations. The research focuses on developing actionable data analytics tailored to urban decision-making, tackling key issues including noisy and incomplete real-world observations, complex and dynamic urban system behaviors, and the necessity of human-in-the-loop decision-making to ensure interpretability and trust. Additionally, the project emphasizes the development of reproducible and cost-effective benchmarking environments to bridge the simulation-to-reality (sim-to-real) gap. By addressing these challenges, this project aims to advance the progress of science and support sustainable urban development. Technically, this project investigates the sim-to-real gap in a systematic way by addressing gaps in observations, system dynamics, and human interactions in urban decision-making. The research introduces innovative methodologies such as iterative optimization techniques, diffusion policy models, and a grounded action transformation framework enhanced by controllable domain context generation. It also develops uncertainty quantification and rule-based methods to support human collaboration in decision-making tasks. A significant output of this project is the creation of benchmarking environments to evaluate and refine RL policies und