We live in a 3D world, and our perception systems, despite only receiving 2D retinal percepts, can effortlessly understand the underlying 3D structure, e.g., observing a set of stacked dishes, we understand how they support one another. Even more remarkably, we understand not just what it is, but also what can be, e.g., removing a dish in the middle can cause the tower to collapse. While the computer vision community has made impressive advances in developing computational systems that can reconstruct the underlying 3D from visual input, these systems do not understand about actions and their effects in this context. This project will bridge this gap and build perception systems which have an actionable understanding of the 3D world they observe. Such systems can be broadly useful across applications in computer vision, robotics, and mixed reality, e.g., allowing robots to intelligently act and efficiently learn in generic scenarios and helping virtual assistants better understand and guide their user's actions. This project will also contribute to the development of undergraduate and graduate students via research engagements and development of a specialized course on embodied agents, as well as benefit the community at large through dissemination of research and organization of tutorials. To achieve its goal, this project will make research contributions along three thrusts: a) developing approaches to learn about 2D affordances (what actions can be performed) and world