Abstract In many surgical procedures, image guided surgery (IGS), also known as surgical navigation, facilitates precise surgical manipulations near critical structures such as the brain and eye. IGS suffers from critical limitations such as loss of precision and correspondence with preoperative imaging once surgery begins and the anatomy is altered. These limitations may be addressed through intra-operative imaging, which effectively “resets” the model and thus enables both improved precision of IGS as well as improved surgical decision- making. However, intraoperative radiologic imaging has many limitations, including disruption of the workflow, cost, and additional radiation exposure. This project aims to develop solutions to resolve limitations of IGS without the need for intra-operative radiologic imaging. Our solutions rely upon data from a device that is present in every procedure using IGS, namely, the endoscope. Our goal in this project is use endoscopic video images to reconstruct surgical anatomy as it changes. This enables sustained precision in registration with pre- operative imaging throughout the procedure by updating the anatomical model as surgery progresses. We will achieve this by 1) using advances in computer vision to develop improved methods for continuous direct registration between the endoscope and the CT; and 2) developing methods that continuously compute the geometry of observed surfaces and update the preoperative CT with changes in that geometry. The technology developed in this proposal thus enables the following: 1) precise surgical navigation using the endoscope throughout the procedure; 2) surgical decision-making without the need for additional intra-operative imaging; and 3) new tools to reconstruct surgical anatomy and quantitatively evaluate the extent of surgery.