ABSTRACT Tumor infiltrating lymphocytes (TILs) are an important component of the immune cells that reside in the tumor microenvironment (TME). The type and number of TILs in the TME have an impact on overall survival and are an indicator of response to immunotherapy. Despite their importance as an indicator of a patient’s immune response to cancer, there are multiple challenges for analyzing TILS from large population data sets involving thousands of samples. There is a lack of methods that can automate an analysis of histopathologic images for different features such as the spatial distribution of TILs, their topological interactions with their neighboring cells in the TME and their association with specific clinical outcomes. Even more challenging is integrating TIL metrics with cancer genomic data. Most other methods provide qualitive metrics of TILs and frequently rely on manual inspection from pathologists – this approach lacks scalability and is subject to observer bias. To address these challenges, we developed a computational framework that uses a deep learning model to identify multiple cell types from histopathology images. The major innovation of our approach is molecular label transferring that annotates tens of thousands of small areas extracted from histopathology images without manual inspections. This approach is highly accurate, efficient, scalable and readily automated for the analysis of millions of images. The objective of this project is to address a key challenge in the application of deep learning to histopathological image: large number of labeled images as training data set. We have three specific aims to 1) identify spatial quantification of TILs from over 10,000 histopathological images from the Cancer Genome Atlas Project; 2) correlate TIL metrics with clonal tumor mutation burden (TMB); 3) determine association of TILs with immune checkpoint blockade responses. This research is significant because our approach enables for a comprehensive characterization of TILs from histopathological images at cellular level, using data that is commonly accessible in clinical settings and can be readily integrated with cancer genomic data.