Abstract Alzheimer's Disease (AD) is characterized by the presence and distribution of intracellular neurofibrillary tangles tau and extracellular amyloid-β. Tau pathology is deeply associated with cognitive decline in AD in a region-specific manner. It will be highly valuable to predict how tau burden would spread in the future given a baseline tau PET measurement. Such approach can be used to predict how region-wise tau burden changes versus age and, therefore, stratify potential candidates for AD therapy based on their “trajectory” of progression. It can also be used to determine how different a tau PET distribution is after a therapy from the predicted distribution if no therapy is applied. The recent availability of longitudinal tau PET data provides us a unique opportunity for technology development: to model tau-propagation using state-of-the-art deep learning methods. However, current available longitudinal tau PET datasets are relatively small, therefore, insufficient to train a deep neural network. To solve this problem, we propose a novel approach that incorporates a simple mathematical model in the training of the deep neural network. We first develop a simple network diffusion model that fits part of the available longitudinal tau PET data. We then generate a very large number of longitudinal tau PET datasets using the fitted model to pretrain a U-net-like autoencoder deep neural network. Finally, we further train the neural network by freezing all the parameters except those directly associated with the bottleneck layer of the neural network. This approach makes it possible to model tau propagation directly from measured longitudinal tau PET data while avoid overfitting caused by insufficient training data.