TL;DR: In this work we propose to use weakly-supervised multi-task computational pathology to aid the differential diagnosis for cancers of unknown primary (CUP). CUPs represent 1-3% of all cancers and have poor prognosis because modern cancer treatment is specific to the primary. We present TOAD (Tumor Origin Assessment via Deep-learning) for predicting the primary origin of these tumors from H&E images without using immunohistochemistry, molecular testing or clinical correlation. Our model is trained on 17,486 gigapixel diagnostic whole slide images (WSIs) from 18 different primary cancer origins and tested on an internal set of 4,932 (WSIs) and an external set of 662 WSIs from 200+ institutions. Furthermore, we curated a large multi-institutional dataset of 717 CUP cases originiating in 150+ different medical centers and validated our model against a subset of 290 cases for which a primary differential was assigned based on evidence from extensive IHC testing, radiologic and/or clinical correlation.