Background: Treatment of obstructive sleep apnea is crucial for long term health and reduced economic burden. For those considered for surgery, drug-induced sleep endoscopy (DISE) is a method to characterize location and pattern of sleep-related upper airway collapse. According to the VOTE classification system, four upper airway sites of collapse are characterized: velum (V), oropharynx (O), tongue (T), and epiglottis (E). The degree of obstruction per site is classified as 0 (no obstruction), 1 (partial obstruction), or 2 (complete obstruction). Here we propose a deep learning approach for automatic scoring of VOTE obstruction degrees from DISE videos. Methods: We included 281 DISE videos with varying durations (6 s–16 min) from two sleep clinics: Copenhagen University Hospital and Stanford University Hospital. Examinations were split into 5-s clips, each receiving annotations of 0, 1, 2, or X (site not visible) for each site (V, O, T, and E), which was used to train a deep learning model. Predicted VOTE obstruction degrees per examination was obtained by taking the highest predicted degree per site across 5-s clips, which was evaluated against VOTE degrees annotated by surgeons. Results: Mean F1 score of 70% was obtained across all DISE examinations (V: 85%, O: 72%, T: 57%, E: 65%). For each site, sensitivity was highest for degree 2 and lowest for degree 0. No bias in performance was observed between videos from different clinicians/hospitals. Conclusions: This study demonstrates that automating scoring of DISE examinations show high validity and feasibility in degree of upper airway collapse.