Objective: Significant interscorer variability is found in manual scoring of arousals in polysomnographic recordings (PSGs). We propose a fully automatic method, the Multimodal Arousal Detector (MAD), for detecting arousals. Methods: A deep neural network was trained on 2,889 PSGs to detect cortical arousals and wakefulness in 1-second intervals. Furthermore, the relationship between MAD-predicted labels on PSGs and next day mean sleep latency (MSL) on a multiple sleep latency test (MSLT), a reflection of daytime sleepiness, was analyzed in 1447 MSLT instances in 873 subjects. Results: In a dataset of 1,026 PSGs, the MAD achieved an F1 score of 0.76 for arousal detection, while wakefulness was predicted with an accuracy of 0.95. In 60 PSGs scored by nine expert technicians, the MAD performed comparable to four and significantly outperformed five expert technicians for arousal detection. After controlling for known covariates, a doubling of the arousal index was associated with an average decrease in MSL of 40 seconds (p = 0.0075). Conclusions: The MAD performed better or comparable to human expert scorers. The MAD-predicted arousals were shown to be significant predictors of MSL. Significance: This study validates a fully automatic method for scoring arousals in PSGs.