Emotion recognition is an important aspect of social interaction. Deficits in emotion recognition have been tied to poor social competence, interpersonal functioning, and communication along with a reduced quality of life and inappropriate social behavior. Recent experimental evidence suggests that emotion recognition ability across modalities seems to change in an age-dependent way, with deficits specifically noted for negative emotion recognition. Therefore, the primary purpose of this study is to assess emotion recognition ability in both the visual and auditory modalities in healthy elderly patients versus healthy young patients to test the hypothesis that deficits in emotion recognition ability are age-related and increase with age across visual and auditory modalities. The secondary purpose of this study is to assess valence-specific emotion recognition ability to test the hypothesis that increased deficits for elderly individuals as compared to young healthy subjects occur for negative emotion recognition specifically. We examined the ability to recognize emotion in the auditory and visual modality in two groups of patients: healthy elderly subjects and healthy young subjects. The Montreal Affective Voices task (MAV), the Distorted Tunes task (DTT), the Awareness of Social Inference test (TASIT), and the Baron-Cohen Mind in the Eyes Test (EYES) were used to gauge emotion recognition ability. For the visual tasks (TASIT and EYES), the elderly subjects and young subjects did not have significant differences in performance. In addition, a significant difference was also not found for the DTT, indicating that both populations had intact lower-level auditory processing abilities. Significant differences were found between the two populations on the MAV task in both baseline accuracy and valence-specific performance. These findings suggest that age-related deficits in emotion recognition may be specific to the recognition of auditory emotion.