Proceedings of the 7th International Conference on Music Perception and Cognition, Sydney, 2002 C. Stevens, D. Burnham, G. McPherson, E. Schubert, J. Renwick (Eds.). Adelaide: Causal Productions.
PERCEIVING EMOTION IN EXPRESSIVE PIANO PERFORMANCE: A FUNCTIONAL MRI STUDY
Dinesh G. Nair , Edward W. Large , Fred Steinberg and J A Scott Kelso

1 1,2 1

Center for Complex Systems and Brain Sciences, Florida Atlantic University, Boca Raton, Florida, USA.
University MRI of Boca Raton, Boca Raton, Florida, USA.
appoggiaturas and relatively sudden changes in harmonies evoked shivers. Krumhansl [7] observed that sad music resulted in increased systolic, diastolic and mean arterial pressures and decreased heart rate, skin conductance and finger temperature. But these physiological responses tell us neither about the nature of the processes underlying emotional experience nor their relation to the piece of music. The neural correlates of musical processing have been studied widely in the last decade and researchers have identified brain areas involved in the detection of pitch, contour, rhythm, meter and other structural aspects of music [10, 17]. Fewer studies have investigated neural correlates of emotional responses to music. Peretz et. al., [14] studied a patient with amusia (but without aphasia) who exhibited normal emotional judgment for a piece of music but had gravely impaired music processing abilities, and suggested the existence of separate neural pathways for emotional interpretation compared to structural interpretation of music. Using Positron Emission Tomography (PET) Blood et. al., [2] showed that when subjects listened to musical passages which varied systematically in the degree of dissonance, cerebral blood flow changes in the right parahippocampal gyrus and precuneus regions correlated with increasing dissonance while activity in the orbitofrontal, subcallosal cingulate and the frontal polar cortex correlated with decreasing dissonance. These were distinct from the areas involved in the analysis of structural components of music. Performers use various cues to convey emotion and meaning to listeners, collectively these are called performance expression. In piano performance, the cues are limited mainly to fast timescale fluctuations in timing (rubato and articulation), and intensity (dynamics). Similar fluctuations are also observed in speech communication and they are known to communicate many types of information to listeners. The ways in which performance timing and intensity variations communicate musical structure (e.g. phrasing, meter) has been extensively studied [12] and has even been modeled in some detail [9]. Yet we know few details about how listeners perceive affect in music performance. In this functional Magnetic Resonance Imaging (fMRI) study, participants listened to two versions of the same piece, one performed by a highly trained musician, and the other generated by computer to conform as precisely as possible to the notated composition. Thus, the two listening conditions were matched for melody, harmony, tonality, and rhythm. They differed only along performance parameters used by pianists to communicate with listeners: dynamics, articulation and rubato. We address the following questions. Are different brain areas activated when listening to expressive versus mechanical performances?

ABSTRACT

We aimed to identify brain areas involved in responding to affect communicated by expressive piano performance. Our subjects listened to two versions of Chopins Etude in E major, Opus 10, No. 3. The first version was an expressive performance, recorded by a highly trained musician on a computer-monitored piano. Our control was a computergenerated, mechanical performance of the same composition. Data analysis revealed differential brain activation in the two listening conditions. The expressive performance elicited greater activation in anterior cingulate, right temporal pole, right inferior frontal gyri, inferior parietal lobe and superior temporal gyri, areas that have been associated with emotion, attention and speech perception. The mechanical performance elicited greater activation in cerebellum, parahippocampal gyrus, supplementary motor area and dorsolateral prefrontal cortex, areas primarily involved in motor and sequencing tasks. Our results confirm that expressive music performance communicates affect beyond the melody, harmony, tonality, and rhythm of the notated composition. Our observations also suggest that the perception of emotion in music shares neural resources with the perception of emotion in speech, and that these pathways may be different from those recruited during other types of emotional experience.

1. INTRODUCTION

Listening to music involves perceiving sound stimuli, grouping them into patterns and relating these patterns to one another. But understanding musical expression calls for processes by which the listener experiences sound patterns as feelings and emotions. Music is believed to evoke a wide range of affective states in the absence of external associations [15]. Listeners, whether musically trained or not, are in general able to name the emotion that a musical excerpt was intended to convey, even across cultures [1]. However, the study of emotions conveyed by musical excerpts is susceptible to a number of problems. First, because music flows through time, it is difficult to pinpoint musical processes that evoke particular affective responses. Second, although some music is meant to convey specific emotions, a great deal of music is not intended to convey stereotypical emotions at all [11]. To paraphrase Kraut, listeners rarely experience envy, indignation, love, or fear when listening to uptempo Ornette Coleman performances [6]. Nevertheless, such musical experiences can arouse intense affective responses. Music is also known to evoke physiological responses and certain structural properties of music have even been linked to specific physiological responses. For instance, Sloboda [15] showed that tears were evoked by melodic

ISBN 39 6

ICMPC7, Sydney, Australia, July 2002
Can we draw inferences from our observations about how musical performance conveys emotion and meaning? Does communication of affect in music involve the same brain areas as other types of emotional responses, or is musical communication special in some way?
2. METHOD 2.1 Participants
Our listeners were four musicians (mean performance experience of 31.5 years; range: 25-40 yrs). Informed consent was obtained from all subjects after explaining to them the nature of the experiment. Subjects filled out a questionnaire after the experiment in which they responded to questions about their musical experience and their familiarity with the piece of music.

2.2 Stimuli

We used Chopins Etude in E major, Opus 10, No. 3 as our stimulus. This piece was performed by a senior piano major on a Kawai CA 950 digital piano, and recorded into Studio Vision running on a Macintosh G3 computer (Mac OS 9.0.4). To conform to the block design of the functional MRI paradigm, the performance was divided into six 30-45 second listening blocks. Blocks were chosen to conform to musical sections or subsections to cause minimal interruption to the natural flow of the music. Next, a mechanical performance was synthesized on the computer by changing the onset time and duration of each note to precisely match that of the musical notation, the onset velocity (MIDI) of each note was set to 64, and pedal information was eliminated. The mechanical version was then divided into listening blocks, and each block was matched for mean tempo with the corresponding block of the expressive performance by time stretching or compression. Listening blocks were interspersed with 30-second blocks of silence. The stimuli were played back via MIDI, through the Kawai CA 950, and recorded on a Sony PCM 2500B digital tape recorder.
listen, 30-45 sec) during which subjects listened to music and 6 baseline (OFF, rest, 30 sec) periods in which subjects heard only the ambient machine noise. Subjects were instructed to close their eyes and carefully listen to the performance. The subjects head was supported by a comfortable foam mold and head movement was further minimized using foam padding and forehead restraining straps. Scanning started with the acquisition of full head, 3D SPGR (spoiled gradient) anatomical images, with the following imaging parameters: Field of view (FOV) of 26 cm, frequency-phase matrix size = 256 x 256, repetition time (TR) = 34ms, echo time (TE) = 5 ms, flip angle (FA) 45, slice thickness 2mm, and one excitation (NEX) per phase encoding step. For each subject, T2*-weighted gradient echo, echo planar multi-slice datasets were acquired during ON and OFF periods, with a TR of 3 sec, TE 60 ms and FA = 90 (20 axial slices, matrix = 64 x 64, FOV = 24 cm, slice thickness = 5 mm and inter-slice gap = 2.5 mm). Thus the voxel size was 3.75 x 3.75 x 7.5 mm. High-resolution background images (same 20 slices, matrix = 256 x 256, NEX = 2) were also acquired to overlay the functional data. During the structural scans, subjects listened to a different piece of music recorded using the same settings, so that a comfortable loudness level for the stimulus was achieved. This also accustomed them to the process of listening to music in an MRI environment.

2.5 Data Analysis

The software used for analysis was AFNI (Analysis of Functional NeuroImages, Medical College of Wisconsin), [4]. We first performed movement correction of the functional datasets by using the Fourier method in AFNI. The raw time series were low pass filtered (cut off = 0.07 Hz) and spatially filtered using a Gaussian kernel (FWHM = 6mm) to enhance signal to noise ratio. Alternating periods of baseline and listening-related activation were modeled using boxcar reference waves shifted by 3, 6 or 9 seconds respectively to account for the hemodynamic response delay. This delay was determined by examining the raw time series data. Regions of task-related activity were determined by cross correlation of the image time series with the reference waveforms. The first stage of analysis used a thresholding procedure in which voxels with correlation coefficients greater than or equal to a threshold of 0.5 were identified and retained for further analysis. The correlation values were then converted to z-scores for all task conditions and all subjects. The resulting data were transformed into the Talairach and Tournoux stereotaxic space [16] for comparison across subjects. The mean intensity of activation across all subjects in the expressive performance was compared with that in the mechanical performance, to look for differences in brain activation across the two tasks. The significance of these differences was determined by using a paired t-test (p<0.05). In order to correct for multiple comparisons, we used probability thresholding in combination with cluster size thresholding. Only those voxels above p<0.05 (corresponding to t = 3.16) within a radial distance of 2mm from an active voxel and those that formed a volume of at least 1050 l (10 times the volume of one original voxel) were labeled as an active cluster. Images were created by mapping voxel t-values to colors using a scale from red (minimum) to yellow (maximum) when

2.3 Equipment

Whole brain fMRI data acquisition was carried out using a 1.5 Tesla Signa scanner (General Electric Medical Systems, Milwaukee, USA). The stimulus was played to the subjects from the digital tape through non-magnetic tubes and headphones (Avotec Inc). Headphones were custom-modified to deliver sound directly into the external auditory canal, by attaching soft-tipped earplugs of a Littmann Cardiology III stethoscope to a thick plastic tube that was shaped to precisely match the shape of the stethoscope, and then inserted into the sound-protective Avotec headphone shells. Sound barriers (Sonex, 1 inch thick, mean 30 dB attenuation within the frequency range of our performance) were used to insulate the auditory junction box and the head coil of the magnet from scanner noise.

2.4 Procedure

Each condition (performed and notated) lasted for 6 minutes and 33 seconds, and was comprised of 6 periods of activation (ON,

expressive > mechanical and blue (minimum) to cyan (maximum) when mechanical > expressive.
3. RESULTS AND DISCUSSION
The results of the paired t-test between the two conditions (expressive and mechanical) are shown in Figures 1 and 2. When listening to the mechanical performance, subjects showed stronger activation of right parahippocampal gyrus (PHG; Fig. 1a), and ventral posterior cingulate gyrus (not shown) during the mechanical performance. A previous PET study showed that blood flow to PHG was positively correlated with increasingly unpleasant musical stimuli [2]. Indeed, our subjects reported finding the mechanical performance somewhat unpleasant to listen to. Further, ventral posterior cingulate has been implicated in the processing of emotion, and its activity is known to correlate with increasingly painful stimuli. It may also be recalled here that extensive connections exist between the parahippocampal gyrus and the cingulate cortex, both being components of the limbic system and involved in the Papez circuit [5].
rhythm and meter [10, 17]. Thus, this observation was somewhat surprising since these areas were expected to be equally active during both listening conditions. We return to this observation momentarily.
Figure 2. Comparison of brain activation (t-test; p<0.05) between the two tasks revealed increased activation in bilateral superior temporal gyri BA 22 and bilateral transverse temporal gyri - BA 41/42 (white arrows, panel a), right middle temporal gyrus, right inferior frontal gyrus - BA 44 (blue oval panel a), right supramarginal gyrus and right angular gyrus BA 40, 39 (red voxels, panel b), right cingulate gyrus - BA 24, 32 (rectangle, panel c) right precentral gyrus (blue oval, panel c) and right inferior parietal lobule (white circle, panel c). Bilateral superior and medial frontal gyri (BA 9; blue voxels, panel b) were more active when subjects listened to the mechanical version. Panels a, b and c correspond to slices at z coordinates 10, 34 and 46 (in the vertical axis) of the Talairach coordinate system. R right; L left. Higher intensity of activation was seen in the right anterior cingulate cortex (ACC, BA 24 & 32, Fig. 2c) during the expressive performance. ACC has been implicated in a variety of functions including emotion, attention, novelty and error detection [3, 5]. Increased activation in ACC may reflect affective or emotional responses of listeners to the expressive performance. This would be consistent with our additional observation of increased activity in the temporal pole (BA 38, Fig. 1b), which also forms part of the limbic system and has functional connectivity with ACC [5]. Additionally, it is likely that expressive timing and intensity variations resulted in increased levels of attention, thus recruiting neurons in the ACC. This interpretation is consistent with our observation of increased activity in auditory areas, which may also play a role in affective processing. Finally, in the expressive listening condition, higher intensity activation was observed in the right inferior frontal gyrus (BA 44; Fig. 2a), right supramarginal gyrus (BA 40), right angular gyrus (BA 39; Fig. 2b), right inferior parietal lobule (Fig. 2c) and right frontal operculum (Fig. 1a). These areas play a variety of roles in speech and language processing. For example, it has been shown that patients with lesions in the right inferior parietal lobe fail to appreciate aspects of a verbal message that are conveyed by prosodic cues [5]. Activation of these areas while listening to expressive music performance implies sharing of neural resources that are important in linguistic function, including the processing of both prosody and semantics. This observation is also in agreement with previous neurological evidence showing

Figure 1. Increased intensity of activation was seen in the right parahippocampal gyrus (yellow oval) and the right cerebellum (white arrows) while listening to the mechanical performance. The red voxels in the left panel represent inferior frontal gyrus (BA 11, BA 47 or frontal operculum) and the yellow voxels in the right panel represent left temporal pole (BA 38), both of which showed higher intensity of activation when subjects listened to the expressive performance. R - right; L left. Increased intensity of activation was also observed in dorsolateral prefrontal areas (Brodmanns area [BA] 9, Fig. 2b), cerebellum (Fig. 1, arrows), and SMA (not shown) during the mechanical listening task. The mechanical performance had a strong, predictable rhythm, thus activation in these brain areas may have been due to some form of mental beat-following behavior. This could have recruited a loop involving the STG, limbic, and dorsolateral prefrontal areas, with projections to the cerebellum. Direct connections between these areas are known to exist in the human brain [5]. During the expressive listening condition, higher intensity of activation was observed bilaterally in both the transverse temporal gyri (BA 41 & 42) and the superior temporal gyri (BA 22; Fig. 2a), which included parts of primary, secondary, and associative auditory cortices. Earlier studies have provided evidence for the role of these areas in processing pitch, contour,
that the processing of music-like sound patterns involves the same neural resources as the processing of prosodic patterns in speech [13].
correlate with activity in paralimbic brain regions, Nat. Neurosci., Vol 2 (4): 382-387, April 1999. 3. Carter, C.S., Braver, T.S., Barch, D.M., Botvinick, M.M., Noll, D. and Cohen, J.D. Anterior Cingulate cortex, Error detection and online monitoring of performance, Science, Vol 280: 747-749, May 1998. 4. Cox, R.W., AFNI: software for analysis and visualization of functional magnetic resonance neuroimages, Comput. Biomed. Res., Vol 29: 162-173, 1996. 5. Kandel, E.R., Schwartz, J.H. and Jessell, T.M., Principles of Neural science, Appleton and Lange, Norwalk, Connecticut, 1991. 6. Kraut, R. On the possibility of a determinate semantics for music, In M. R. S. Holleran (Eds.) Cognitive Bases of Musical Communication, Washington, DC: APA, 1992. 7. Krumhansl, C.L., An exploratory study of musical emotions and psychophysiology, Can. J. Exp. Psych., Vol 51(4): 336-352, 1997. 8. Large, E.W., and Jones, M. R. The dynamics of attending: How we track time-varying events, Psych. Rev., Vol 106 (1): 119-159, 1999. 9. Large, E.W., and Palmer, C., Perceiving temporal regularity in music, Cogn. Sci., Vol 26: 1-37, 2002. 10. Liegeois-Chauvel, C., Peretz, I., Babai, M., Laguitton, V. and Chauvel, P., Contribution of different cortical areas in the temporal lobe to music processing, Brain, Vol 121, 1853-1867, 1998. 11. Meyer, L.B., Emotion and meaning in music, Chicago: University of Chicago Press, 1956. 12. Palmer, C., Music performance, Ann. Rev. Psych., Vol 48, 115-138, 1997. 13. Patel, A., Peretz, I., Tramo, M., and Labrecque, R. Processing prosodic and musical patterns: A neuropsychological investigation, Brain Lang., Vol 61: 123-144, 1998. 14. Peretz, I., Gagnon, L. and Bouchard, B; Music and emotion: perceptual determinants, immediacy and isolation after brain damage, Cognition, Vol 68: 111141, 1998. 15. Sloboda, J.A., Music structure and emotional response: some empirical findings, Psychol. Music, Vol 19: 110120, 1991. 16. Talairach, J., Tournoux, P., Co-planar Stereotaxic atlas of the brain, New York: Thieme, 1988. 17. Zatorre, R.J., Evans, A.C., and Meyer, E., Neural mechanisms underlying melodic perception and memory for pitch, J Neurosci,. Vol 14 (4): 1908-1919, 1994.

4. CONCLUSION

We observed differential brain activation depending upon whether participants listened to a mechanical or expressive performance of the same musical composition. The simple rhythm of the mechanical performance preferentially activated regions involved in timing and movement planning. The microstructure of the expressive performance, on the other hand, recruited an intricate neural network that functionally links bilateral auditory and auditory association areas with limbic/paralimbic (cingulate, temporal pole) and speech processing areas (inferior frontal, frontal operculum, inferior parietal lobe). Interestingly, although we observed activation of certain emotion areas such as the cingulate, other important limbic areas such as the amygdala did not show significant activity. Although preliminary, this finding raises the possibility that music communicates affect in a way that is distinct from many other emotional experiences. It would also offer indirect support for the theory that musical experiences tend to produce non-specific affective arousal, which may or may not be interpreted as emotion, rather than directly communicating specific identifiable emotions [11]. Meyers approach holds that violation of expectancy forms the basis for the communication of emotion and meaning in music [11]. If rhythmic expectancies are violated by expressive timing deviations [8, 9] this would provide a theoretical basis for increased emotional response to the expressive performance. It would also explain the apparent increase in attention, since violation of temporal expectancy would result in attentional capture [8]. In addition, the extensive activation of neural areas previously associated with prosody suggests that similar processes are at work in communicating affect in speech. An expressive music performance is more than a sonic realization of a musical score. It is widely understood that expressive performance communicates aspects of musical structure [9, 12]. In this study we observed, for the first time, recruitment of limbic/paralimbic areas in response to music performance, implying communication of affect. We also found recruitment of neural structures related to components of attention and speech processing, which may help us to better understand this process of musical communication between performer and listener.

5. ACKNOWLEDGEMENTS

This research was supported by NSF grant BCS-0094229 awarded to EWL and NIMH grant MH 42900.

6. REFERENCES

1. Balkwill, L-L., Thompson, W.F., A cross-cultural investigation of the perception of emotion in music: Psychophysical and cultural cues, Music Percept., Vol 17(1): 43-64, Fall 1999. 2. Blood, A.J., Zatorre, R.J., Bermudez, P. and Evans, A.C. Emotional responses to pleasant and unpleasant music

Dynamic Emotional and Neural Responses to Music Depend on Performance Expression and Listener Experience
Heather Chapin1, Kelly Jantzen2, J. A. Scott Kelso1,3, Fred Steinberg4, Edward Large1*
1 Center for Complex Systems and Brain Sciences, Florida Atlantic University, Boca Raton, Florida, United States of America, 2 Department of Psychology, Western Washington University, Bellingham, Washington, United States of America, 3 Intelligent Systems Research Centre, University of Ulster, Magee Campus, Derry, North Ireland, 4 University MRI of Boca Raton, Boca Raton, Florida, United States of America

Abstract

Apart from its natural relevance to cognition, music provides a window into the intimate relationships between production, perception, experience, and emotion. Here, emotional responses and neural activity were observed as they evolved together with stimulus parameters over several minutes. Participants listened to a skilled music performance that included the natural fluctuations in timing and sound intensity that musicians use to evoke emotional responses. A mechanical performance of the same piece served as a control. Before and after fMRI scanning, participants reported real-time emotional responses on a 2-dimensional rating scale (arousal and valence) as they listened to each performance. During fMRI scanning, participants listened without reporting emotional responses. Limbic and paralimbic brain areas responded to the expressive dynamics of human music performance, and both emotion and reward related activations during music listening were dependent upon musical training. Moreover, dynamic changes in timing predicted ratings of emotional arousal, as well as real-time changes in neural activity. BOLD signal changes correlated with expressive timing fluctuations in cortical and subcortical motor areas consistent with pulse perception, and in a network consistent with the human mirror neuron system. These findings show that expressive music performance evokes emotion and reward related neural activations, and that musics affective impact on the brains of listeners is altered by musical training. Our observations are consistent with the idea that music performance evokes an emotional response through a form of empathy that is based, at least in part, on the perception of movement and on violations of pulse-based temporal expectancies.
Citation: Chapin H, Jantzen K, Scott Kelso JA, Steinberg F, Large E (2010) Dynamic Emotional and Neural Responses to Music Depend on Performance Expression and Listener Experience. PLoS ONE 5(12): e13812. doi:10.1371/journal.pone.0013812 Editor: Antoni Rodriguez-Fornells, University of Barcelona, Spain Received March 5, 2010; Accepted October 12, 2010; Published December 16, 2010 Copyright: 2010 Chapin et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: This work was supported by National Science Foundation career award BCS-0094229 and a Fulbright visiting research chair awarded to E.W. Large, and by National Institute of Mental Health grant T151063 and the Pierre de Fermat Chair to J.A.S. Kelso. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: The authors have declared that no competing interests exist. * E-mail: large@ccs.fau.edu

Introduction

Although much research has focused on the ability of humans to detect and recognize emotional stimuli [13], we know less about affective responses that people experience in natural settings. Precise quantification of emotional reactions and their associated brain responses is challenging because emotional responses are often dynamic experiences that unfold on several timescales [4]. Emotional responses to identical stimuli vary among people, making it difficult to link specific stimulus parameters to individual responses. Moreover, measuring and imaging emotional responses simultaneously is problematic because explicit instructions to monitor and report emotional reactions can interfere with the affective responses one is attempting to measure [5]. Music has a number of key properties that make it an excellent model system for the study of emotion, addressing some of these issues. It is an ecologically valid stimulus that is used daily across cultures to communicate and modulate emotion. It is a dynamic stimulus whose parameters evolve over timescales ranging from fractions of a second to minutes. The existence of populations with and without music performance experience allows for the opportunity

PLoS ONE | www.plosone.org 1
to explore the role of learning and experience in modifying the relationship between a stimulus and its associated emotional response. Here, we investigate the dynamics of emotional and neural responding to a natural music performance, focusing on the role of stimulus parameter dynamics and listener experience. Previous neuroimaging work has revealed the involvement of several brain areas in emotional responses to music, focusing on contrasting musical attributes such as consonant/dissonant, pleasant/unpleasant, and happy/sad. Not surprisingly, areas associated with emotion processing and reward in general are also involved in emotional responding to music. Parahippocampus and precuneus activity were found to increase in response to increasing dissonance of short chord sequences [6], whereas increasing consonance was associated with activation of orbitofrontal and frontopolor cortices and subcallosal cingulate, a region implicated in emotion processing [7,8]. Similarly, Koelsch, Fritz et al. [9] found unpleasant, compared to pleasant, excerpts activated parahippocampal gyrus as well as amygdala, and the temporal poles. Listening to pleasant, relative to unpleasant, music was associated with activation of insula, inferior frontal gyrus (IFG, including Brodmann Area (BA) 44), and the ventral striatum, a key
December 2010 | Volume 5 | Issue 12 | e13812
Expressive Musical Dynamics
structure in reward and addiction circuits [1012]. Levitin and Menon [13] found blood oxygen level dependent (BOLD) response increases in non-musicians for normal versus temporally scrambled musical excerpts in IFG (BA 47), anterior cingulate, nucleus accumbens (part of the ventral striatum), brainstem, and posterior cerebellar vermis. These participants also showed increased BOLD responses in reward related brain areas for the normal excerpts. Another group of non-musicians rated the normal versions as more pleasant than the scrambled versions [14]. Similar emotion and reward related networks were found to be associated with increasing pleasurable chill ratings in response to listening to self-selected musical excerpts [15] and while listening to music rated as happy (versus sad) [16]. Thus, current research has identified emotion related limbic and paralimbic activations (e.g., amygdala, subcallosal gyrus, ventral anterior cingulate, and parahippocampal gyrus) and reward related activations (in ventral striatum) associated with affective responses to music. However, these approaches did not tackle the issue of how specific stimulus parameters may give rise to emotional responses. Behavioral studies of emotion in music have focused on the role of specific musical parameters in communicating emotion [17]. In one approach, performers are asked to record short musical excerpts in a way that will convey basic emotions, such as anger, fear, or joy. Listeners then attempt to name the basic emotion that the performance was intended to convey. Regardless of musical training or cultural background, people are generally able to name the intended emotion, providing evidence that the expression of basic emotions (happiness, sadness, and fear) in music can be recognized universally [18,19]. Moreover, listener judgments of intended emotion have been linked to specific musical features, including tempo, articulation, intensity, and timbre [2023]. These studies have mostly used short excerpts and required participants to express their responses using single word labels. Thus, while listeners recognize intended emotions, it is possible that they do not actually experience emotional responses to music in such tasks [24]. Furthermore, short excerpts and discrete categories employed in many behavioral and neuroimaging studies do not capture dynamic aspects of musical emotion that unfold over larger time scales [2426]. In order to explore dynamic affective responses of listeners to entire pieces of music, Schubert [24,27,28] developed a continuous response paradigm in which listeners report perceived emotion in real-time in a 2-dimensional emotion space [28], with emotional valence and arousal as the orthogonal dimensions [29]. In one study, Schubert [27] used four compositions to capture a wide range of musical feature variations and instructed participants to report the emotion the music (is) trying to express. Musical variables such as melodic contour, tempo, loudness, texture, and timbral sharpness, were shown to predict emotion ratings. Interestingly, tempo and loudness accounted for over 60% of the variance between musical pieces along the emotional arousal dimension. Musicians use fluctuations in timing and sound intensity within a performance to express structural interpretations and intensify emotional communication [30]. Other studies have considered dynamic fluctuations in tempo and sound intensity within individual performances. Koelsch et al. [31] showed that emotion-related physiological and neural responses to unexpected chords were greater when the chords were played in an expressive context. Bhatara et al. [32] created versions of music performances in which changes in timing and intensity were parametrically manipulated, and asked participants to rate the emotional expressivity. Emotion judgments monotonically increased with performance variability, and timing changes were found to explain

PLoS ONE | www.plosone.org 2
more variance in reported emotional expressivity than sound intensity. Musically experienced listeners (with.6 yrs. of musical training) were more sensitive to performance expression than less experienced listeners. Sloboda and Lehmann [33] showed that in music performance, changes in tempo and sound intensity are correlated with one another, and with real-time ratings of emotional arousal. They also showed a systematic relationship between emotionality ratings, timing, and loudness when listeners rated their moment-to-moment level of perceived emotionality while listening to music performances. These observations about expressive performance lead to several important questions. First, we asked whether listening to an expressive musical performance compared to one that does not contain dynamic stimulus fluctuations would lead to limbic and paralimbic activations in areas such as amygdala, parahippocampus, ventral anterior cingulate, and subcallosal gyrus, and perhaps to reward related activation in ventral striatum. We were also interested in understanding the relationship between feature variations, emotional responses, and neural activations within a temporally fluctuating musical performance. Based on previous studies [27,32,33], we expected to observe that real-time ratings of emotional arousal would correlate with fluctuations in tempo and sound intensity. Next, to illuminate the neural mechanisms of emotional responding to musical performance, we compared temporal performance fluctuations and reported emotional arousal with fluctuations in the BOLD signal. We hypothesized that tempo fluctuations would lead to violations of temporal expectancies [25,34,35] based on the perceived pulse [3641], and that temporal expectancy violations would be associated with emotional responses. Activity in motor areas such as basal ganglia [42,43], pre-SMA [4244], SMA [4245], and premotor cortex (PMC) [4245] is present during rhythm perception, even in the absence of overt movement, and basal ganglia have been specifically linked to pulse perception [42,46,47]. Recently it has been shown that temporal unpredictability in the auditory domain is sufficient to produce amygdala activation in mice and humans [48]. Additionally, activity in IFG 47 has been linked to the perception of temporally coherent structure in music [13], and dorsal anterior cingulate cortex (dACC) has been associated with error detection in general [49] and could also be involved more specifically in temporal expectancy violations. Thus, it may be that activity in the motor areas related to rhythm and pulse perception, IFG 47, and dACC relate to temporal expectancy and violations of expectancy and that these violations may evoke emotion through activation of limbic areas such as the amygdala. The current experiment focused on how performance expression influences the dynamic emotional responses to a musical stimulus that unfolds over a period of minutes. An expressive music performance, recorded by a skilled pianist, with natural variations in timing and sound intensity, was used to evoke emotion, and a mechanical performance was used to control for compositional aspects of the stimulus [35,50], and for average values of tempo and sound intensity. To study emotional experience, deep listeners [51] were recruited. These participants reported listening to and enjoying classical music, but were not professional musicians and were not familiar with the piece used in the experiment. However, these non-expert listeners had varying degrees of musical experience and training, allowing us to address the role of moderate levels of musical experience (such as singing in a choir) in modulating emotional responses. Participants were asked to report their emotional responses to the music in realtime. Emotional responses were imaged separately to prevent selfreport from interfering with experienced affect. This procedure

PLoS ONE | www.plosone.org 3
Figure 1. 2-Dimensional emotion response space. Valence is represented on the horizontal dimension and arousal is represented on the vertical dimension. doi:10.1371/journal.pone.0013812.g001
negative emotions (like sadness or anger). The position of the cursor for all participants started at zero arousal, zero valence (bottom middle point in the response space). The software recorded cursor position automatically during music playback with an average sampling period of 135 ms. Participants performed this task immediately before entering the scanner and after scanning, but not during fMRI acquisition. Two behavioral sessions were conducted to test reliability of reported emotional responses. It was assumed that if participants reported emotional responses were reliable over time, similar emotional responses would be experienced in the scanner, allowing for correlations between behavioral and physiological data. Performances were presented in counterbalanced order across the two sessions. fMRI Recording. A custom Visual Basic 5 program running on a Dell Optiplex GX260 was used to play sound stimuli which were presented to participants using custom noise-attenuating headphones (Avotec, Inc). They were instructed to lie motionless in the scanner with eyes closed and listen attentively to the music without actively monitoring or reporting their emotional response. During the rest period, participants were instructed to rest quietly with eyes closed and wait for the music to begin again.
Magnetic Resonance Imaging
Changes in blood oxygenation (BOLD response) were measured using echo planar imaging on a 3.0 T Signa Scanner equipped with real time fMRI capabilities (General Electric Medical Systems, Milwaukee, WI). Echo-planar images were collected using a single shot, gradient-echo, echo planar pulse sequence (field of view (FOV) = 24 cm, echo time (TE) = 35 ms, flip angle
(FA) = 90u, in plane resolution = 64664). All images were collected using a sparse temporal sampling technique with a repetition time (TR) of 12 seconds. Adequate coverage of the brain was achieved by collecting thirty interleaved 4 mm axial slices with no spacing between (voxel size = 3.7563.7564 mm). Immediately following the functional imaging, high resolution anatomical spoiled gradient-recalled at steady state (SPGR) images (4 mm thick, no spacing, number of excitations = 2, TE = in phase, TR = 325 ms, FA = 90u, in plane resolution 2566256, bandwidth = 31.25) were collected at the same slice locations as the functional images. Acquisition. A sparse temporal sampling technique was used in the scanner to increase the signal response from baseline (which was silence) and to avoid nonlinear interaction of the scanner sound with the auditory stimulus [53]. There were a total of two trials. Within each trial, there was one minute of rest between the two stimuli and after the last stimulus presentation. One trial started with twelve seconds of rest, followed by the expressive performance and then the mechanical performance. The other trial started with six seconds of rest, followed by the mechanical and expressive performances. The variable amount of time in the first rest period enabled imaging of 36 unique time points over the two trials. Thus, when combined, the scans yielded an effective repetition time (TR) of 6 seconds (Figure 2). The total number of scans acquired for each participant over both trials was 96 (18 scans per performance per trial (36 scans total for each performance) +24 rest scans across trials). Trial order was randomized across participants. In summary, participants performed the real-time emotional rating task immediately prior

Figure 2. Piano roll notation and fMRI scan times. Piano roll notates Chopins Etude in E major, opus 10, no.3. doi:10.1371/journal.pone.0013812.g002
PLoS ONE | www.plosone.org
to scanning, then were scanned without reporting emotional responses for two trials of each performance, and finally performed the real-time emotional rating task immediately after scanning.

Data Analysis

Behavioral measures. Data analysis was performed using Matlab 7.2.0 (The MathWorks, Inc.) running on an Apple G5. First, the performance was matched to its score using a custom dynamic programming algorithm [54,55]. Chords were grouped by the same dynamic programming algorithm, and onset time of a chord was defined to be equal to the average of component note onset times. This procedure enabled the identification of timing fluctuations. Beat times were extracted as the times of performed events that matched notated events occurring on sixteenth-note level beats. Sixteenth-note level beats to which no event corresponded were interpolated using local tempo. Inter-beat intervals (IBIs, in seconds) were calculated by subtracting successive beat times, and a tempo curve in beats per minute (bpm) was constructed according to the formula (local tempo = 60/IBI). Each value in the tempo curve was divided by 4 because calculations had been made at the sixteenth note level (four sixteenth notes per quarter note; for details see [35]). One measure provided by MIDI is the velocity at which the key is struck, which is related to sound intensity [56]. A velocity curve, showing how sound intensity changed over the piece, was created by averaging the velocities of notes in chords. These procedures resulted in tempo and velocity curves that were sampled at event onset times, i.e. at unequal sampling intervals. Therefore, the tempo and velocity curves were resampled at equal intervals (10 ms; i.e. 100 Hz). Next, the frequency content of the tempo curve was measured using a fast Fourier transform, and the Nyquist frequency was determined to be approximately 0.5 Hz. Finally, to prevent spuriously high correlations with emotion rating time-series (discussed next) due to over-sampling, the time-series data were low-pass filtered and down-sampled at 2-second intervals (0.5 Hz). Arousal and valence time-series were decimated to 0.5 Hz to match the sampling rate of the tempo curve. Then they were correlated across trials at an optimal time lag to test for reliability. For participants whose ratings were reliable, means of first and second trial rating curves were calculated for use in subsequent analyses. Next, optimal lag times (12-second maximum) were calculated for correlations between the behavioral data and the tempo curve. Finally, mean arousal and valence ratings, each advanced by the optimal lag time, were correlated with the tempo curve of the expressive performance. fMRI. Data analysis and display were performed using AFNI [57,58] and image registration was conducted using FSL (Analysis Group, FMRIB, Oxford, UK), both running on an Apple G5. Functional data sets were corrected for motion, and mean scaling of intensity was globally normalized. To model the BOLD response to the presence of music, a hemodynamic response function (HRF) was convolved with a binary vector representing the off/on timing of each condition (performance = on, silence = off, with each performance modeled separately). Each participants head motion information, obtained during volume registration, was added to the baseline model to account for the variance due to head motion in all of the comparisons. A general linear approach, as implemented in AFNI, was used to determine the contribution of the model to the data from each voxel. Functional images were then coregistered and transformed into Talairach & Tournoux [59] coordinates. A separate second-level analysis was used to compare activations across participants. A mixed-design 2-way ANOVA

PLoS ONE | www.plosone.org 5
with the factors performance type (expressive vs. mechanical) and musical experience (experienced vs. inexperienced) was conducted on the beta weights from each voxel. To correct for multiple comparisons, a Monte Carlo simulation was conducted to determine the random distribution of voxel cluster sizes for a given threshold [60]. A two-tailed alpha level of p,.02 was achieved through the combination of a per voxel threshold of p,.05 and a cluster size of 10 contiguous voxels (640 mm). Additional analyses were performed to determine the contribution of dynamic fluctuations in stimulus tempo and reported emotional arousal to BOLD activity. The tempo curve and the time-advanced mean arousal ratings were filtered with a hemodynamic response function (HRF) and were resampled at six-second intervals to match the sampling rate of the fMRI data. HRF filtered emotion ratings were then normalized about 0 by dividing each time-series by its mean and subtracting 1. The HRF filtered tempo curve for the expressive performance and the HRF filtered arousal ratings of each participant were regressed against the BOLD time series. The expressive and mechanical performance models used in the first analysis were added to the baseline model to account for tonic changes in BOLD intensity associated with listening to the music. Head motion parameters were also included as part of the baseline model. Group analysis was conducted by submitting individual beta weights to between subject t-tests. Again, a Monte Carlo simulation was conducted to correct for multiple comparisons and an alpha level of p,0.02 was achieved with a per voxel threshold of p,0.02 and a cluster size of 12 contiguous voxels (768 mm).
Results Behavioral measures
For the expressive performance, the patterns of emotional arousal ratings were significantly positively correlated across the two trials for nineteen out of twenty-one participants (mean r =.62, SD =.23, p,.05). Emotional valence ratings were less reliable across trials, but still significantly positively correlated in eighteen out of twenty-one participants (mean r =.46, SD =.22, p,.05). Given the reliability of emotion ratings across trials, mean arousal and valence ratings were calculated across the two trials for each participant for comparison with dynamic measures of the music performance. Mean arousal and valence ratings were positively correlated in six participants, negatively correlated in nine participants, and uncorrelated in six participants. Therefore, arousal and valence ratings did not have a reliable relationship across subjects. See Table 1 for a summary of the behavioral results. Regression analysis revealed that BOLD filtered tempo and velocity (loudness) curves (as described in the Methods section) for the expressive performance were highly correlated (r =.87, p,.05). For this reason, and because previous findings suggest that tempo fluctuation accounts for more variance in emotionality ratings than loudness changes in performed music [32], subsequent analyses focused on tempo changes. Arousal ratings were significantly positively correlated with the tempo curve in nineteen out of twenty-one participants at their optimal time lag (mean of significant correlations r =.50, SD =.20, p,.05, mean lag 7.2 seconds). Figure 3 illustrates the relationship between one participants mean emotional arousal ratings and the tempo curve of the expressive performance. Linear regression was used to test the pairwise correlations between participants mean arousal rating time series. All but two (99.5%) of the pairs were significantly correlated at the p,0.05 level (mean r =.62). Thus, arousal measures were highly reliable across participants. Mean valence ratings were not as reliable across participants

Table 1. Behavioral correlations based on reported emotional responses, *p,.05.
Subject Exp1 Exp2 Exp3 Exp4 Exp5 Exp6 Exp7 Exp8 Exp9 Exp10 Inexp1 Inexp2 Inexp3 Inexp4 Inexp5 Inexp6 Inexp7 Inexp8 Inexp9 Inexp10 Inexp11 Mean St. Dev.
Arousal (Trial 1/Trial 2) 0.77 0.70 0.65 0.62 0.80 0.53 0.64 0.04 0.63 0.87 0.75 0.76 0.52 0.75 0.78 0.57 0.71 0.74 0.73 20.07 0.46 0.62 0.23 * * * * * * * * * * * * * * * * * * *
Valence (Trial 1/Trial 2) 0.30 0.50 0.10 0.41 0.65 0.48 0.64 0.51 0.47 0.50 0.83 0.80 0.16 0.59 0.68 0.01 0.63 0.52 0.35 0.20 0.35 0.46 0.22 * * * * * * * * * * * * * * * * * * *
Arousal Valence (Mean of Trials) 20.56 0.04 0.26 20.87 0.14 20.21 20.57 20.21 0.64 0.78 0.24 0.54 20.16 20.71 20.80 20.08 20.67 20.31 0.19 0.07 20.15 20.11 0.47 * * * * * * * * * * * * * * *
Arousal Tempo (Mean of Trials) 0.54 0.51 0.71 20.10 0.41 0.66 0.71 0.33 0.59 0.73 0.55 0.49 0.43 0.63 0.75 0.44 0.18 0.54 0.62 0.47 0.34 0.50 0.20 * * * * * * * * * * * * * * * * * * *
doi:10.1371/journal.pone.0013812.t001
(mean r =.32, 64.3% of pairs were significantly correlated at p,0.05). Emotional arousal correlations with the tempo curve of the expressive performance did not differ significantly between experienced and inexperienced participants (paired t(6) =.97, p =.18). Emotional valence ratings were significantly negatively correlated with the tempo curve of the expressive performance in sixteen out of twenty-one participants and significantly positively correlated in four out of twenty-one participants (mean r = 2.25, SD =.32, p,.05). Because of the stronger and more reliable relationship between tempo and arousal, arousal was
focused on in subsequent analyses. Additionally, only participants whose arousal ratings correlated significantly with tempo were included in further analyses (eliminating one experienced and one inexperienced participant). In summary, tempo fluctuation in the expressive performance predicted emotional arousal ratings for both experienced and inexperienced participants.
ANOVA. Table 2 and Figure 4 present summaries of the fMRI ANOVA results. A main effect of performance type
Figure 3. Correlation between tempo and mean arousal for the expressive performance for one participant. For this participant, the optimal lag time was 25.4 seconds and time shifted arousal ratings are shown. (r =.73, p,.0001) doi:10.1371/journal.pone.0013812.g003
Table 2. Brain activations (ANOVA results) showing a significant main effect of performance type and musical experience (F (1,24). 7.19, corrected p ,.02).

Volume (mm3)

REGION (cluster peak) Main Effect: Expressive vs. Mechanical R R R R R L R L L R L parahippocampus fusiform gyrus inferior parietal inferior frontal parahippocampus medial frontal (frontopolar) ventral anterior cingulate precuneus supramarginal (Decrease) medial frontal superior frontal

Cluster includes

F-values
bilateral culmen and vermis declive (19, 37)
18.94 10.96 10.73 9.63 10.25 9.18 8.16 8.3 210.34 8.39 8.3

24,32 7

superior parietal (7)
amygdala, hippocampus, subcallosal (34)

bilateral (24)

inferior parietal (40) 8 8
Main Effect: Experienced vs. Inexperienced R L L L ventral striatum parahippocampus (Decrease) ventral anterior cingulate ventral anterior cingulate lentiform, putamen, subcallosal hippocampus (10) subcallosal 7.91 28.16 13.82 15.04
doi:10.1371/journal.pone.0013812.t002
(expressive versus mechanical) was the result of an increased BOLD signal for the expressive performance in right posterior parahippocampal gyrus (extending into amygdala and hippocampus), fusiform gyrus, inferior parietal lobule (BA 40), IFG (BA 47), anterior parahippocampal gyrus, bilateral ventral anterior cingulate (predominately right lateralized), left medial prefrontal cortex (BA 10, frontopolor area), right dorsal medial prefrontal cortex (BA 8) and precuneus (BA 7). An increased BOLD response for the mechanical performance was observed only in the supramarginal gyrus (BA 40). A main effect of musical experience was found with greater BOLD activity for experienced compared to inexperienced listeners occurring in right ventral striatum (extending into lentiform nucleus, putamen, and subcallosal gyrus) and left ventral anterior cingulate. Inexperienced listeners had greater BOLD activity than experienced listeners in the left anterior parahippocampal/hippocampal gyrus (see Figure 4b). Several brain regions showed a significant interaction of performance type with musical experience (see Figure 5). In the left cerebellum (culmen), left IFG (BA 47), right inferior parietal lobe (BA 40), and right dorsal cingulate (near pre-SMA), experienced participants showed a greater percent signal change for the expressive than the mechanical performance whereas inexperienced listeners showed the opposite response. Tempo. BOLD signal changes in a number of areas showed significant positive correlations with the tempo curve of the expressive performance (see Table 3 and Figure 6). They included right lingual gyrus, PMC, left primary motor and somatosensory cortex (extending into inferior parietal lobe BA 40), right postcentral gyrus (BA 43, extending into primary motor, PMC, and inferior parietal lobe BA 40), bilateral dorsal lateral prefrontal cortex (right BA 9/10 extending into ventral anterior cingulate, left BA 10), right lentiform nucleus and putamen, left insula, BA 44, and right secondary auditory cortex (BA 21 and 22).

Figure 5. Peak voxels showing a significant interaction between performance type and experience. IPL = inferior parietal lobe. doi:10.1371/journal.pone.0013812.g005
pulse of the expressively timed musical performance, though not as accurately as when tapping the pulse of the mechanical performance. The perception of pulse is thought to underlie the development of temporal expectancies [36,41]. In this study, we observed that activation of motor-related areas, such as pre-SMA, SMA, PMC, primary motor, basal ganglia (lentiform nucleus, putamen), cerebellum, and thalamus correlated with the tempo of the expressive performance. Activation of these motor regions has previously been reported during rhythm perception [64,65]. Basal ganglia activation has been specifically linked to pulse perception [42,46,47] and has been shown to be involved in emotion
PLoS ONE | www.plosone.org 9
processing in general [7]. Additionally, in this study, dACC activity correlated with emotional arousal in the experienced participants. Thus, consistent with its involvement in error detection and correction [49], the dorsal portion of the anterior cingulate may play a role in the detection and correction of temporal expectancy violations in music. Moreover, activity in IFG (BA 47) was associated with listening to the expressive performance overall, and more so for experienced participants. Previous work has linked IFG 47 (along with insula activity) to the perception of temporal structure in music [13]. Along with the previously mentioned experiment conducted by Bhatara et al.
Table 3. Brain activations as a function of expressive tempo and reported emotional arousal, corrected p,0.02, df = 12.
REGION (cluster center) Tempo R lingual gyrus

T-values

cuneus (17, 18), bilateral cuneus (17), L culmen, vermis and declive, L posterior cingulate, L middle occipital, inferior occipital SFG (6), precentral (6), bilateral precentral (4), postcentral (3, 5), bilateral cingluate middle occipital (18, 19), inferior occipital, fusiform gyrus (19, 37), MTG (37) precentral (4, 6), inferior parietal, insula (13) precentral (4, 6), insula (13), inferior parietal middle frontal (10), ventral anterior cingulate pulvinar of thalamus insula (13), precentral (44) 13 middle frontal (10) (22)

R R L R R L L L R R R L

medial frontal, SMA lingual gyrus postcentral postcentral superior frontal insula precentral superior frontal superior temporal insula lentiform nucleus, putamen insula Emotional Arousal

Figure 6. Brain activations as a function of the tempo of the expressive performance. SFG = superior frontal gyrus, dPMC = dorsal PMC, SMA = supplementary motor area, S1 = primary somatosensory cortex, M1 = primary motor cortex, vPMC = ventral PMC, STS = superior sulcus. doi:10.1371/journal.pone.0013812.g006
It has been shown that those with musical training listen to music differently [72], exhibit different activation patterns during music perception [7376], show enhanced processing of affective vocal sounds [77], and even show differences in brain anatomy [78,79]. Also, expertise in a particular type of movement, such as dance, has been shown to alter the way mirror neurons respond during observation [80], demonstrating that experience is an important factor in determining whether perceptions will resonate with ones own motor repertoire. In the current study, mirror neuron regions were correlated with dynamic ratings of emotional arousal only in the experienced listeners, who also showed increased activation in emotion and reward-related areas. Thus, the pathways mediating emotional responses based on temporal fluctuations may differ with musical experience. It may be that all listeners perceive motion arising from changes in tempo through activation of mirror neuron and motor systems. However, it is possible that mirror neuron activity only influences emotional responses in listeners with explicit experience conveying emotion through music performance. Therefore, network interactions between the mirror neuron system, insula, and limbic system may be more readily engaged in those with musical experience. Such individuals have the experience of conveying emotion through music performance and may have a more developed mapping between musical structure, motor experience, and emotion. Increased sensitivity to expressive performance parameters may help explain why experienced participants showed enhanced
PLoS ONE | www.plosone.org 11
responses in areas related to emotion, memory, and reward. Overall, listening to the expressive performance increased activity in parahippocampal gyrus, hippocampus, medial prefrontal cortex (mPFC), and ventral anterior cingulate (vACC) for all participants. Experienced participants showed a greater response than inexperienced listeners in the ventral striatum and ventral anterior cingulate while listening to music in general. The ventral striatum (including nucleus accumbens) has been associated with the rewarding property of music [9,1416]. The vACC has been shown to be related to emotional processing in general [7,49] and more specifically in music [15,16]. The mPFC, specifically BA 10, has been implicated in emotion processing and attending to ones own emotional state [81]. Parahippocampal gyrus activity has been shown in previous studies investigating emotional responses to music [6,9,16] and has been linked to understanding social contextual cues [82]. These results suggest that expressive music performance activates emotion related structures in listeners and that even a moderate level of musical experience enhances this emotion associated activation and increases the rewarding aspect of music listening, perhaps through the engagement of the mirror neuron system and its interactions with the limbic system via the insula. Familiarity with a specific musical piece does not seem to be necessary to evoke this response. The journey from listening to feeling involves a dynamic interplay between several neural systems [83]. Auditory and attentional systems (which are both modulated by previous experience, expectancy, and initial emotional conditions) must

File S2 The Chopin mechanical performance maintains con-

Acknowledgments

We would like to thank Dinesh Nair for preliminary work which led to this study, Judith Becker for advice on listener screening, and Petr Janata for valuable comments on the analyses. We would also like to thank Steve Sedita for his help with data acquisition and analysis, Marc Velasco, Ajay Pillai, and Brian McFarland for their technical assistance, and Summer Rankin for her comments on an earlier version of this manuscript.
stant tempo and sound intensity throughout the piece (equal to the mean tempo and sound intensity of the expressive performance). Found at: doi:10.1371/journal.pone.0013812.s002 (3.46 MB MP3)
File S3 Animation of the real-time changes in neural activity that were time-locked to the tempo fluctuations in a musical performance of Frederic Chopins Etude in E major, Op.10, No. 3. This animation includes a subset of the brain regions that exhibited time-locked activity. Shown are cortical and subcortical motor areas thought to be involved in pulse perception, and a network of areas consistent with the human mirror neuron

Author Contributions

Conceived and designed the experiments: HC EL. Performed the experiments: HC. Analyzed the data: HC KJJ EL. Contributed reagents/materials/analysis tools: JASK FS EL. Wrote the paper: HC KJJ JASK EL.

References

1. Ekman P (1992) An argument for basic emotions. Cognition and Emotion 6: 169200. 2. Ekman P, Friesen WV, Ellsworth P (1972) Emotion in the human face: Guidelines for research and an integration of findings. New York: Pergamon. 3. Matsumoto D, LeRoux J, Wilson-Cohn C, Raroque J, Kooken K, et al. (2000) A new test to measure emotion recognition ability: Matsumoto and ekmans japanese and caucasian brief affect recognition test (jacbart). Journal of Nonverbal Behavior 24: 15733653. 4. Lewis M, Haviland-Jones JM, Feldman Barrett L (2008) Handbook of emotions. New York: The Guilford Press. 5. Neale JM, Liebert RM (1986) Science and behavior: An introduction to methods of research: Prentice Hall. 6. Blood AJ, Zatorre RJ, Bermudez P, Evans AC (1999) Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions. Nature Neuroscience 2: 382387. 7. Phan KL, Wager T, Taylor SF, Liberzon I (2002) Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI. Neuroimage 16: 331348. 8. Royet JP, Zald D, Versace R, Costes N, Lavenne F, et al. (2000) Emotional responses to pleasant and unpleasant olfactory, visual, and auditory stimuli: A positron emission tomography study. Journal of Neuroscience 20: 77527759. 9. Koelsch S, Fritz T, von Cramon DY, Muller K, Friederici AD (2006) Investigating emotion with music: An fMRI study. Human Brain Mapping 27: 239250. 10. Berridge KC, Robinson TE (2003) Parsing reward. Trends in Neurosciences 26: 507513. 11. Knutson B, Adams CM, Fong GW, Hommer D (2001) Anticipation of increasing monetary reward selectively recruits nucleus accumbens. Journal of Neuroscience 21: 159. 12. Koob GF, Moal ML (1997) Drug abuse: Hedonic homeostatic dysregulation. Science 278: 5258. 13. Levitin DJ, Menon V (2003) Musical structure is processed in Language Areas of the brain: A possible role for Brodmann area 47 in temporal coherence. Neuroimage 20: 21422152. 14. Menon V, Levitin DJ (2005) The rewards of music listening: Response and physiological connectivity of the mesolimbic system. Neuroimage 28: 175 184. 15. Blood AJ, Zatorre RJ (2001) Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proceedings of the National Academy of Sciences USA 98: 1181811823. 16. Mitterschiffthaler MT, Fu CH, Dalton JA, Andrew CM, Williams SC (2007) A functional MRI study of happy and sad affective states induced by classical music. Human Brain Mapping. 17. Juslin PN (2001) Communicating emotion in music performance: A review and a theoretical framework. Music and emotion: Theory and research. 18. Balkwill LL, Thompson WF (1999) A cross-cultural investigation of the perception of emotion in music: Psychophysical and cultural cues. Music Perception 17: 4364. 19. Fritz T, Jentschke S, Gosselin N, Sammler D, Peretz I, et al. (2009) Universal recognition of three basic emotions in music. Current Biology. pp 573576. 20. Juslin PN, Sloboda JA, eds (2001) Music and emotion: Theory and research. New York: Oxford University Press. 21. Gabrielsson A, Juslin PN (1996) Emotional expression in music performance: Between the performers intention and the listenerss experience. Psychology of Music 24: 6891. 22. Juslin PN, Laukka P (2003) Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin 129: 770814. 23. Juslin PN (2000) Cue utilization in communication of emotion in music performance: Relating performance to perception. Journal of Experimental Psychology: Human Perception and Performance 26: 17971812. 24. Schubert E (2001) Continuous measurement of self-report emotional response to music. In: Juslin P, Sloboda JA, eds. Music and emotion: Theory and research. New York, New York: Oxford University Press. pp 393414. 25. Meyer LB (1956) Emotion and meaning in music. Chicago: University of Chicago Press. 26. Sloboda JA, Juslin PN (2001) Psychological perspectives on music and emotion. Music and emotion: Theory and research. pp 71104. 27. Schubert E (2004) Modeling perceived emotion with continuous musical features. Music Perception 21: 561585. 28. Schubert E (1999) Measuring emotion continuously: Validity and reliability of the two-dimensional emotion-space. Australian Journal of Psychology 51: 154165. 29. Russell JA, Pratt G (1980) A description of the affective quality attributed to environments. Journal of Personality and Social Psychology 38: 311322. 30. Palmer C (1997) Music performance. Annual Review of Psychology 48: 115138. 31. Koelsch S, Kilches S, Steinbeis N, Schelinski S (2008) Effects of unexpected chords and of performers expression on brain responses and electrodermal activity. PLoS ONE. e2631 p.

32. Bhatara AK, Duan LM, Tirovolas A, Levitin DJ (2010) Musical expression and emotion: Influences of temporal and dynamic variation. Manuscript under; review. 33. Sloboda JA, Lehmann AC (2001) Tracking performance correlates of changes in perceived intensity of emotion during different interpretations of a Chopin piano prelude. Music Perception 19: 87120. 34. Huron D (2006) Sweet anticipation: Music and the psychology of expectation. Cambridge: MIT Press. 35. Rankin S, Large EW, Fink P (2009) Fractal tempo fluctuations and pulse prediction. Music Perception 26: 401413. 36. Large EW, Jones MR (1999) The dynamics of attending: How people track timevarying events. Psychological Review 106: 119159. 37. Jones MR (1976) Time, our lost dimension: Toward a new theory of perception, attention, and memory. Psychological Review 83: 323335. 38. McAuley JD (1995) Perception of time as phase: Toward an adaptive-oscillator model of rhythmic pattern processing. [doctoral dissertation]. Bloomington: Indiana University. 39. Snyder JS, Large EW (2005) Gamma-band activity reflects the metric structure of rhythmic tone sequences. Cognitive Brain Research 24: 117126. 40. Fujioka T, Large EW, Trainor LJ, Ross B (2009) Beta and gamma rhythms in human auditory cortex during musical beat processing. The Neurosciences and Music III - Disorders and Plasticity: Annals of the New York Academy of Sciences 1169: 8992. 41. London JM (2004) Hearing in time: Psychological aspects of musical meter. New York, New York: Oxford University Press. 195 p. 42. Grahn JA, Brett M (2007) Rhythm and beat perception in motor areas of the brain. Journal of Cognitive Neuroscience 19: 893906. 43. Schubotz RI, Friederici AD, Cramon DYv (2000) Time perception and motor timing: A common cortical and subcortical basis revealed by fMRI. Neuroimage 11: 112. 44. Bengtsson SL, Ullen F, Ehrsson HH, Hashimoto T, Kito T, et al. (2009) Listening to rhythms activates motor and premotor cortices. Cortex 45: 6271. 45. Chen JL, Penhune VB, Zatorre RJ (2008) Listening to musical rhythms recruits motor regions of the brain. Cerebral Cortex 18: 28442854. 46. Grahn JA (2009) The role of basal ganglia in beat perception. The Neurosciences and Music III - Disorders and Plasticity: Annals of the New York Academy of Sciences 1169: 3545. 47. Grahn JA, Rowe JB (2009) Feeling the beat: Premotor and striatal interactions in musicians and nonmusicians during beat perception. The Journal of Neuroscience 29: 75407548. 48. Herry C, Bach DR, Esposito F, Di Salle F, Perrig WJ, et al. (2007) Processing of temporal unpredictability in human and animal amygdala. The Journal of Neuroscience 27: 59585966. 49. Bush G, Luu P, Posner MI (2000) Cognitive and emotional influences in anterior cingulate cortex. Trends in Cognitive Science 4: 215222. 50. Nair DG, Large EW, Steinberg FL, Kelso JAS (2002) Perceiving emotion in expressive piano performance: A functional MRI study. In: Stevens C, Burnham D, Schubert E, Renwich J, eds. Proceedings of the 7th International Conference on Music Perception and Cognition: Adelaide: Causal Productions. pp 627630. 51. Becker JO (2004) Deep listeners: Music, emotion, and trancing: Indiana University Press. 52. Schubert E. Continuous response to music using a two dimensional emotion space; 1996; Montreal, Canada. ICMPC 263268. 53. Hall DA, Haggard MP, Akeroyd MA, Palmer AR, Summerfield AQ, et al. (1999) Sparse Temporal sampling in auditory fMRI. Human Brain Mapping 7: 213223. 54. Large EW (1992) Judgments of similarity for musical sequences unpublished manuscript. ColumbusOH: Center for Cognitive Science. Ohio State University. 55. Large EW, Rankin SK (2007) Matching a score to a performance. In: Eerola T, Toiviainen P, eds. MIDI Toolbox: MATLAB Tools for Music Research. KopijyvaJyvaskyla, , Finland: University of Jyvaskyla. 56. Palmer C, Brown JC (1991) Investigations in the amplitude of sounded piano tones. Journal of Acoustical Society of America 90: 6066. 57. Cox RW, Hyde JS (1997) Software tools for analysis and visualization of fMRI data. NMR in Biomedicine 10: 171178.