Synsthetic Music: the Ocular Harpsichord
Julin Villegas & Michael Cohen {d8091104, mcohen}@u-aizu.ac.jp
Computer Arts Laboratory University of Aizu Aizu-Wakamatsu, Japan
7th Int. Conf. on Computer and Information Technology.
J. Villegas & M. Cohen (University of Aizu)

Synsthetic Music

1 / 23

Outline

Introduction Hue and pitch Implementation Results Conclusions

2 / 23

Introduction
With the development of music, more than 3,000 types of scales have been proposed. Scales are usually described by relative ratios between steps, or more recently by the number of cents of each tone. Such description is useful for the construction of scales, but fails to convey the spirit of each intonation system.

3 / 23

Differences between three tuning systems

I II III IV V VI VII

Pythagorean 1 1.13 1.27 1.33 1.5 1.69 1.9
Natural Major 1 1.13 1.25 1.33 1.5 1.67 1.88
Natural Minor 1 1.13 1.2 1.33 1.5 1.6 1.8

4 / 23

Bachs approach

5 / 23

J. S. Bach demonstrated the different character of every tonality in his well-tempered clavier by musical examples. Bachs preludes and fugues capture the character of each tonality in his irregular temperament, but provide almost no clue about the tuning system itself. As a consequence, his temperament system is often perhaps not known.

Our approach

We propose to use the inherent relationship between audible frequency and visible frequencies to produce the emissive spectrum of each scale. Every chord activates corresponding wavelengths and produces a unique color in each tuning system.

6 / 23

Clarications
This software is intended to illustrate the differences between musical scales. Spectograms and sonograms are useful to illustrate differences between timbres.
Sonogram of a 12-TET chromatic scale played by a piano.

7 / 23

8 / 23
Hue and pitch observations
Most prominent light wavelengths Hue The greatest common divisor frequency of a complex tone Pitch 6 to 7 million color sensors in our eyes 18 thousand sound sensors in our ears We can hear across a range of 10 octaves We can see across a range of 1 octave c = f , where c is the speed of the light or sound, the wavelength, and f the frequency.

9 / 23

Previous matchings
Newton identied seven main colors in the spectrum produced by diffracting a beam of sunlight with a prism. Newton also observed that the width of the colors in the spectrum coincided with the ratios of the minor diatonic scale. Louis-Bertrand Castel, in 1725, envisioned the ocular harpsichord.

Newtons mapping.

(Taken from Opticks 4th. Edition, digitized by Google).

10 / 23

Visual music: music - color synsthesia
Kandisky W. Composition VII (1913).
J. Villegas & M. Cohen (University of Aizu) Synsthetic Music

11 / 23

Implementation Overview

S CALA les

Tonality

12 / 23

sequencer

Tuning information

receiver

SK-88 Pro

MIDI MIDI

Color renderer

display

Specications

Only twelve tone scales Only plays MIDI les Three visualizations available:
The tonic of the scale is associated to the low visible frequency ( 394 THz) The velocity of the notes affects the lightness and the transparency.

(MIDI) A 0 (MIDI) 0

Two or more simultaneous notes produce an emissive spectrum.

13 / 23

S CALA is a software system for creation and manipulation of musical scales. It supports tuning of electronic instruments via the MIDI protocol. A user can experiment with new scales or edit one of its more than 3,400 presets. S CALA is freeware and available on internet.

14 / 23

Microtuning
We used the Roland SoundCanvas SK-88 Pro. Scale Tune Feature, a System Exclusive message that changes each notes pitch up to 64 cents. This adjustment affects all the notes in the synthesizer regardless of octave, and this limitation is the reason for using only twelve tone scales in our application.

15 / 23

Java/JOGL Implementation

MIDI note

n(mod12)

16 / 23

n(mod12) ratio = f (tonality, tuning)

c ratioredfreq ,

a =MIDIvel /127.
R 13.361 @ G A=@ -2.91 B 0.03

-5.066 8.06 -0.08

-2.7 X 0.58 A @ Y A 5.70 Z
13.361 R @ G A=@ -2.91 0.03 B 0
X -2.7 0.58 A @ Y A Z 5.70
The RGBA values are corrected, normalized, and stored to be used for the JOGL renderer.

Results

12-TET

Pythagorean

Well-temperament Chopins Prelude No. 4, Op. 28 in Em

17 / 23

Applications
Didactic of music For physically or musically challenged persons For fast comparison between tunings As reference for artists (like the well-known Pantone cards) For artistic purposes

18 / 23

Future work
Using real sounds as inputs. Subjective test to measure the impact of introducing visual cues in the process of learning musical tunings. Explore artistic applications of the presented software. Visualization of other music properties/techniques (i.e., portamento, vibrato, glissando, etc.) along with pitch and velocity.

19 / 23

Demonstration

20 / 23

Conclusions
Only few people can relate the ratios of a scale with its actual sound and characteristics. Mapping microtuned scales into a more familiar phenomenon such as color eases the appreciation of the subtle differences between them. Although the efcacy of the implementation in learning new tunings hasnt been veried yet with subjective tests, the ideas discussed here could be used in music education.

21 / 23

Thank you Q&A

22 / 23

Scala le format
The les are encoded using ASCII or 8-bit character text-les. The le format used for S CALA consists of a set of lines: i. describes briey the contained scale ii. the number of notes in the scale the rest: the pitch of a tone in the scale, either as an integer ratio (e.g., 2/1), or as oating point number of cents (e.g., 408.6).

23 / 23

Supporting Musical Composition by Externalizing the Composer's Mental Space
Shigeki Amitani Department of Advanced Interdisciplinary Studies University of Tokyo amitani@ai.rcast.u-tokyo.ac.jp
ABSTRACT In the field of design and creativity support, "externalization of mental space" has been recognized as an important challenge. In this paper, we tackle this challenge on musical composition, which is one of the important human creative activities. In our research, we focus on the analysis of cognitive processes in musical composition. That is, we analyze what the cognitive processes in musical composition are like and how the process is affected when a representation of information is changed. For this research, we propose a musical composition supporting system named "MACSS (MAcroscopic Composition Supporting System)" which offers a spatial representation of music to a composer to support his/her composition process. Comparing with ordinary musical editors, which give chronological (1-dimensional, scoremetaphored) representation, this system provides a macroscopic view by locating phrases on the 2-dimensional space. We have investigated how the cognitive process changes when the spatial representation is introduced, i.e., the representation of information is changed. Moreover, we have analyzed the observed change microscopically, especially the processes of "mental fixation" and "mental leap". We have found how the spatial representation triggers mental leap escaping from mental fixation in musical composition. Categories & Subject Descriptors: H.5.2: User Interfaces (see http://www.acm.org/class/1998/) General Terms: Design, Human Factors Keywor ds Creativity support, musical composition, externalization, mental space, cognitive process, spatial representation INTRODUCTION In the field of creativity support, it is important to investigate how a change of external representation of information gives effects on human cognitive processes in creative activities. In this paper musical composition process is selected as one of the important creative activities. Norman [8] claims that cognitive artifacts should support our activities by amplifying our reflective thinking. It means that tools are required to provide a user with proper information in a proper
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permis sion and/or a fee. C&C'02, October 14-16, 2002, Loughborough, Leic, United Kingdom. Copyright 2002 ACM 1-58113-465-7/02/0010 $5.00.
Koichi Hori Department of Advanced Interdisciplinary Studies University of Tokyo hori@ai.rcast.u-tokyo.ac.jp
way of representation when the user is engaged in a creative activity. In our research, we focus on the analysis of the cognitive process in musical composition. Tanaka [16] points out that fewer analyses on musical composition have been conducted compared to those on the cognitive processes of musical listening and performance, and that most of them are macroscopic analysis through flashing an idea to completion of the musical piece. That is the reason we focus on detailed analysis of composition process. ?? How composition process proceeds with an ordinary musical editor ?? How the human cognitive process in musical composition is affected when the representation of information changed. For this research, we propose a musical composition supporting system named "MACSS (MAcroscopic Composition Supporting System)" which offers a spatial representation to the composer to support his/her composition process. Comparing with ordinary musical editors, which provide chronological (1-dimensional, score-metaphored) representation, this system is expected to provide a macroscopic view of pieces by locating phrases on a 2dimensional space. In the context of musical composition, the term "macroscopic view" means to grab the whole of the musical piece being composed or to grab the relationship among the pieces (phrases) the composer has composed. The observed change was analyzed in detail and described, especially the process of "mental fixation" which means the deadlock of the thought and "mental leap" which means to escape from the deadlock of the thought. We observed how the spatial representation triggers mental leap escaping from mental fixation in musical composition more frequently than chronological representation. RELATED WORK Schoen [12] has explained the design process as "Seeing-DrawingSeeing Cycle". Design process starts from "Seeing", proceeds to "Drawing", or generating (externalizing) a partial solution to the design problem, returns to "Seeing" and repeats this cycle, which is shown in Figure 1. This process can be regarded as a "dialog" between a designer and his/her material. Usually this process is ambiguous and dealt in the designer's mind. It is possible to put this design process into concrete by "externalizing the designer's mental space", which enables the designer to grasp his/her whole of mental space. It is expected that this method can support the creative activities [4][5]. This process is shown in Figure 2. Zhang [21] has proposed a theoretical framework for the problemsolution process based on the external representation. In Zhang's research, he conducted his experiment with a game called Tic-Tac-Toe and its isomorph. The experiment demonstrated that actions in the problem-solution process were determined by information directly available to human cognition or perception. Based on this experiment, Zhang suggested

"representational determinism", that is, the form of representation determines what information can be perceived, what processes can be activated, and what structures can be discovered from a specific representation. Suwa et al. [15] have devised a scheme for coding designer's cognitive actions from video/audio design protocols. Designer's actions were coded into four cognitive levels (i.e., physical, perceptual, functional and conceptual). Relations between each cognitive process were systematically described and analyzed. Through this analysis, Suwa concluded that the interaction between the designer and his/her sketches provides the designer with such effects as re-interpretation of the design problems and unexpected discoveries. This method is u seful to code and to analyze human cognitive process.
representation. He concluded that the significance of the spatial representation was to represent mainly the similarity among information in accordance with their proximity. Their system aimed to support the process "incremental formalization", that is, the user gradually structures his/her concept. Hori [5] has conducted an experiment to investigate the effect of the spatial representation on a conceptual design. A system named AA1 presented words that rep resent the concept the user vaguely conceived, on the two-dimensional space. The user c ould change the location of the words on the space. This action help ed the user clarify his/her concept gradually and the phenomenon was observed that the user came to generate new concept by looking at the blank area on the space. Yamamoto et al. [18] have presented a theoretical framework for a method to support information design with an interactive system at the early stage of the creation of information. The interactive system aimed to support the process to create information that was still ambiguous in the user's mind through trial-and-error. They concluded that it is not "the representation for the design solution" but "the representation for understanding the design solution" that the designer needs at the early stage of the creation of information. That is, the spatial representation can support to clarify gradually the designer's concept that is ambiguous through the interaction between the designer and the system in the divergent thinking process. From these studies, the spatial representation can support to structure and clarify the designer's ambiguous concept gradually through the interaction. Furthermore, it can help generate a new idea. Information with "Linearity" and "Gestalt" Reybook [10] has examined limitations and possibilities of application of Gestalt concepts in relation to music. The research was conducted mainly on the cognitive process of listening music, but the assertion that music analysis has to be broadened from a structural description to a description in terms of processes must be taken into consideration to analyze the cognitive process of musical composition. Yamamoto et al. [18] have also referred to "linearity" and "Gestalt" in semiotics and they attempted to present with the system Gestalt to linear information, i.e., document in the case of their research, simultaneously. They concluded that this approach allowed the designer (writer): ?? To inspect the information in detail with the linearity ?? To grasp the information from the bird's-eye view with Gestalt This semiotics approach can be applied to music that is also information with linearity. As well as the document composition, the spatial representation that provides Gestalt allows the designer (composer) to grasp whole structure of the music piece, and moreover, this representation can support the musical composition process. In our research, it is attempted to introduce Gestalt concept in musical composition process with a spatial rep resentation that is appropriate to grasp the relation among the objects. Introducing this representation, it is expected that the composer use both spatial and score-metaphored representation, i.e. he/she uses a spatial representation (Gestalt) to grasp the information from bird's-eye view and uses score-metaphored representation (linearity) to inspect the information in detail.

Figure 1. Seeing-Drawing-Seeing Cycle
Figure 2. A Cycle of Design Process with Externalizing the Designer's Mental Space
The Spatial Representation as a Way of Externalizing Mental Space
In this research, the spatial representation has been adopted
as a way of externalizing mental space. Several studies have been conducted that employed the spatial representation for externalization. Shipman et al. [13] have investigated significance of the spatial
Research on Musical Composition Process A few researches on musical composition processes have been conducted in the music psychology field. Sloboda [14] has described the composition process (Figure 3.) Though Umemoto [ 7] has also presented macroscopic analysis 1 like Sloboda's, as Tanaka [ 6] has mentioned in his paper, a 1 problem in this field is that almost no research has been conducted to investigate what the each process is like and what the transition among processes is like. In this paper, we investigate the cognitive process in the musical composition microscopically with protocol analysis method. It is expected that the analysis provide us with the rich knowledge for designing a creativity support system in the future. In the field of creativity support, as few practical researches have been conducted it is necessary to analyze a lot of cognitive processes through various instances. As our approach to this challenge, the musical composition process, which is one of the representative and important human creative activities, is selected. The "mental space" in this research refers to the part named "A: idea" and "B: theme" in "CONSCIOUS" area in Figure 3. It is expected that the spatial representation decrease the cognitive load when the composer grasps his/her unfinished phrases and offers macroscopic view that the ordinary chronologi cal editor does not provide. MACSS (MAcroscopic Composition Supporting System) MACSS aims at providing the composer with ability to grasp whole of his/her piece. That is, MACSS provides a "macroscopic view". Figure 4 shows the screen shot of MACSS. MACSS arranges the phrase objects (MIDI files only) on the twodimensional space by Multi-Dimensional Scaling method (MDS) based on calculated similarity among the phrases. In the music psychology field, it is said that the similarity of phrases is grasped mainly based on three musical characteristics, such as melody contour, key and rhythm [1]. We adopted these criteria for the calculation of similarity among music phrases on MACSS, though there is room for further discussion to define the similarity [2]. MACSS was developed with JDK ver.1.3. There are composers who accumulate small phrases they hit upon and develop or combine them to compose a musical piece. Though ordinary musical editors are useful for detailing a phrase, it lacks a global view of the whole piece. MACSS provides flexibility to reconstruct the piece by its spatial representation. It cooperates with the ordinary editor and they complement with each other.

The composer is expected to proceed his/her composition by exploring the "whole" and "a part" of the piece. That is, MACSS allows the composer to rearrange the phrase objects on the space and to listen to the phrases in an order the composer specifies so that he/she can compose the "whole structure of the piece". On the other hand, the "Edit Phrase" command allows the composer to edit each phrase, i.e. "a part of the piece" on the ordinary chronological editor. When the composer rearranges the phrase objects, MACSS recalculates the similarity of the phrases from distances among the phrase objects and tunes weights of the three musical characteristics used for similarity. It is expected to become easier for the composer to grasp the phrases in his/her long-term composition that can provide the composition process with the nature of Gestalt to. Furthermore, it is also expected such interaction that the composer come to generate new concept by looking at a blank area on the s pace. To enhance the nature of Gestalt, an aspect of each phrase object is editable with "Edit Picture" command (right-click menu). It opens the icon with an external image editor and the composer can put an image on the object. For example, "This color expresses atmosphere of this phrase", "This picture can be a jacket of this piece", etc. This function was added because one of the composers who participated in this experiment wished. The Representation of Music on a Computer Hiraga [ has proposed Functional Representation of Music 3] (FRM) as a representation of music on a computer. We have partly adopted this representation to implement the system. Hiraga also suggested the representation of similarity of musical phrases based on such formal elements as m elody contour, contrary motion, retrograde motion, repercussa, rhythm and chord.
Figure 4. A Snapshot of MACSS Similarity Calculation Melody Contour and Rhythm Similarity A melody contour is represented as an array of the musical intervals, that is: Interval =currentMIDINoteNumber - previousMIDINoteNumber Rhythm is also represented as an array of the time interval between the Note-on times, that is: Rhythm = currentMIDINoteOnTime - previousMIDINoteOnTime Figure 3. Sloboda's Diagram of Musical Composition Process
To define the similarity of the phrases, Direct Pattern Matching (DPM) method is adopted [19]. This method is to count how many elements in two groups with discreet values are correspondent with each other. Key Similarity Several key recognition methods are proposed and the model of Yoshino and Abe [20] is adopted for MACSS because it offers "the most appropriate key name", while the others do not unless an input phrase completely conforms to the key rule. Total Similarity Total similarity is defined as: Similarity = w1 * similarityOfInterval + w 2 * similarityOfKey + w 3 * similarityOfRhythm These values of w1, w 2 and w3 change according to changes of the object positions on the space. They are recalculated with difference of the distances before and after the position changes. Though there is a room for discussion about the method of similarity calculation, it is useful to show similarity among phrases from these view points.

used in the experiment are shown in Table 2. EOS provides almost same functions as Cakewalk, that is, it is possible to edit various musical characteristics and it has the chronological representation. As the subject E used to composition of musical pieces on EOS, he used EOS in this experiment instead of Cakewalk. Procedures The process of the composition was recorded by digital video camera (SONY DCR-TRV20). Recorded visual and audio protocol data were analyzed. The procedure of the experiment is based on the reference [9]. The subjects were presented with a picture and asked to compose a music piece/music pieces about it or whatever it suggested. Figure 5 shows the picture presented to the subjects as a theme. The subjects were asked to answer a questionnaire based on the reference [9]. Table 1. List of the Subjects
The Spatial Representation
Multi-Dimensional Scaling (MDS) is a set of data analysis techniques that disp lay structure of distance-like data as a geometrical picture. In this research, an adopted method for MDS is Torgerson's method [11]. The Expected Interactions Using MACSS, following interactions are expected. As a Macroscopic Editor MACSS facilitates for the composer to grasp characteristics of phrases, and their relations by arranging the phrases in accordance with the similarity the composer decides. It is expected that it promote to think how to compose whole structure of the piece. With only a chronological editor, it is sometimes difficult to grasp the whole of the piece. As MACSS provides the composer with a macroscopic viewpoint, it is helpful to develop the piece macroscopically. As a Macroscopic Database In a long-term composition, as the number of phrase increases, it gradually becomes difficult to grasp what each phrase is like with the use of an ordinary "list" representation on the computer system. In MACSS, as the arrangement of phrases is defined by the composer, it is easier for the composer to know what each phrase is like. EXPERIMENT In this experiment, we are going to investigate how the cognitive process in musical composition changes when the spatial space representation is presented during the composition. Table 1 shows five composers who participated in the experiment. They all have experience of musical composition. A place of the experiment was up to the subjects. If they do not specify any particular place, experiments were held at our laboratory. Hardware and software Figure 5. The Theme Picture PC Keyboard Musical Editor

Table 2. Equipment

Equipment SONY VAIO PCV-R63K or Gateway SOLO Roland SK-88 Pro or YAMAHA EOS B-900 Roland Cakewalk Audio Pro 9 and MACSS
Preliminary Experiment A preliminary experiment was held to establish the procedure of the experiment. The subject was a graduate student (the subject A) in our laboratory. In this preliminary experiment, whole process was recorded on VTR and analyzed to grasp the process roughly. In this paper, the words "microscopic action" and "macroscopic action" are used. The former means " action to detail a phrase", such as adjusting velocity of a note, changing length of a note, etc. The latter means "an editing action on plural phrases", such as thinking a structure of a piece, etc. It was observed that the composition proceeds chronologically even at the early stage of divergent thinking process when the subject composed only with the ordinary score-metaphored musical editor. Note that "the composition proceeds chronologically" means that
the composer develops the next phrase to the phrase which he/she has composed just now. In this case the subject seemed to regard "the next phrase to the phrase which he/she has composed just now" as one and only solution at that point and mental fixation was often observed. It is often said that a part of a piece does not consistent to the other part of the same piece, which can happen even to a professional composer. It is necessary for the composer to have means to grasp whole musical information in some way because music is classified to "Temporal art" where it is impossible to grasp the piece until it is played. In this research, we introduce the spatial representation that may promote the composer's understanding of the information. From the observation of this preliminary experiment, following hypothesis was established to investigate the change of cognitive process. Though musical composition tends to proceed chronologically when the composer uses a musical editor with a chronological score-representation, the composition process will proceed unchronologically with a spatial representation musical editor MACSS. Main Ex periment Based on the preliminary experiment, the procedure was modified for ease of understanding and a main experiment was designed. The main experiment was held four times for each subject under following conditions. As the composition does not finish at one time, the rest of the composition was continued on the other day. 1st & 2nd Chronological editor only 3rd & 4th Chronological editor and MACSS The subjects were presented with a picture and asked to compose a music piece/music pieces about it or whatever it suggested. The process of the composition was recorded and was analyzed in the same way as preliminary experiment. Time length of the experiment was completely up to each subject and it continued by the time when the subject declared the end. So, the experiment time ranges from 40 minutes to 3 hours for each session. The instruction was presented to the subjects before the experiment. Analysis Method In this experiment, the composition process was analyzed based on the scheme that Suwa et al. [15] have proposed to analyze architectural design. The unit cognitive processes are defined in Table 3, which consists of three cognitive levels such as "Thought", "Perception" and "Action". The composition process was analyzed along with these unit cognitive processes, so that the processes were analyzed systematically. In this experiment, we focused on: 1. 2. The "mental fixation " and "mental leap" in the development of the phrase The change of the cognitive processes before/after the spatial representation

three levels "What did (=Action)". figures are compared:
that we defined: "What did you see? (=Perception)", you think? (=Thought)" and "What did you do? The unit cognitive processes on the nodes in the defined inTable 3. The following two cases were
Case 1 Composition process with the chronological editor only Case 2 Composition process with the chronological editor and MACSS A main observed difference between the two cases is: ?? ?? In the case 1, the subject composed the piece in a chronological way. In the case 2, the subject composed the piece by combining some phrases he/she composed in their own order. Table 3. Defined Unit Cognition Processes
Category Names Thought Compare Confirm Analyze Description Compare objects Confirm object Analyze object Examples Compare one combination with another Listen to whole phrase Analyze how 2 phrases are different Remember which phrase is like what Plan to change the timbre Decide to extend the phrase Listen to the phrase just editted Look at objects on the mental space, score Read the sentences attatched to the phrase Play the phrases
Remember Remember object Plan Decide Perception Listen Look Read Action Play Explore Adjust Enter Delete Combine Move Extend Select Make Plan the next action Decide the next action Trial listening of object Look at object Read object Play object
Explore to find something Look for a timbre Adjust attributes of object Adjust the length, pitch, timing Enter object Delete object Combine objects Enter a note, or notes Delete a note, notes or a phrase Combine 2 or 3 objects
Move objects on the space Move to group the phrases Extend the object Select the object(s) Make a new object Extend the phrase Select a new phrase Make a new file
These processes were analyzed microscopically and transitions of the cognitive processes were described. RESULTS AND DISCUSSION The cognitive processes were analyzed especially the process where a subject hit upon a new development or failed in developing the phrase after he/she said, "What should I do next?" Figure 6 to 8 shows the transition of the cognitive processes. These diagrams indicate the transition of the cognitive processes at
Figure 6 shows the composition process where the subject could not develop the phrase in the case 1. In this diagram, the loop 1 shows that the subject looked at and wandered some windows on the screen trying to hit upon a new development. The process changed to the loop 2 (transition arrow A) where the subject repeated microscopic actions. Eventually, the subject did not extended the phrase and said, "What should I do next?" again (transition arrow B and C). After this process, the subject repeated trial listenings and many of actions the subject took were only microscopic ones such as adjusting the tempo. Figure 7 shows the composition process where the subject could develop the phrase in the case 2. In this diagram, the same loop 1 as in the figure 6 was observed. Repeated microscopic actions were also observed (loop 2). Then the subject did a trial listening through whole phrase to hit upon a phrase that comes to the right next to the phrase the subject has just listened. As mentioned, in the two cases of the case 1, composition processes proceeded in a chronological way.

Figure 8 shows the composition process where the subject could develop the phrase in the case 2. In this diagram, such new processes as "PlanMakePhrase", "MakeNewFile" and "PlanCombination" (transition A) where the subject plans how to combine the phrases into a piece, were observed. This process is, what we call, a macroscopic composition that did not appear in the case 1. In the loop 2 in figure 8, the subject selected phrases for trial listenings (SelectPhrase), did trial listenings (PlayOnScreen) and moved phrases to decide how to organize the piece. After some repetition of the loop 2, the subject finally could decide the structure of the piece (transition B). In this third case the process was not chronological. It was observed that the spatial representation offered a macroscopic viewpoint to the subject. The subject composed not in a chronological way, but in such a way as "The subject composes some phrases and then examines the structure of the piece from the macroscopic view point". The subjects commented as follows in an interview after the experiment: At the second time (without the spatial representation), it was the failure that I tried to decide the phrase only in my mind. That's why I came to mental fixation. I wanted to change its base line vastly (though he did not do it indeed). (At the third time with the spatial representation) as the change did not go along with my taste, I composed three candidates. Then I tried to choose which were suitable for this part. I wanted to compare with each other. While I moved the phrases and did trial listenings (with the spatial representation), it went well though it was difficult when I did it only in my mind. Visual representation helped a lot. By comparing phrases on the space, one of the subject found that he unconsciously composed the completely same phras es. Another subject talked about positions of the phrases on the space to analyze his own piece. Table 4 shows the ratio of generated bars before and after MACSS was introduced. As to the subject E, he was very accustomed to composition so he could complete composition of a piece at one time. This table shows that the ratios of the subject A and B increases while those of the subject C and D decreases. Though the subject A and B used MACSS, C and D did not. It is interesting that the users' ratio increased and non-users' decreased. Of course it is possible to say that the subject C and D already finished developing their phrase by the time when MACSS was introduced. However, the subject C developed the phrase chronologically and there were some clues for mental fixations. In the middle of the second experiments the subject C referred to further development of the piece: "A dramatic development is coming soon"; "Main phrase is coming after this phrase. In spite of these statements, there was no further development. Though the subject D composed some phrases, he did not come to organize whole of the piece. That is, the composition of subject C was restricted by the "chronological limitation", i.e., "The feeling the representation lets the composer compose in a chronological way" and that of D lacked the "macroscopic viewpoint". From these results of the analysis, there are two patterns of escaping from mental fixation: 1. 2. Develop the phrase chronologically Develop some phrases unchronologically and try to organize their combination Figure 6. Cognitive Processes without MACSS (Mental Fixation)

Figure 7. Cognitive Processes without MACSS (Chronological Development)
Figure 8. Cognitive Processes with MACSS (Mental Leap)
Table 4. The Ratio of the Number of the Generated Bars before/after Using MACSS Before 4.92% 42.75% 76.58% 56.86% 100.00% After 95.08% 57.25% 23.42% 43.14% 100.00%
Subject A Subject B Subject C Subject D Subject E
in composition is also interesting topic, but that is beyond this paper. With such analyses on various aspects of the musical composition process, it is expected to obtain knowledge about the design theory to construct a supporting environment for musicians that enables them to be more creative. And it is also expected to generalize the theory to the various systems for creativity support. REFERENCES 1. Abe, J. How is melody processed. Music and Cognition, Cognitive Science Library, Vol.12, pp.41-68, University of Tokyo Press, 1993 2. Hewlett W. B., Selfridge-Field E. Melodic Similarity: Concepts, Procedures, and Applications. Computing in Musicology 11, MIT Press, 1998 Hiraga, Y.: Knowledge Representation for Music Cognition. Music and Cognition, Cognitive Science Library, Vol.12, pp.97-130, University of Tokyo Press, 1993 Hori, K A Model to Explain and Predict the Effect of. Human-Computer Interaction in the Articulation Process for Concept Formation. Information Modelling and Knowledge Bases, Vol.7, pp.36-43, IOS press, 1996 Hori, K. A System for Aiding Creative Concept Formation. IEEE Trans. Systems, Man, and Cybernetics, Vol.24, No.6, pp.882-894, 1994 Nakakoji, K., Yamamoto, Y., Reeves, B.N. and Takada, S. Two-dimensional Positioning as a Means for Reflection in Design. Proceedings of DIS 2000, ACM, pp.145-154, August, 2001 Nishimoto, K. Mase, K. and Nakatsu, R. Melody Creation Support by Visualization of Relationships among Phrases and Musical Primitives. Journal of Information Processing Society of Japan, Vol.40, No.2, pp.687-697, 1999 Norman D. A. Things That Make Us Smart: Defending Human Attributes in the Age of the Machine. Addison Wesley Publishing Company, 1994 Patrick, C. Creative Thought in Artists. Journal of Psychology, Vol.4, pp.35-73, 1937 Reybrook, M. Gestalt concepts and music: limitations and possibilities. in M. Leman (ed.) Music, Gestalt and computing. Lecture notes in artificial intelligence, no.1317 (Berlin: Springer-Verlag), 1997 Saito, T. Non-measuring Multi-Dimensional Scaling Method. Multi-Dimensional Scaling Method, Chapter 5, pp.70-86, Asakura Books, 1983 Schoen, D A. The Reflective Practitioner: How Professionals Think in Action. Basic Books, NY, 1983 Shipman, F. M., Marshall, C. C., and Moran, T. P. Finding and Using Implicit Structure in Human-Organized Spatial Layouts of Information. Proceedings of CHI'95, ACM, pp.346-353, 1995 Sloboda, J.A. Composition and Improvisation. The Musical Mind, chapter 4, Oxford, pp.102--150, 1985 Suwa, M., Purcell, T. and J. Gero. Macroscopic analysis of design processes based on a scheme for coding designers' cognitive actions. Design Studies, Vol.19, No.4, pp.455-483, 1998

The second pattern can be triggered by the spatial representation because: 1. 2. 3. It allows the composer to edit the piece not in "note level" but "phrase level". It facilitates for the composer to grasp the relationship among phrases. It does not show chronologicality with which ordinary musical editors usually provides the composer.
As these patterns seem to be dependent on the representation, the result should be taken into consideration when the musical composition supporting system is designed. Though further statistical analysis is necessary, this is one of the researches to tackle the real-world analysis of creative activities. In the research field of creative activities, it is necessary to investigate various types of real creative activities in detail. Nakakoji et al. [6] have referred to the support for the reflectionin-action and reflection-on-action in the early phase of the design, especially writing process. In their system, a writer can move a document objects on the two-dimensional space and it supports the reflection-in-action by facilitating comparison between the document objects. In our research the subject reflected on himself/herself by comparing some patterns of organization which the subject composed by moving the phrases and changing their order. Therefore, MACSS offers a macroscopic view with the spatial representation and facilitates for the composer to grasp all the phrases and the relations between them, which supports the reflection-on-action phase on the musical composition. In addition, by allowing the composer to listen to plural phrases in a sequence in which the composer would like to listen to, MACSS supports the reflection-in-action phase. We believe that MACSS facilitates for a composer to change his/her viewpoint from microscopic to macroscopic and vice versa, which enables him/her to compose a piece with consistency that he/she would like to realize. CONCLUSION Though there has been a lot of research on cognitive processes of listening and musical performance, few researches on those of musical composition have been conducted [16]. In the research field of creativity support, as few microscopic analysis have been conducted it is one of the important challenges to detail knowledge on control and measurement of cognitive processes through various instances. Nishimoto et al. [7] have proposed and implemented a supporting system for improvisation in jazz session on the application of a spatial representation to musical composition. Our research focused on a microscopic analysis of the effects on cognitive processes in the musical composition caused by the change of the representation of information. An analysis of behaviour difference in terms of the degree of expertize

9. 10.

12. 13.

14. 15.

Tanaka, Y. Musical Composition as a Creative Cognition Process. Report of Cultural Sciences Faculty of Tokyo Metropolitan University, Vol.307, No.41, pp.51-71, 2000 Umemoto, T. The Psychology of Musical Composition. Musical Psychology, Chapter 6, pp.377-413, Seishin Shobo, 1972 Yamamoto, Y. Amplifying Representational Talkback: Interactive Systems Using Spatial Positioning to Su pport Early Stages of Information Design. Ph.D thesis at Graduate School of Information Science Nara Institute of Science and Technology, 2000
Yanase, T. Melody Searching System by Feature Excerption from Play Information. Ph.D thesis, the Department of Electric Engineering, Graduate School of University of Tokyo, 1999 Yoshino, I. and Abe, J. Key Recognition: Calculation Model to Interpret the Key of the Melody. The World of Computer and Music, pp.117-131, Kyoritsu Press, 1998 Zhang J. The Nature of Ext ernal Representations in Problem Solving. Cognitive Science, Cognitive Science Society, vol.21 (2), pp.179-217, 1997