A Web-Based Game for Collecting Music Metadata
Michael I Mandel Columbia University LabROSA, Dept. Electrical Engineering mim@ee.columbia.edu Daniel P W Ellis Columbia University LabROSA, Dept. Electrical Engineering dpwe@ee.columbia.edu

August 14, 2008

Abstract We have designed a web-based game, MajorMiner, that makes collecting descriptions of musical excerpts fun, easy, useful, and objective. Participants describe 10 second clips of songs and score points when their descriptions match those of other participants. The rules were designed to encourage players to be thorough and the clip length was chosen to make judgments objective and specic. To analyze the data, we measured the degree to which binary classiers could be trained to spot popular tags. We also compared the performance of clip classiers trained with MajorMiners tag data to those trained with social tag data from a popular website. On the top 25 tags from each source, MajorMiners tags were classied correctly 67.2% of the time, while the social tags were classied correctly 62.6% of the time.

Introduction

The easiest way for people to nd music is by describing it with words. Whether this is from a friends recommendation, browsing a large catalog to a particular region of interest, or searching for a specic song, verbal descriptions, although imperfect, generally suce. While there are notable community eorts to verbally describe large corpora of music, e.g. Last.fm, these eorts cannot suciently cover new, obscure, or unknown music. Eorts to pay expert listeners to describe music, e.g. Pandora.com, suer from similar problems and are slow and expensive to scale. It would be useful in these cases to have an automatic music description system, the simplest of which would base its descriptions wholly on the audio and would require human generated descriptions on which to train computer models. An example use of such a system can be seen in Figure 1 along with the data that can be used to train it. Building a computer system that provides sound-based music descriptors requires human descriptions of the sound itself for training data. Such descriptions are most likely to be applied to clips, short segments of relatively obscure songs that provide little context to the listener. Mining available writings on music is not sucient, as only a small portion of the vast quantity of these, e.g. record reviews and blogs, describes aspects of the sound itself, most describes the musics social context. Broad discussions of genre or style generally focus on the social aspects of music, while specic descriptions of a short segment generally focus on the sound. Similarly, due to the heterogeneous nature of musical style, it is not certain that a description

Outkast: Liberation

freq / Hz (Mel scale)
-20 -40 -60 -80 level / dB
m vxs, bs, dr, pi m vx, pi, perc
+vxs f vx, bs, dr, pi +f vx, dr m vxs, bs
m rap, bs, dr, pi pi, bs, dr, conga

dr stop

spoken urban soul r&b rap jazz hip hop piano female vocal pop male bass drums
-0.8 time / min classifier output
Figure 1: Automatic tagging of all ten second segments within a track, illustrating one goal of this work. The top pane shows the mel-scale spectrogram of OutKasts Liberation, with major instrumentation changes manually labeled (m vx = male voice, f vx = female voice, m vxs = multiple male voices, pi = piano, bs = bass guitar, dr = drums, + indicates instruments added to the existing ensemble). The lower pane shows the output of automatic classiers trained on the 14 shown labels. Note, for example, how the switch to a single female lead vocal from 4:04 to 4:52 is strongly detected by the labels female, jazz, r&b, soul, and urban. The six columns of outlined cells indicate the six clips for which human tags were collected in the MajorMiner game; thin outlines indicate that the tag was used at least once, and thick outlines show the most common tags for each clip. Notice that on the whole human tags are consistent with higher scores from the classiers. of a genre or style applies to all possible segments of music within that style. Shorter clips are more likely to be homogeneous, which makes the connection between language and music more denite. Thus in this project, we endeavor to collect ground truth about specic, objective aspects of musical sounds by asking humans to describe clips in the context of a web-based game1. Such a game entertains people while simultaneously collecting useful data. Not only is the data collection process interesting, but the game itself is novel. The overarching goal of collecting thorough descriptions of the music shaped many design decisions about the game-play, including the rules for scoring, the way in which clips are chosen for each player, and the ways in which players can observe each other. We have also run a number of experiments to test the eectiveness with which an automatic music classier can be trained on the data collected by such a game, comparing binary classication accuracy for many individual tags on sets of clips that balanced positive and negative examples. We rst measured the eect of varying the amount of MajorMiner data available to a classier, showing that a six-fold increase in training data can improve classication for some tags by ten percentage points. For the other experiments, we used two variations of tags from MajorMiner and three variations of tags from the social music website Last.fm2. We compared these datasets on their seven common tags and found that MajorMiners data were more accurately classied on all but two tags. We also compared classication accuracy for the 25 tags most frequently applied to our music collection by each dataset. Classiers trained on MajorMiners data achieve

1 The 2 http://www.last.fm/
game is available to play at http://www.majorminer.com and http://www.audioscrobbler.com
Figure 2: A screenshot of a game in progress. The player describes a 10 second clip of an unknown song. Italicized descriptions have scored 1 point, red descriptions 0 points, and gray descriptions have scored no points, but will score 2 points if they are subsequently veried through duplication by another player. an average accuracy of 67.2% and those trained on Last.fms data achieve 62.6%. An example of one possible application of this work is shown in Figure 1. To make this gure, Outkasts song Liberation was broken down into contiguous 10-second clips. While a number of these clips were labeled in the MajorMiner game, most of them were not. By training autotaggers on other songs that were tagged in the game, we are able to automatically ll in the descriptions that should be applied to the unseen clips. The agreement between the autotags and the tags collected in the game for this song can be seen in the gure as dark regions of high predicted relevance correspond to outlined regions of human judgements. In addition, the structure of the song becomes visible as the autotags change in response to changes in the music.

Example of game play

An example session of the game will provide a sense of its rules and strategy. A screenshot of this example session can be seen in Figure 2. First the player, mim, requests a new clip to be tagged. This clip could be one that other players have seen before or one that is brand new, he does not know which he will receive. He listens to the clip and describes it with a few words: slow3 , harp, female, sad, love, fiddle, and violin. The words harp and love have already been used by one other player, so each of them scores mim one point. In addition, the players who rst used each of those words have, at this time, two points added to their scores (regardless of whether they are currently playing the game). Since the words female and violin have already been used by at least two players, they score mim zero points. The words sad and fiddle have not been used by anyone before, so they score no points immediately, but have the potential to score two points for mim at some later time should another player use one. When the player has nished tagging his clips he can go to his game summary, an example of which can be seen in Figure 3. The summary shows both clips that he has recently seen and those that he has recently

will use a monospaced font to denote tags
Figure 3: A screenshot of the players game summary. The artist, album, track, and start time are listed for clips the player has recently seen or scored on. The player can also see their own tags and those of another player. scored on, e.g. if another player has agreed with one of his tags. It also reveals the artist, album, and track names of each clip and allows mim to see one other players tags for each clip. In the gure, the other player has already scored two points for describing the above clip with bass, guitar, female, folk, violin, love, and harp, but has not scored any points yet for acoustic, country, drums, or storms. When he is done, mim logs out. The next time he logs in, the system informs him that three of his descriptions have been used by other players in the interim, scoring him six points while he was gone.

Previous work

A number of authors have explored the link between music and text. Whitman and Ellis (2004) trained a system for associating music with noun phrases and adjectives from a collection of reviews from the All Music Guide and Pitchfork Media. This work was based on the earlier work described by Whitman and Rifkin (2002). More recently, Turnbull et al. (2006) used a naive Bayes classier to both annotate and retrieve music based on an association between the music and text. This work is inspired by similar work in the eld of image retrieval, such as Barnard et al. (2003); Carneiro and Vasconcelos (2005). Eck et al. (2008) used boosted classiers to identify the top k tags describing a particular track, training the classiers on tags that the users of Last.fm had entered for the tracks artist. There have also been a number of games designed to collect metadata about multimedia. Ellis et al. (2002) described the Musicseer system for collecting ground truth about artist similarity, one aspect of which was a game. In this work, players chose which of a list of artists was most similar to a goal artist. The purpose of the game was to link a starting artist to a goal artist with as short a chain of intermediate similar artists as possible. By performing this forced choice of the most similar artist from a list, triplets of relative

(a) ListenGame

(b) Tag a Tune

(c) MoodSwings

Figure 4: Screenshots from current human computation games for collecting music metadata similarity were collected, which could then be used to infer a full similarity matrix. (von Ahn and Dabbish, 2004) described the ESP Game4 , which asks pairs of players to describe the same image. Once both players agree on a word, they score a certain number of points and move on to the next image. The players attempt to agree on as many images as possible within a time limit. While previous data-collection games maintained data integrity by forcing players to choose from predened responses, this was the rst game to popularize the idea of allowing any response, provided it was veried by a second player.

MajorMiner game design

We designed the MajorMiner game with many goals in mind, but our main goal was to encourage players to describe the music thoroughly. This goal shaped the design of the rules for scoring. Another important goal, which informed the method for introducing new clips into the game, was for the game to be fun for both new and veteran players. Specically, new players should be able to score points immediately, and veteran players should be rewarded with opportunities to score additional points. Other, lesser goals inform the architecture and implementation of the game. The rst of these was the avoidance of the cold start problem. Specically, players should be able to score points as soon as possible after the launch of the game, and a single player should be able to play any time he or she wants, without the need for others to be playing at the same time. Another was to avoid the possibility of cheating, collusion, or other manipulations of the scoring system or, worse, the data collected. Our nal goal was to make the game accessible to as many people as possible, implementing it as a standard web page, without requiring any special plugins, installation, or setup. While many games team one player with a single partner, ours, in a sense, teams one player with all of the other players who have ever seen a particular clip. When a player is paired with a single cooperator, it is possible that the two players could be at very dierent skill levels, have dierent levels of familiarity
7 http://schubert.ece.drexel.edu/moodswings/
with the clip under consideration, or even speak dierent languages, detracting from each players enjoyment of the game. It is also possible that during times of low usage only one player might be online at a time (although this problem has been addressed in other games by replaying recorded sessions). The non-paired format, on the other hand, allows the most compatible, creative, or expert players to cooperate with each other asynchronously, since an obscure description used by one player will remain available until a second player veries it. It also provides a more transparent means of introducing new clips into the labeled dataset, as opposed to pretending that a user can score on a new clip when playing against a mostly prerecorded game. These benets of non-paired games come at the price of vulnerability to asynchronous versions of the attacks that aict paired games. For example, collusion in a paired game is only possible between players who are paired with each other, but it is possible in a non-paired game between any two players who have seen a particular clip.

Scoring

The design of the games scoring rules reects our main goal: to encourage players to thoroughly describe clips in an original, yet relevant way. In order to foster relevance, players only score points when other players agree with them. In order to encourage originality, players are given more points for being the rst to use a particular description on a given clip. Originality is also encouraged by giving zero points for a tag that two other players have already agreed upon. The rst player to use a particular tag on a clip scores two points when it is veried by a second player, who scores one point. Subsequent players do not score any points for repeating that tag. These point allocations (2, 1, 0) need not be xed and could be changed depending on participation and the rate at which new music is introduced into the game. The number of players who score points by verifying a tag could be increased to inate overall scoring and point amounts could also be changed to inuence the general style of play. We have found, however, that this simple scoring scheme suciently satises our goal of encouraging players to be thorough. One concern with this system is that later players could be discouraged if all of the relevant descriptions have already been used by two other players. By carefully choosing when clips are shown to players, however, we can avoid this problem and use the tension created by the scoring rules to inspire originality without inducing frustration. The game has high-score tables, listing the top 20 scoring players over the past day, the past week, and over all time. The principal payo of the game may be the satisfaction of reaching some standing in these tables. Including the short-term-based tables gives even new players some chance to see their names in lights.

Picking clips

When a player requests a new clip to describe, we have the freedom to choose the clip any way we want. This freedom allows us to meet our second goal of making the game fun for both new and experienced players. In order for a player to immediately score on a clip, another player must already have seen it. We therefore maintain a pool of clips that have been seen at least once and so are ready to be scored on. For new players, we draw clips from this pool to facilitate immediate scoring. For experienced players, we generally draw clips from this pool, but sometimes pick clips that have never been seen in order to introduce them into the pool. While such clips do not allow a player to score immediately, they do oer the opportunity to be the rst to
use many tags, thus scoring more points when others agree. While clips must be seen by at least one other person to allow immediate scoring, clips that have been seen by many people are dicult to score on. Since more tags are generally veried for clips that have been seen by more people, it becomes increasingly dicult for players to be original in their descriptions. We alleviate this problem by introducing new clips regularly and preferentially choosing clips that have been seen by fewer players from the pool of scorable clips. In order for players to transition from new to experienced, we dene a continuous parameter that indicates a players experience level. On each request for a new clip, an unseen clip is chosen with probability and a scorable clip is chosen with probability 1 . When the website was initially launched, was dened as the ratio of the number of clips a player had seen to the number of clips anyone had seen. This prevented problems from developing when the pool of scorable clips was small. After many clips were added in this way, this denition of experience became too strict. We have now transitioned to dening an experienced player as one who has listened to more than 100 clips, at which point reaches its maximum value of 1. A brand 3 new player has a of 0, and linearly increases up to that maximum as the player labels more clips. This scheme for picking clips has a direct impact on the number of times each clip is seen, and hence the overall diculty of the game. The result, derived in Appendix A:, is that at equilibrium, all clips in the scorable pool will have been seen the same number of times, where that number is the reciprocal of the expected value of over all of the players. Computing the expected value of from the usage data in Figure 5(b), each clip would have been seen 59 times under the original picking strategy, but will only be seen 7 times under the new strategy.

Revealing labels

Seeing other players descriptions is part of the fun of the game. It also acclimatizes new players to the words that they have a better chance of scoring with. The other responses can only be revealed after a player has nished labeling a given clip, otherwise the integrity of the data and the scoring would be compromised. With this in mind, we designed a way to reveal other players tags without giving away too much information or creating security vulnerabilities. We reveal the tags of the rst player who has seen a clip, a decision that has many desirable consequences. This person is uniquely identied and remains the same regardless of how many subsequent players may see the clip. The same tags are thus shown to every player who requests them for a clip and repeated requests by the same player will be answered identically, even if others have tagged the clip between requests. The only person who sees a dierent set of labels is that rst tagger, who instead sees the tags of the second tagger (if there is one). As described in the previous section, the rst player to tag a particular clip is likely to have more experience with the game. Their descriptions are good examples for other players as an experienced player will generally be good at describing clips, will have a better idea of what others are likely to agree with, and will know what sort of formatting to use. Their tags can thus serve as good examples to others. Also, in order to avoid introducing extra-musical context that might bias the player, we only reveal the name of the artist, album, and track after a clip is nished being labeled. This focuses the player much more on describing the sounds immediately present and less on a preconception of what an artists music sounds like. It is also interesting to listen to a clip without knowing the artist and then compare the sound to the 8

Players

3162 10000

(a) Score

(b) Clips heard

(c) Vocabulary

Figure 5: Histograms of player data. Y-axis is number of players, x-axis is the specied statistic in logarithmic units. (a) Number of points each player has scored, (b) Number of clips each player has listened to, (c) Number of unique tags each player has used. preconceptions one might have about the artist afterward.

Strategy

When presented with a new clip, a player does not know which tags have already been applied to it. Trying one of the more popular tags will reveal how many times that tag has been used and thus the approximate number of times the clip has been seen. If the popular tag has never been used or has been used only once, the player can apply other popular tags and collect points relatively easily. If the tag has already been used twice, however, it is likely to be more dicult to score on the clip. The player must then decide whether to be more original or go on to another clip. This clip-wise strategy leads to two overall strategies. The rst is to be as thorough as possible, scoring points both for agreeing with existing tags and by using original tags that will later be veried. By agreeing with existing tags, the thorough player both collects single-point scores and prevents future listeners from scoring on those tags. By using original tags, the thorough player will setup many two-point scores when subsequent players encounter the same clip. The second strategy is to listen to as many clips as possible, trying to use popular tags on clips that havent been seen before. While having a large number of clips with popular labels is worthwhile, in-depth analysis is more useful for us. To encourage breadth of description, we could add a cost to listening to clips or to posting tags, which would motivate players to use tags they were more certain would be veried by others. Similarly, we could post the high scores in terms of the number of points scored per clip heard or the number of points scored per tag. These adjustments to the scoring philosophy would encourage players to be more parsimonious with their tagging and listening. We have not yet encountered the shallow, speedy strategy and so have not instituted such measures. We have guarded against a few possible exploits of the system. The rst is collusion between two players, or the same person with two dierent usernames. Two players could, in theory, communicate their tags for particular clips to each other and score on all of them. We thwart this attack by making it dicult for players to see clips of their choosing and by adding a refractory period between presentations of any particular clip. Since players can only see their most recent clips, we also never refer to clips by an absolute identier, only by relative positions in the recently seen and recently scored lists, making it more dicult to memorize which

(a) Scored tags

(b) Unique tags

(c) All tags

Figure 6: Histograms of clip data. Y-axis is number of clips, x-axis is the specied statistic. (a) Number of tags that have been used by at least two players on each clip, (b) Number of unique tags that have been applied to a clip, and (c) Number of tags that have been applied to a clip. clips have been seen. Another potential exploit of the system is an extreme form of the speedy strategy in which a player repeatedly uses the same tag or tags on every clip, regardless of the music. This is easily detected and can be neutralized by disabling the oending account.

Data collected

At the time of this papers writing, the site has been live for 11 months, in which time 489 players have registered. A total of 2308 clips have been labeled, being seen by an average of 6.99 players each, and described with an average of 31.24 tags each, 5.08 of which have been veried. See Table 2 for some of the most frequently used descriptions and Figures 5 and 6 for histograms of some statistics of player and clip data, respectively. The system was implemented as a web application using the Ruby on Rails framework. The player needs only a browser and the ability to play mp3s, although javascript and ash are helpful and improve the game playing experience. The page and the database are both served from the same Pentium III 733 MHz with 256MB of RAM. This rather slow server can sustain tens of simultaneous players. The type of music present in the database aects the labels that are collected, and our music is relatively varied. By genre, it contains electronic music, indie rock, hip hop, pop, country, mainstream contemporary rock, and jazz. Much of the music is from independent or more obscure bands, which diminishes the biases and extra context that come from the recognition of an artist or song. See Table 4 for the tags that users of Last.fm have applied to this music collection. Those 2308 clips were selected at random from a collection of 97060 clips, which exhaustively cover 3880 tracks without overlap. The clips that were selected came from 1441 dierent tracks on 821 dierent albums from 489 dierent artists. This means that the average artist had 4.7 clips tagged, the average album had 2.8 clips tagged, and the average track had 1.6 clips tagged. The most frequently seen items, however, had many more clips tagged. Saint Etienne had 35 of their clips tagged, Kula Shakers album K had 12 of its clips tagged, and Oasis track Better Man had 9 of its clips tagged. Certain patterns are observable in the collected descriptions. As can be seen in Table 2, the most popular tags describe genre, instrumentation, and the gender of the singer, if there are vocals. People use descriptive
Label drums guitar male rock electronic pop synth bass female dance techno piano electronica vocal synthesizer slow rap voice hip hop jazz vocals beat 80s fast instrumental

Data normalization

In the initial implementation of the system, tags only matched when they were identical to each other. This was too strict a requirement, as hip hop should match hip-hop in addition to misspellings and other variations in punctuation. Since we still had all of the tagging data, however, it was possible to perform an oine analysis of the tags, i.e. replay the entire history of the game, to compare the use of dierent matching metrics. Below, we describe the metric that we settled on for the matching in the previously collected data. After experimenting on the existing data, we implemented a similar scheme in the live game website and re-scored all of the previous game-play. Our oine data analysis consisted of a number of manual and semi-supervised steps. We began with 7698 unique tags. First, we performed a spell check on the collection of tags, in which misspellings were corrected by hand, reducing the total number of tags to 7360. Then, we normalized all instances of &, and, n, etc, leaving us with 7288 tags. Next, we stripped out all of the punctuation and spaces, reducing the collection to 6987 tags. And nally, we stemmed the concatenated tag, turning plural forms into singular forms and removing other suxes, for a nal count of 6363 tags, a total reduction of 1335 duplicate tags. The rst steps generally merged many unpopular tags with one popular tag, for example all of the misspellings of synthesizer. The later steps tended to merge two popular tags together, for example synth and synthesizer. Merging two popular tags generally aects the scoring of the game overall, while merging orphans improves the game playing experience by making the game more forgiving. Because of the way the rules of the game are dened, merging tags aects the score for a particular clip, increasing it when a single player used each version, but possibly decreasing it if both versions had already been veried. Not only does merging tags aect the score on a particular clip, but it also aects the total number of matching tags in the game. A net increase in scoring means that merging two tags was benecial in uniting players with slightly dierent vocabularies, while a net decrease in scoring means that players might have been using two tags to try to increase their scores without conveying new information. See Table 3 for examples of the tags that were aected the most by merging and how much their merging aected the overall number of veried tags.

Autotagging experiments

Autotagging is the process of automatically applying relevant tags to a musical excerpt. While the system of Eck et al. (2008) selects the best k tags to describe a clip, we pose the problem as the independent classication of the appropriateness of each potential tag. Each tag is generally only applied to a small fraction of the clips, so in this version of autotagging there are many more negative than positive examples. Since the support vector machines (SVMs) we use are more eective when both classes are equally represented in the training data, we randomly select only enough negative examples to balance to positive examples. For
Tags vocal, vocals hip hop, hiphop, hip-hop drum and bass, drum n bass,. beat, beats horn, horns drum, drums 80s, 80s synth, synthesizer, synthesizers
Initial 163+128 139+29+22 16+6+. 125+9 27+15 908+54 111+28 471+162+70

Merged 130 498

Net change +64 +24 +24 +20 +9 205
Table 3: Number of veried uses of each tag before and after merging slightly dierent tags with each other. Merging tags can lead to a reduction in veried uses when several separately-veried variants of a term are merged, leaving just a single veried use. testing, we similarly balance the classes to provide a xed baseline classication accuracy of 0.5 for random guessing. To compare dierent sets of tags, we use the system described by Mandel and Ellis (2008), which was submitted to the MIREX evaluations (Downie et al., 2005) on music classication in 2007. While the results of such experiments will certainly vary with the details of the features and classier used and many music classication systems have been described in the literature (e.g. Lidy et al., 2007; Pachet and Roy, 2007; Turnbull et al., 2006; Eck et al., 2008), these experiments are meant to provide a lower bound on the amount of useful information any autotagging system could extract from the data. Furthermore, our system is state-of-the-art, having achieved the highest accuracy in the MIREX 2005 audio artist identication task and performed among the best in the MIREX 2007 audio classication tasks. The system primarily uses the spectral features from Mandel and Ellis (2005), but also uses temporal features that describe the rhythmic content of each of the clips. For each tag, these features were classied with a binary support vector machine using a radial basis function kernel. The performance of the classiers was measured using 3-fold cross-validation, in which
of the data was used for training and

for testing.

Since some of the tags had relatively few positive examples, we repeated the cross-validation experiment for 3 dierent random divisions of the data, increasing the total number of evaluations to 9. We ensured the uniform size of all training and test sets for classiers that were being compared to each other. When there were more positive or negative clips than we needed for a particular tag, the desired number was selected uniformly at random. We used two sets of tags from MajorMiner. The rst consists of all of the (clip, tag) pairs that had been veried by at least two people as being appropriate. We call this dataset MajorMiner veried. The second consists of all of the (clip, tag) pairs that were submitted to MajorMiner. We call this dataset MajorMiner all. The game is designed to collect the veried dataset, and the inclusion of the complete dataset and its poorer performance shows the usefulness of tracking when tags are veried.

Classication with MajorMiner data
Our rst experiment measured the eect of varying the amount of training data on classication accuracy. We rst evaluated all of the MajorMiner tags that had been veried at least 50 times, sampling 50 clips for those tags that had been veried on more than 50 clips. Since we used a three-way cross-validation, this means that approximately 33 positive examples and 33 negative examples were used for training on each fold. The results can be seen as the smallest markers in Figure 7 and the tags are sorted by their average classication accuracy on this task. Notice that even using 33 positive examples, results are quite good for tags such as rap, house, jazz, and electronica. In general, classication accuracy was highest for genres and lowest for individual instruments. This makes sense because the spectral features we use in the classiers describe overall timbre of the sound as opposed to distinct instruments. One exception to this is saxophone, which can be identied quite accurately. This anomaly is explained by the tags strong correlation with the genre jazz. Tags with intermediate performance include descriptive terms such as dance, distortion, and fast, which are classied relatively well. As the amount of training data increased, so too did classication accuracy. For tags like male, female, rock, dance, and guitar an increase in training data from 33 to 200 tags improved accuracy by 1020 percentage points. This trend is also evident, although less pronounced, in less popular tags like slow, 80s, and jazz. No tag performed signicantly worse when more training data was available, although some performed no better.
Last.fm 75 Last.fm 50 Last.fm 25 MajorMiner verified MajorMiner all
hip hop jazz country rock electronica electronic pop 0.4 0.5 0.6 0.7 0.8 Classification accuracy 0.9
Figure 8: Comparison of classication accuracy between data sets for tags that appeared in all of them. In the evaluation, N = 180, so for the accuracies encountered a dierence of approximately 0.07 is statistically signicant under a binomial model.
Direct comparison with social tags
Certain tags were popular in all of the MajorMiner and Last.fm datasets and we can directly compare the accuracy of classiers trained on the examples from each one. See Figure 8 for the results of such a comparison, in which the tags are sorted by average classication accuracy. Each of these classiers was trained on 28 positive and 28 negative examples from a particular dataset. For ve out of these seven tags the veried MajorMiner tags performed best, but on the rock and country tags the classiers trained on Last.fm 25 and Last.fm 75, respectively, were approximately 8 percentage points more accurate. The veried MajorMiner tags were always classied more accurately than the complete set of MajorMiner tags. Of the three Last.fm datasets, Last.fm 75 just barely edges out the other two, although their accuracies are quite similar. This performance dierence can be attributed to three sources of variability. The rst is the amount that the concept being classied is captured in the features that are input to the classier. This variation is mainly exhibited as the large dierences in performance between tags, as some concepts are more closely tied to the sound than others and of those some are more closely tied to sonic characteristics that are captured in our features. While this source of variability is present in both MajorMiner and Last.fm data, we believe that MajorMiners tags are more sonically relevant because they are applied to clips instead of larger musical elements. Also, the Last.fm datasets contain some extra-musical tags like albums I own, which are generally not discernible from the sound. The second source of variability is inter-subject variability, caused by dierences between individuals conceptions of a tags meaning. Because of its collaborative nature, MajorMiner promotes agreement on tags. Last.fm also promotes such agreement through its count mechanism, which scores the appropriateness of a tag to a particular entity. The dierence in performance between classiers trained on the two MajorMiner datasets shows that veried instances of tags are more easily predicted. A comparison of the Last.fm tags from the three dierent thresholds shows some evidence that tags with a higher count are more easily learned by classiers, although the overall dierences are minor. 16

Dataset MajorMiner veried MajorMiner all Last.fm 50 Last.fm 25 Last.fm 75
Mean 0.672 0.643 0.626 0.624 0.620
Std 0.125 0.109 0.101 0.104 0.111
Table 5: Overall mean and standard deviation classication accuracy for each dataset on its 25 most prevalent tags. All classiers were trained on 40 positive and 40 negative examples of each tag. The nal source of variability is intra-subject variability, caused by the inconsistencies in an individuals conception of a tags meaning across multiple clips. This source of variability is present in all datasets, although it can be mitigated by using expert labelers, who have better-dened concepts of particular musical descriptors and are thus more consistent in their taggings. Even though no expert knows all areas of music equally well, it should be possible to construct a patchwork of dierent experts judgments that is maximally knowledgeable. MajorMiners non-paired (or one-paired-with-all) game-play allows experts in dierent types of music to collaborate asynchronously. As long as they use a distinct vocabulary, it will also select their areas of expertise automatically. It is not clear from our data how many such experts have played the game, but we suspect that this phenomenon will emerge with greater participation.

Overall performance

Finally, we compared the accuracy of classiers trained on the top 25 tags from each of the datasets. The overall mean accuracy along with its standard deviation can be seen in Table 5. The variance is quite high in those results because it includes inter-tag and intra-tag variation, inter-tag being the larger of the two. For a breakdown of performance by tag and cross-validation fold, see Figure 9, from which it is clear that the variance is generally quite low, although exceptions do exist. While these tags are by no means uncorrelated with one another, we believe that it is meaningful to average their performance as they are separate tokens that people chose to use to describe musical entities. While many of MajorMiners players might consider hip hop and rap to be the same thing, they are not perfectly correlated, and certain clips are more heavily tagged with one than the other. For example, while the track shown in Figure 1 might be considered to fall entirely within the hip hop genre, only certain sections of it include rapping. Those sections are particularly heavily tagged rap, while the rest of the song is not. Overall, the average classication accuracy was highest for the top 25 veried tags from MajorMiner. The fact that veried tags make better training data than all tags supports the reasoning behind our design of the scoring rules. Thresholding the count of a tag made little dierence in the mean accuracy of the Last.fm datasets, although there were dierences between the datasets performances on specic tags. For low thresholds, there are a small number of tags that perform quite well, while most perform poorly. As the threshold is increased, there are fewer stand-out tags and the classication becomes more consistent across tags. The top 25 tags for our clips on Last.fm do not include musical characteristics or instruments, but do include some extra-musical tags like albums I own, indie, british, and 90s. Such tags are not well classied by our system because they have less to do with the sound of the music than with the cultural 17

Acknowledgments

The authors would like to thank Johanna Devaney for her help and Douglas Turnbull, Youngmoo Kim, and Edith Law for information about their games. This work was supported by the Fu Foundation School of Engineering and Applied Science via a Presidential Fellowship, by the Columbia Academic Quality Fund, and by the National Science Foundation (NSF) under Grants IIS-0238301 and IIS-0713334. Any opinions, ndings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reect the views of the NSF.

Appendix A:

Derivation of clip seen distribution
Section 2.2 sets out a simple rule for choosing clips to present the a player: with probability a clip is selected that no other player has seen before. Otherwise, the scorable clip that has been seen by the fewest number of other players is selected. The way in which grows with user experience induces a distribution over the number of times a clip will be seen. In designing this growth, it is useful to have a model of this relationship and we can create this model from a probabilistic formulation of user experience and past useage data from the game.
To perform this calculation, we dene a Markov chain with states n N. Clips move around in this state space such that a clips state represents the number of users that have seen that clip. Let xn be the number of clips that are in state n, i.e. the number of clips that have been seen n times. The system has x0 = and begins with all other xn = 0. When player i requests a clip, a coin is ipped and with probability i the player is given a new clip, which moves from state 0 to state 1. Otherwise the player is given the scorable clip that has been seen by the fewest other players, moving it from n to n + 1 where n > 0 is the lowest populated state. Assume for the moment that all users have the same probability of receiving a new clip, i.e. i = , i. Then at equlibrium, only two states are occupied, n
and n + 1. The occupancy of state n is
xn = N n and the occupancy of state n + 1 is xn+1 = N xn. This holds true even when = E[i ], where the expectation is taken over the true distribution of i values seen in the game. In this case, at equilibrium, n = heard. This analysis assumes that every player is equally likely to select the next clip, i.e. that there is no correlation between the players that see successive clips. In general, this is not the case, as one person will generally play for a certain amount of time, creating a burst of clips seen by a single player. The analysis does, however, describe the gross eects of these picking strategies.

1 E[i ]

and all clips will have been seen either n or n + 1 times. In quantifying a users
experience level, we are dening the function i = (ci ), where ci is the number of clips player i has already

References

K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D.M. Blei, and M.I. Jordan. Matching words and pictures. Journal of Machine Learning Research, 3(6):11071135, 2003. A. Berenzweig, D. P. W. Ellis, and S. Lawrence. Anchor space for classication and similarity measurement of music. In Proc Intl Conf on Multimedia and Expo (ICME), 2003. G. Carneiro and N. Vasconcelos. Formulating Semantic Image Annotation as a Supervised Learning Problem. In Computer Vision and Pattern Recognition, volume 2, pages 163168, 2005. S. J. Downie, K. West, A. Ehmann, and E. Vincent. The 2005 music information retrieval evaluation exchange (MIREX 2005): Preliminary overview. In J. D. Reiss and G. A. Wiggins, editors, Proc. Intl. Symp. Music Information Retrieval, pages 320323, September 2005. D. Eck, P. Lamere, T. B. Mahieux, and S. Green. Automatic generation of social tags for music recommendation. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20. MIT Press, 2008. D. P. W. Ellis, B. Whitman, A. Berenzweig, and S. Lawrence. The quest for ground truth in musical artist similarity. In Proc. Intl. Symp. Music Information Retrieval, 2002. Y. E. Kim, E. Schmidt, and L. Emelle. MoodSwings: A collaborative game for music mood label collection. In Proc. Intl. Symp. Music Information Retrieval, September 2008. To appear. E. L. M. Law, L von Ahn, R. B. Dannenberg, and M. Crawford. TagATune: A game for music and sound annotation. In S. Dixon, D. Bainbridge, and R. Typke, editors, Proc. Intl. Symp. Music Information Retrieval, pages 361364, 2007. T. Lidy, A. Rauber, A. Pertusa, and J. M. Iesta. Improving genre classication by combination of audio and symbolic descriptors n using a transcription system. In S. Dixon, D. Bainbridge, and R. Typke, editors, Proc. Intl. Symp. Music Information Retrieval, September 2007.
M. I. Mandel and D. P. W. Ellis. Song-level features and support vector machines for music classication. In J. D. Reiss and G. A. Wiggins, editors, Proc. Intl. Symp. Music Information Retrieval, pages 594599, September 2005. M. I. Mandel and D. P. W. Ellis. Multiple-instance learning for music information retrieval. In Proc. Intl. Symp. Music Information Retrieval, September 2008. To appear. F. Pachet and P. Roy. Exploring billions of audio features. In International Workshop on Content-Based Multimedia Indexing(CBMI), pages 227235, 2007. R. E. Thayer. The Biopsychology of Mood and Arousal. Oxford University Press, USA, September 1990. D. Turnbull, L. Barrington, and G. Lanckriet. Modeling music and words using a multi-class naive bayes approach. In Proc. Intl. Symp. Music Information Retrieval, October 2006. D. Turnbull, L. Barrington, D. Torres, and G. Lanckriet. Towards musical query-by-semantic description using the CAL500 data set. In ACM Special Interest Group on Information Retrieval (SIGIR), 2007a. D. Turnbull, R. Liu, L. Barringon, and G. Lanckriet. A game-based approach for collecting semantic annotations of music. In S. Dixon, D. Bainbridge, and R. Typke, editors, Proc. Intl. Symp. Music Information Retrieval, pages 535538, September 2007b. L. von Ahn and L. Dabbish. Labeling images with a computer game. In Proc SIGCHI conference on Human factors in computing systems, pages 319 326, 2004. B. Whitman and D. Ellis. Automatic record reviews. In Proc. Intl. Symp. Music Information Retrieval, 2004. B. Whitman and R. Rifkin. Musical query-by-description as a multiclass learning problem. In IEEE Workshop on Multimedia Signal Processing, pages 153156, 2002.