Lyric Text An Animated Display of Song Lyrics

Lyric TextAn Animated Display of Song LyricsRob Diaz-Marino683CPSCDr. S. CarpendaleSupervisor:Greenberg Saul1.0 IntroductionText is the means by which we express literal language when the use of voice is not possible, suitable, or feasible. In this paper, the purpose of text is to explain ideas. Whether you read the paper yourself or have someone read it to you word-for-word, the effect remains the same. Fortunately, scientific papers usually consist of well-formed text, which is well thought out and written to promote a word smooth flow and avoid ambiguity. Ambiguity can be caused by insufficient explanation, grammatical errors, and spelling errors.Speech, on the other hand, is a spontaneous interaction that is not nearly as well thought out. When we set out to say something, we ordinarily do not have an exact sentence in our head – merely an idea of what we want the utterance to mean. Even prepared speeches are customarily written out in point form. Transcriptions of speech often results in text that is riddled with ambiguity, grammatical errors, and awkward wording that would be unacceptable as a serious piece of writing unless it is identified as a quote. Even then it is sometimes necessary to introduce extra punctuation and minor word changes to make it sound reasonable.Why is it so difficult to understand transcribed speech when it is trivial for us to understand spoken language? Birdwhistell [6] suggests that the information conveyed by words amounts to only 20-30% of the information conveyed in a conversation. By transcribing text, we lose a great deal of incidental information: Visual cues such as body language and posture [1], as well as auditory cues such as intonation, pace, volume, and other vocal qualities. All these factors help us to understand sentences that are less than perfectly well-formed.2.0 BackgroundThe concept of Kinetic Typography has been used to set mood and direct attention in Film and Television Advertising for many years now [3]. Only recently have Computer Scientists started seriously investigating the expressive power of text animation.The Kinedit System [2] provides a framework for easily producing a wide range of text animations in order to promote further study of the capabilities of Kinetic Text. It provides a set of combinable, parameterized animation effects that can be applied to substrings within a larger text body. The animations are used to reveal the entire text body bit by bit.Several papers [3, 4, 5] have already jumped into trying to apply Kinetic Text to online messaging applications, presumably to make the expressiveness of these applications more closely resemble that of true speech interactions. The main drawback is thatadditional effort must be put into the manual selection of text animations, which can be an unwanted burden.I feel that there are plenty of non-real-time textual applications that could greatly improve with the use of Kinetic Text.2.1 Closed CaptioningClosed captioning provides a transcription of Television dialogue and sound for those that are hard of hearing or have language difficulties. This text is commonly revealed in chunks, or as gradual reveal with vertical line scrolling. The table belowsummarizes some of the similarities and differences between these two revealmethods. Special text annotations that are used such as explicit descriptions ofsounds and character labels are done inconsistently from programme to programme. Reveal Method Synchronization Emotive ContentSound Content Chunked Coarse, by Paragraph or Sentence Gradual Steady reveal ratecan lag and leaddepending on speedof speech.Text,Punctuation,Character Expressions Punctuation, Explicit Descriptions ex. (whispering) Character Labels ex. Rob: …Table 1: Chunked vs. Gradual text reveal 2.2 Video GamesOnly recently with the use of larger storage media such as CDs and DVDs have some games used speech recordings. Traditionally, character dialogue has been presented in a textual form as a space-saving alternative to sound recordings of characterspeech. Kinetic Text has been utilized in a very limited fashion in some games, such as oscillation of the dialogue window to represent a character being startled, or larger fonts to indicate yelling.In essence, Character dialogue exhibits the properties of well-formed text, and thus can seem contrived and unrealistic at times. Emotive content is most prominently provided through character animations, though not necessarily synchronized to the text reveal in a meaningful way. Character labels are consistently provided alongside of dialogue windows, however the actual voices of the characters were mostly left to the imagination.2.3 Music Lyrics VisualizationsApplications such as Karaoke utilize a “bouncing ball” or changing text color tosynchronize lyric presentation with music. This text is usually revealed in a chunk so that singers have the ability to read ahead and gain pre-emptive information.However, it is assumed that the singer already knows the tune – no pre-emptiveinformation is provided for the musical properties of the song. Timing information is only visualized at the exact moment the text is supposed to be sung (or with a veryshort offset to allow the singer time to visually process the indication). Long pauses are denoted explicitly (ex. 8-bar instrumental interlude), or a series of non-textsymbols that serve to count down to the next section of lyrics.3.0 Project TopicI began this project by wanting to develop a basic kinetic text system to visualize intonation in dialogue using visual variables such as font size, color, vertical positioning, and motion. My goal was to achieve the effect that viewers not only see what was being said, but also how it was said. Such cues could convey incidental information about the mood of the speaker, or even give insight into their speech patterns and voice. My primary concern with this direction was that it would be difficult to create the dialogue intonation data.This pushed me in the direction of visualizing song lyrics, as it occurred to me that music has non-arbitrary timing and a discrete set of pitch levels. Lyric visualization has slightly different requirements from dialogue visualization however. With regular speech, we do not know what is going to be said until it is actually said, so it may be favorable to adopt the same policy in dialogue visualization. On the other hand, with lyric visualization, pre-emptive data is required in order to prepare the singer for what they will be singing in the immediate future.4.0 Final ProductUnfortunately during the course of programming my project, I uncovered an API bug in Managed DirectDraw that limited my ability to change font face, size, and style. For this reason I had to branch into other visual mechanisms to satisfy the dimensionality of my song data.4.1 File Data StructurePersistent storage of my song data was in the form of an XML file. The initial data was created by hand and gradually refined as the program developed further.Fortunately the XML data for this song is not program-dependent, and it can bereused for any future visualizations.The following data are represented in the XML file:•General Song Information – song title, artist name, song filename. For this project I used the song “Feel Good Time” by Pink.•Tempo Data – time indices coupled with an indication of Beats per Minute (BPM), Measure (beats per bar), and snap resolution (number of notespossible per beat).•Lyric Data – the actual lyric text, divided into verses (regular or chorus). The text contains pipe delimiters to split words into syllables, and carriage returnsto delimit lines.•Timing Data – a sequential set of data elements to indicate timing for each syllable from the text. Here, timing is indicated by a reference to a tempoentry, the bar number, beat number, fraction (corresponding to the snapresolution), and duration.•Segment Data – a sequential set of sound segments that contain a start time and duration indication the same as with the timing data, as well as a pitch,blend function, and force.Blend functions are as follows:•Stab – indicates a short burst of sound, such as the word “now”.•Sustain – indicates a sound that is maintained until the next segment begins.This occurs in words where a syllable ends in a vowel, such as “mo” in theword “mo|ney”. It also can occur with any other syllable that is purposelyheld until the beginning of the next syllable, such as “our” in the sequence“our time”.Force variables describe the type of voice to be used. From softest to loudest, they are as follows: Whisper, Soft, Spoken, Sonorant, Hard, Yell, and Scream.4.2 Program Data StructuresThe lyrics are stored in a hierarchical data structure consisting of Verses, which contain Lines, which contain Words, which contain Syllables. Syllable timing data is stored in a flat list of Span objects, but contain references to their corresponding Syllable and vice versa. Segments are contained in a flat list independent of the lyric and timing data.All time positions in the XML file are converted to exact time indices and lengths when loaded. Tempo entry, bar, beat, and fraction are used to calculate the starting time index, while duration is used to calculate the note length.4.3 VisualizationsFigure 1: Screenshot from LyricTextPast, Present, and FuturePre-emptive, current, and expired lyrics are shown in sequence in the bar along the bottom of the screen. Expired data is grayed out, current data is highlighted inyellow, and pre-emptive data is displayed prominently in white.Beat and MeasureAs a type of metronome functionality, I visualize the beat and measure as apulsating dot. In the Figure 1 above, the dot flashes blue at the beginning of each beat, while the beginning of each measure is seen as a red ring surrounding the blue dot.PitchUpon loading the file, a maximum and minimum pitch is found and used as avertical range for positioning all notes in between. Notes closer to the top of the screen indicate higher pitches, while notes near the bottom bar are lower pitches.When syllables in the same word are sung at different pitches, the word is split over the vertical range. To combat this visual discontinuity, I use lines to connectsyllables that are sung at different pitches. Syllables that are at the same pitch are not connected with lines as they already appear side by side.Figure 2: Syllables at different pitches are connected by lines to prevent visual discontinuity.Syllable / Word DurationAs each syllable expires, the text in the main area of the screen is shifted to the left to make room for the next syllable being displayed. Initially the newest syllable appears yellow (see Figure 1), but fades linearly to either gray or white depending on the blend function. The duration of this fade corresponds to the temporal duration of the syllable.SegmentsThe beginning of each new segment is marked with a ripple that emerges in the background from the center of the syllable. Color, thickness, and size of the ripple all vary based on the blend function and force. The duration of the ripples equates to the duration of the current syllable and the next 2 to follow – this allows for 3 ripples to be present on the screen simultaneously, for a more visually pleasing effect than simply one ripple at a time.Blend FunctionsAs shown in Figure 2 above, stabbed syllables fade to gray while sustained syllables fade to white. The connecting line takes on the color of the previous syllable’s blend function. Less prominently, the segment ripples appear thicker and spread out further for sustained syllables than for stabbed ones.Vocal ForceVocal force is visualized through the use of ripple color. Rainbow colors correspond to force levels, where Whisper is shown as dim purple and Scream is shown as vivid red. In the song data used for this project, only the Spoken (blue), Sonorant (green), and Yell (yellow) levels occur.Blue = Spoken Green = Sonorant Yellow = YellFigure 3: Ripple colors indicate different vocal force levelsLead-InWhen pauses of more than 1 second occur between syllables, a lead-in indicator is used to visualize the span of time the singer must wait. This occurs in the form of a thick yellow ring that fades in and converges on the beat and measure indicators. When the ring reaches the center, the waiting time has expired.Figure 4: Lead-in indicator visualizes pauses between lyrics5.0 Analysis and Future WorkMy project has several shortcomings that I hope to address in future versions.Duplication of TextAs it currently stands, this program is better directed toward a passive user whomerely watches rather than sings. This results from several problems. The bar along the bottom of the screen is the only source of pre-emptive data; however it iscompletely separate from the visualizations which appear in the main area of the screen. This discontinuity makes it difficult to get both a sense of what is to come and how the current text is being sung.My solution is to eliminate the need for the bottom bar by placing the pre-emptive text in the main panel. It would appear vertically offset by pitch, and perhaps employ color and shadow to also show blend function and force in a non-animated fashion. Lack of Time and Pitch ScaleThe horizontal space is allocated based on the width of syllable text, not as a time scale. A user would get a better sense of timing if the X-axis were meaningfully connected to the start time and duration of each syllable. Similarly, the user would get a better sense of pitch if the Y-axis showed labeled horizontal lines at each note level.6.0 ConclusionFrom this project I discovered that visualizing song lyrics has a different set ofchallenges from that of visualizing dialogue. I created a program that visualized auditory aspects of song lyrics that are not apparent from plain text transcriptions. I also demonstrated that Kinetic Typography can be successfully combined with other simple animation effects that do not disrupt the readability of the text.7.0 References1)Cassell, J., Vilhjálmsson, H. H., Bickmore, T. (2001) BEAT: the BehaviorExpression Animation Toolkit. Proceedings of ACM SIGGRAPH 2001, 477-486.2)Forlizzi, J., Lee, J., Hudson, S. E. (2003) The Kinedit System: AffectiveMessages Using Dynamic Texts. CHI Letters, 5 (1), 377-384.3)Wang, H., Prendinger, H., Igarashi, T. (2004) Communicating Emotions in OnlineChat Using Physiological Sensors and Animated Text. Proceedings of CHI 2004, 1171-1174.4)Bodine, K. Pignol, M. (2003) Kinetic Typography-Based Instant Messaging.Proceedings of CHI 2003, 914-915.5)Möhler, G., Osen, M., Harrikari, H. (2004) A User Interface Framework forKinetic Typography-enabled Messaging Applications. Proceedings of CHI 2004, 1505-1508.6)Birdwhistell, R. (1970) Kinetics and Context: Essays on Body MotionCommunication. University of Pennsylvania Press, 1970.。