CS4706 Spring 2011 - Project 1 - "Building a TTS System"

Submission procedure explained below.

Part A: Setup and first small TTS system

A.1. Setup

Add these lines to your ~/.bashrc file:

export PATH=/proj/speech/tools/festival/festival/bin:$PATH
export PATH=/proj/speech/tools/festival/speech_tools/bin:$PATH
export FESTVOXDIR=/proj/speech/tools/festival/festvox
export ESTDIR=/proj/speech/tools/festival/speech_tools
 

Remember that you have to log out and back in for the changes to the ~/.profile file to take effect. You can check if the changes are in effect by running echo $FESTVOXDIR. The value specified above should be displayed.

A.2. Small TTS system: a talking clock

NB:  Everyone must do Part A.  Teams will turn in all 2-3 talking clocks with the team submission.

Log on locally to a Linux computer in the Speech Lab. Open a Terminal window (Applications » Accessories » Terminal) and run the following commands, replacing USERNAME with the user name of your CS account (e.g. fb2175).

In http://festvox.org/bsv/x1003.html you can find a detailed explanation of each step.

Documentation

• » Festival & FestVox home page
• » Building Synthetic Voices

Requirements:

1. To grade your work, we need to run your TTS system. Therefore, please double check that all your files under /proj/speech/users/cs4706/USERNAME have read permissions for everybody, and also exec permissions in the case of directories (that is the default setting, so if you have not chmoded anything, you do not need to change anything).
2. You should generate three wav files using your TTS with the following procedure:

   cd /proj/speech/users/cs4706/USERNAME/time
   festival
   
   (load "festvox/SLP_time_xyz_ldom.scm")
   (voice_SLP_time_xyz_ldom)
   
   (Parameter.set 'Audio_Method 'Audio_Command)
   (Parameter.set 'Audio_Required_Rate 16000)
   (Parameter.set 'Audio_Required_Format 'wav)
   
   (Parameter.set 'Audio_Command "cp $FILE time1.wav")
   (saytime)
   
   (Parameter.set 'Audio_Command "cp $FILE time2.wav")
   (saythistime "07:57")
   
   (Parameter.set 'Audio_Command "cp $FILE time3.wav")
   (saythistime "14:22")
    

Part B: Preparing your limited-domain TTS

B.1. Choosing a limited domain:

In Part A, you built a simple talking-clock TTS system. Now, you will build a TTS system for the Spoken Dialogue System application you have chosen. And, instead of recording it using a neutral voice, you may want to choose a particular style or personality that you think is most appropriate for your domain and application.

B.2. Designing the input and output of your TTS system

Define as formally as possible what the input and output of your TTS system is going to look like.

• For example, in the talking clock the input is a string of the form HH:MM, and the output is a sentence of the form:

  The time is now, EXACTNESS MINUTE INFO(, in the DAYPART).

  where:

  EXACTNESS = {exactly, just after, a little after, almost}
  MINUTE = {-, five past, ten past, quarter past, twenty past, twenty-five past, half past, twenty-five to, twenty to, quarter to, ten to, five to}
  INFO = {one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, midnight}
  DAYPART = {morning, afternoon, evening}

  Every time the talking clock receives an input like 07:57, it will translate it into a sentence like The time is now, a little after five to eight, in the morning, and later synthesize it.

  This TTS has four degrees of freedom (EXACTNESS, MINUTE, INFO, DAYPART). The number of possible sentences is approximately 4 x 12 x 12 x 2 = 1152.

Your limited domain must have at least five degrees of freedom, and you have to provide an estimate of the number of possible sentences that could be generated.

B.3. Setting things up

Log on to your CS account (NB: /proj/speech/ is only available from the following machines:  chat, felix, fluffy, veu, voce, voix, and lincoln) and run the following commands, where USERNAME is the user name of your CS account (e.g. fb2175), and TOPIC is a string such as 'number', 'weather', 'street', etc.:

B.4. Designing the prompts

Next, you need to design the prompts for your TTS system. As you saw in Part A, the talking clock uses the prompts in time/etc/time.data:

Now, you have to create a similar file for your domain, and save it as partc/etc/TOPIC.data.  NOTE:  The spaces after '(' and before ')'  in each line are critical.

For an explanation on how to design the prompts, go to http://www.festvox.org/bsv/c941.html#AEN952.

Documentation

• » Festival & FestVox home page
• » Building Synthetic Voices
• » Designing the Prompts

Part C: Completing your limited-domain TTS

C.1. Introduction

In Part B, you started building your limited-domain TTS system. You defined its input and output, designed the set of prompts you will record.

Now, in order to complete your TTS system you need to:
a) record the set of prompts (section C.2);
b) write a script that transforms an input string into an English sentence, and sends it to Festival to synthesize it (section C.3).

C.2. Recording and processing the prompts

Log on locally to a Linux computer in the Speech Lab. Open a Terminal window (Applications » Accessories » Terminal) and run the following commands, replacing USERNAME with the user name of your CS account (e.g. fb2175), and TOPIC with a string such as 'number', 'weather', 'street', etc.

Before starting, read the tips for part C, which you may find useful.

C.3. Writing a script that transforms the input and synthesizes it

Now, you need to write a script that receives an input string as you defined in Part B, and transforms it into an English sentence that your TTS system can synthesize. For example, in the time domain the script would transform a string like "14:22" into a sentence like "The time is now, a little after twenty past two, in the afternoon." A simple (and perhaps long) case or switch statement should be enough to achieve this.
You must test your script properly before submission (in one of the speech-lab machines). If you sibmit a not working script, you will lose a lot of points, and we will not debug your code.

Use this Perl script as a basis. Update the variables $USERNAME and $TOPIC, complete the code where marked, and define the function generate_sentence, which does the input-output transformation. Please comment your code thoroughly.

The rest of the code in this script creates a temporary Festival script and runs it. That temporary Festival script loads your limited domain and creates a wav file with the resulting synthesis (you did exactly the same thing by hand in Part A for the time domain). You should not need to modify any of this.

Note: It is also possible to do the transformation part of the script (input string to English sentence) using the Scheme programming language, and later import your script into Festival. If you want to do it that way, please check with the TA first.

Documentation

• » Festival & FestVox home page
• » Building Synthetic Voices

Submission

1. For Part A, save each team member's three wave files: time1.wav, time2.wav, and time3.wav (see part A) in a subfolder named <YourUni-parta>.  All 2 or 3 of these subfolders should be put into a subfolder named parta.  So you should have a folder names parta which contains subfolders for each team members own clock.

2. For Part B, save the following files in partb subfolder:

• A text document specifying:
  1. a description of your limited domain as currently defined (5 sentences at most)
  2. the definition of the input and output of your TTS system, specifying the degrees of freedom and the number of possible sentences that could be generated.
• The file TOPIC.data which contains the prompts you have designed.

• A plain text document (not Word, PDF, etc.) specifying:
  1. path and name of the Perl script you did in section C.3.
  2. an example of how to run your perl script (must work on one of our speech lab machines).
  3. a description of any changes you made to your design in Part B, and an explanation of why you think they were necessary.
  4. a description of any special features of your system, e.g. of your grammar, domain coverage, which you think are particularly interesting or sophisticated.
  5. a paragraph or two on what it would require to turn your TTS system into a real application in your domain and the difficulties you might encounter in doing so.
• Do not submit any wav files.
   
• A shell script sample.sh that generates three wav files (from three different inputs you choose). That is, given three inputs a, b and c, this script should generate a wav file for each input. Three wav files in total.
  Choose the inputs that you consider show your system to its best advantage.
  The script should look like this:
  
  #!/bin/bash
  
  # path and name of the tts script (**update**)
  SCRIPTDIR=/proj/speech/users/cs4706/USERNAME
  SCRIPTNAME=tts.pl
  
  # the resulting wav files are saved in the present directory
  DESTDIR=`pwd`
  
  # generate a wav file for input #a
  $SCRIPTDIR/$SCRIPTNAME "03:39" $DESTDIR/utt1a.wav
  
  # generate a wav file for input #b
  $SCRIPTDIR/$SCRIPTNAME "14:47" $DESTDIR/utt1b.wav
  
  # generate a wav file for input #c
  $SCRIPTDIR/$SCRIPTNAME "23:55" $DESTDIR/utt1c.wav
   

Tips for recording (for parts A and C only)

• Do not start without testing the microphone! To test it, try:
  rec -s test.wav
  play test.wav
  Change the settings in the Volume Control (Double click on the speaker in the upper-right corner) until your recorded speech is clear and loud enough, with as little background noise as possible.
• Try to make the recordings in a silent environment.
• Make sure that your microphone is not directly in front of your mouth, to avoid noise produced by your breath.
• The script bin/prompt_them in step 4 only lets us record all prompts in one sitting. If you make mistakes and want to record again a few specific prompts, you do not need to run that script again. Instead, follow these instructions (from the shell, not in Festival):
   
  1. Let PROMPT be the id of the prompt you want to record again (e.g. time0001).
  2. Run the command $ESTDIR/bin/ch_wave -info wav/PROMPT.wav
  3. Let DURATION be the duration in seconds of the wav file, which is given in the first line of the output.
  4. Run the command $ESTDIR/bin/na_record -f 16000 -time DURATION -o wav/PROMPT.wav -otype riff
     This will start recording immediately for the specified amount of seconds, and will replace the file wav/PROMPT.wav with the new recording.

Further tips for part C

• (This is very specific, most probably you will not need it.)
  If you want to ellicit the same word (or words) with a different intonation depending on its position in the sentence, you will find that Festival does not take word position into account, with one exception: a word immediately before a pause (comma, period) will be pronounced differently. For example, in "Pin number, one one one." Festival differentiates between the first two and the third "one", but not between the first and the second.
  Therefore, we have to find a way to change the pronunciation of a word, depending on its position.
  Suppose we have this set of prompts:
  ( pin0001 "Pin number, one twoh threehh." )
  ( pin0002 "Pin number, two threeh onehh." )
  ( pin0003 "Pin number, three oneh twohh." )
  Numbers in the second position have one extra h, and numbers in the third position have two extra h's.
  This lets us ellicit each number differently, depending on its position. The h's might mess up the synthesizer that reads out the sentences before recording, but do not seem to affect the rest of the process.
  Then, to access the different word instances when synthesizing speech, we have to use the correct version (e.g. one or oneh or onehh). Our prompts to the TTS system will look like:
  ( SayText "Pin number, two twoh onehh" )
  ( SayText "Pin number, three threeh twohh" )
  etc.