Configuring for ToBI annotations
Jonathan Harrington
WiSe 2023
- 1 Preliminaries
- 2 Overall aim
- 3 Forced alignment
- 4 Configuring a database for ToBI (tones and break indices) annotation
1 Preliminaries
If you have not already done so as a consequence of completing earlier modules, follow the setup instructions here.
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.3 ✔ readr 2.1.4
## ✔ forcats 1.0.0 ✔ stringr 1.5.0
## ✔ ggplot2 3.4.3 ✔ tibble 3.2.1
## ✔ lubridate 1.9.3 ✔ tidyr 1.3.0
## ✔ purrr 1.0.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors##
## Attaching package: 'emuR'
##
## The following object is masked from 'package:base':
##
## norm2 Overall aim
Starting out with just a text collection the aim is to create an Emu database that is useful for tones-and-break-indices annotation. The more specific aim is to configure the database to obtain a configuration as in Fig. 2.1.
Figure 2.1: A ToBI annotation.
The starting point will be the text collection of Albanian data analysed in the previous module.
3 Forced alignment
The first task is to apply forced alignment to these Albanian data. The commands will be presented in this section without detailed comment given that these were discussed in the previous module.
The resulting Emu database will be stored as alb2_DB (to distinguish it from the one created in the last module).
# the path of the text collection of Albanian utterances
path.albanian = file.path(sourceDir, "albanian")
# convert it to an Emu database called alb2_DB
convert_txtCollection(dbName = "alb2",
sourceDir = path.albanian,
targetDir = targetDir)## INFO: Parsing plain text collection containing 4 file pair(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Copying 4 media files to EMU database...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%## INFO: Loading EMU database from ./emu_databases/alb2_emuDB... (4 bundles found)
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%# run forced alignment
runBASwebservice_all(alb2_DB,
transcriptionAttributeDefinitionName = "transcription",
language = "sqi-AL",
runMINNI = F)## INFO: Preparing temporary database. This may take a while...
## INFO: Checking if cache needs update for 1 sessions and 4 bundles ...
## INFO: Performing precheck and calculating checksums (== MD5 sums) for _annot.json files ...
## INFO: Nothing to update!
## INFO: Sending ping to webservices provider.
## INFO: Running G2P tokenizer on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running G2P on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running MAUS on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running Pho2Syl (canonical) on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running Pho2Syl (segmental) on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Autobuilding syllable -> segment links from time information
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%4 Configuring a database for ToBI (tones and break indices) annotation
The database currently has an ITEM tier ORT that dominates an ITEM tier MAS containing syllabifications that dominates the SEGMENT tier MAU containing the phonetic segmenation as shown by either serve(alb2_DB, useViewer=F) or by summary(alb2_DB). The task is to add a tier Tone for marking intonational events and to add a new SEGMENT tier, ORT2 showing word segmentations and their times. The tonal events of the Tone tier and annotations of the ORT2 tier are to be linked automatically, just as in Fig. 2.1.
Here are the steps needed:
- Create the
SEGMENTtierORT2containing word annotations linked to times. - Create a
Tone tierand link it toORT2. - Calculate and display f0.
- Annotate the
Tonetier and link the annotations automatically to word annotations.
4.1 Creating and displaying the SEGMENT tier ORT2 with word annotations
This requires four steps:
- Make a
SEGMENTtier calledORT2. - Make a segment list of all words in the database using
query(). - Add the word annotations to
ORT2. - Display the annotations of
ORT2time-aligned to the signals.
4.1.1 Make a SEGMENT tier called ORT2
As discussed in an earlier module, this can be done with the add_levelDefinition() function, thus:
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%## name type nrOfAttrDefs attrDefNames
## 1 bundle ITEM 2 bundle; transcription;
## 2 ORT ITEM 3 ORT; KAN; KAS;
## 3 MAU SEGMENT 1 MAU;
## 4 MAS ITEM 1 MAS;
## 5 ORT2 SEGMENT 1 ORT2;4.1.2 Make a segment list of all words in the database
This can be done with the query() function.
# find all words in the database and get their start and end times.
text.s = query(alb2_DB, "[ORT =~ .*]")
text.s## # A tibble: 17 × 16
## labels start end db_uuid session bundle start_item_id end_item_id level
## <chr> <dbl> <dbl> <chr> <chr> <chr> <int> <int> <chr>
## 1 lena 1050. 1670. 4cf4d0a1… 0000 0001B… 2 2 ORT
## 2 leu 1670. 2100. 4cf4d0a1… 0000 0001B… 3 3 ORT
## 3 nje 2100. 2290. 4cf4d0a1… 0000 0001B… 4 4 ORT
## 4 mur 2290. 2770. 4cf4d0a1… 0000 0001B… 5 5 ORT
## 5 malena 1300. 2190. 4cf4d0a1… 0000 0001B… 2 2 ORT
## 6 leu 2190. 2670. 4cf4d0a1… 0000 0001B… 3 3 ORT
## 7 murin 2670. 3420. 4cf4d0a1… 0000 0001B… 4 4 ORT
## 8 marilena 1090. 2110. 4cf4d0a1… 0000 0001B… 2 2 ORT
## 9 leu 2110. 2500. 4cf4d0a1… 0000 0001B… 3 3 ORT
## 10 njomezen 2500. 3420. 4cf4d0a1… 0000 0001B… 4 4 ORT
## 11 lena 900. 1370. 4cf4d0a1… 0000 0001C… 2 2 ORT
## 12 leu 1370. 1620. 4cf4d0a1… 0000 0001C… 3 3 ORT
## 13 njomezen 1620. 2400. 4cf4d0a1… 0000 0001C… 4 4 ORT
## 14 jo 2550. 2840. 4cf4d0a1… 0000 0001C… 5 5 ORT
## 15 lena 2840. 3280. 4cf4d0a1… 0000 0001C… 6 6 ORT
## 16 leu 3280. 3550. 4cf4d0a1… 0000 0001C… 7 7 ORT
## 17 murin 3550. 4290. 4cf4d0a1… 0000 0001C… 8 8 ORT
## # ℹ 7 more variables: attribute <chr>, start_item_seq_idx <int>,
## # end_item_seq_idx <int>, type <chr>, sample_start <int>, sample_end <int>,
## # sample_rate <int>The attribute and name of the tier from which this segment list was derived is ORT as shown by this:
## [1] "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT"
## [13] "ORT" "ORT" "ORT" "ORT" "ORT"## [1] "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT"
## [13] "ORT" "ORT" "ORT" "ORT" "ORT"These all need to be changed to the name of the new SEGMENT tier ORT2. This can be done as follows:
text.s$attribute = paste(text.s$attribute, "2", sep = "")
text.s$level = text.s$attribute
text.s$attribute## [1] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"
## [11] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"## [1] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"
## [11] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"4.1.3 Add the annotations to ORT2
The next step is to add the information from text.s to the ORT2 tier. This is done with the function create_itemsInLevel(), thus:
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%So it should now be possible to query the word annotations directly from ORT2:
## # A tibble: 17 × 16
## labels start end db_uuid session bundle start_item_id end_item_id level
## <chr> <dbl> <dbl> <chr> <chr> <chr> <int> <int> <chr>
## 1 lena 1050. 1670. 4cf4d0a1… 0000 0001B… 26 26 ORT2
## 2 leu 1670. 2100. 4cf4d0a1… 0000 0001B… 27 27 ORT2
## 3 nje 2100. 2290. 4cf4d0a1… 0000 0001B… 28 28 ORT2
## 4 mur 2290. 3665. 4cf4d0a1… 0000 0001B… 29 29 ORT2
## 5 malena 1300. 2190. 4cf4d0a1… 0000 0001B… 28 28 ORT2
## 6 leu 2190. 2670. 4cf4d0a1… 0000 0001B… 29 29 ORT2
## 7 murin 2670. 3665. 4cf4d0a1… 0000 0001B… 30 30 ORT2
## 8 marilena 1090. 2110. 4cf4d0a1… 0000 0001B… 34 34 ORT2
## 9 leu 2110. 2500. 4cf4d0a1… 0000 0001B… 35 35 ORT2
## 10 njomezen 2500. 3665. 4cf4d0a1… 0000 0001B… 36 36 ORT2
## 11 lena 900. 1370. 4cf4d0a1… 0000 0001C… 54 54 ORT2
## 12 leu 1370. 1620. 4cf4d0a1… 0000 0001C… 55 55 ORT2
## 13 njomezen 1620. 2550. 4cf4d0a1… 0000 0001C… 56 56 ORT2
## 14 jo 2550. 2840. 4cf4d0a1… 0000 0001C… 57 57 ORT2
## 15 lena 2840. 3280. 4cf4d0a1… 0000 0001C… 58 58 ORT2
## 16 leu 3280. 3550. 4cf4d0a1… 0000 0001C… 59 59 ORT2
## 17 murin 3550. 3665. 4cf4d0a1… 0000 0001C… 60 60 ORT2
## # ℹ 7 more variables: attribute <chr>, start_item_seq_idx <int>,
## # end_item_seq_idx <int>, type <chr>, sample_start <int>, sample_end <int>,
## # sample_rate <int>4.1.4 Display the annotations of ORT2 time-aligned to the signals
Entering serve(alb2_DB, useViewer = F) shows that the SEGMENT tier MAU is displayed underneath the signals. As also discussed in the same earlier module, the reason for this is because of the setting in the get/set_levelCanvasesOrder() functions. Thus this confirms that the current display is to MAU:
## [1] "MAU"But the aim is to display only the newly created ORT2 tier. This can be done as follows:
If you view the database now, it will show only the segment tier ORT2 underneath the signals:
4.2 Create a Tone tier and link it to ORT2
4.2.1 Creating the new Tone tier
This can be done using the add_levelDefinition() function used earlier in creating ORT2. In this case, Tone should be an EVENT tier for marking intonational targets as single points in time. The command is:
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%4.2.2 Linking Tone and ORT2
The tiers Tone and ORT2 are currently unlinked as the following verifies:
## type superlevelName sublevelName
## 1 ONE_TO_MANY bundle ORT
## 2 ONE_TO_MANY ORT MAS
## 3 ONE_TO_MANY MAS MAUBut they need to be linked in order that their annotations can be queried relatively to each other. On the assumption that a word can be associated with one or more tones, but that a given tone can only be associated with one word, then the association between the tiers is ONE-TO-MANY. The tiers should therefore be linked with the add_linkDefinitions() function in the following way:
The same command as earlier now verifies that these tiers are linked (see the last line of the following output):
## type superlevelName sublevelName
## 1 ONE_TO_MANY bundle ORT
## 2 ONE_TO_MANY ORT MAS
## 3 ONE_TO_MANY MAS MAU
## 4 ONE_TO_MANY ORT2 Tone4.2.3 Displaying the Tone tier
As before, this is done with the function set_levelCanvasesOrder().
You can verify the relationship that the Tone and ORT2 tiers are being displayed underneath the signals with:
4.3 Calculating and displaying f0
There are three steps here:
- Calculate f0.
- Add the f0 information to the database.
- Configure the database to display the f0 track.
Most of these steps were covered in an earlier module. Therefore, only a brief summary is given as follows.
4.3.1 Calculating f0
Identify the .wav files for which fundamental frequency data is to be calculated. They are here:
alb_wav_paths = list.files(path.albanian,
pattern = ".*wav$",
recursive = T,
full.names = T)
alb_wav_paths## [1] "./testsample/albanian/0001BF_1syll_1.wav"
## [2] "./testsample/albanian/0001BF_2syll_1.wav"
## [3] "./testsample/albanian/0001BF_3syll_1.wav"
## [4] "./testsample/albanian/0001CF_ord1_4.wav"Calculate pitch for each .wav file and store the output:
##
## INFO: applying mhspitch to 4 files
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%Add these f0 files to the Emu database with the add_files() function:
4.3.2 Configure the database to display the f0 track
Use the add_ssffTrackDefinition() function to define the pitch track. See also this earlier module for further details.
As explained in an earlier module, the pitch data can be displayed with the function set_signalCanvasesOrder(). Currently, the waveform and spectrogram are being displayed, as shown by:
## [1] "OSCI" "SPEC"To set things up so that the pitch data is displayed underneath the spectrogram and without displaying the spectrogram:
Overlaying the pitch on the spectrogram is a bit more intricate. For this, you will have to use a text editor (outside of R) to edit alb2_DBconfig.json that is located here.
## [1] "./emu_databases/alb2_emuDB/alb_DBconfig.json"Then proceed as follows:
- Optionally make a backup copy of
alb2_DBconfig.jsonin case anything goes wrong. - Open
alb2_DBconfig.jsonwith a plain text editor. - Search for
"assign" - Carefully replace
"assign": [],
with
"assign": [{ "signalCanvasName": "SPEC", "ssffTrackName": "pitch" }],
- Save
alb2_DBconfig.json - Look at the database again
- Finally, get rid of the pitch track.
And look again at the database. It should now be as in Fig. 2.1.
4.4 Automatically linking annotations at the Tone tier
The first task is to provide a couple of annotations. More specifically add two pitch targets (e.g., L* to Lena, L* to leu) for the utterance 0001BF_1syll_1 as in Fig. 2.1. See the sub-section in this module (scroll down to the grey box) to see details of how to annotate in Emu.
set_levelCanvasesOrder(alb2_DB,
perspectiveName = "default",
order = c("Tone", "ORT2"))
serve(alb2_DB, useViewer = F)The next task is to link automatically the annotations at the tonal level Tone tier to the corresponding words in ORT2 using the function autobuild_linkFromTimes(). This function links an annotation at time t at the Tone tier to an annotation at the ORT2 tier if w_onset < t < w_offset where w_onset and w_offset are the word’s start and end times respectively (i.e. automatically link any tone to a word if the tone’s time falls within the boundary times of the word).
Look at the database again: Check in the hierarchy view that the links between ORT2 and Tone were created in the path ORT2 -> Tone.
It should now be possible to query tones with respect to words and vice-versa. For example:
# Find all pitch-accented words
# (i.e. find any word linked to any tone):
query(alb2_DB, "[Tone =~.* ^ #ORT2 =~.*]")
4.5 Adding a CV tier
It can be helpful to have another tier in which every vowel corresponds to V and every consonant to C. This could then be used to establish the syllable types in the database. To do this, a new tier called CV will be created that is an attribute of the MAU tier (which contains the phonemic annotations).
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%Here are all of the annotations at the MAU tier in the database:
## # A tibble: 13 × 2
## labels n
## <chr> <int>
## 1 4 4
## 2 <p:> 9
## 3 E 15
## 4 J 3
## 5 L 10
## 6 O 3
## 7 a 7
## 8 i 3
## 9 j 1
## 10 m 7
## 11 n 9
## 12 u 8
## 13 z 2of which the following are the vowels:
The following makes use of a few tidyverse commands in order to create a new column labels which is V for vowels, otherwise C. It also changes the attribute column to CV:
new_items = mau.s %>%
mutate(attribute = "CV") %>%
rename(old_labels = labels) %>%
mutate(labels = case_when(old_labels %in% vowel_phonemes ~ "V",
TRUE ~ "C"))
count(new_items, labels) ## # A tibble: 2 × 2
## labels n
## <chr> <int>
## 1 C 41
## 2 V 40The function update_itemsInLevel adds these new annotations to the attribute tier CV:
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%It should now be possible to query the CV tier for all of its annotations:
## # A tibble: 2 × 2
## labels n
## <chr> <int>
## 1 C 41
## 2 V 40Perhaps more importantly, it is now possible to identify the different types of syllable structure. Recall that the MAS tier parses the Phoneme tier into syllables.
## # A tibble: 13 × 2
## labels n
## <chr> <int>
## 1 4 i 1
## 2 4 i n 2
## 3 J E 1
## 4 J O 2
## 5 L E 10
## 6 j O 1
## 7 m E 2
## 8 m a 2
## 9 m u 2
## 10 m u 4 1
## 11 n a 5
## 12 u 5
## 13 z E n 2Here are the syllables in terms of their representation at the CV tier:
## # A tibble: 6 × 2
## labels n
## <chr> <int>
## 1 C->V 25
## 2 C->V->C 2
## 3 C->V->V 1
## 4 V 5
## 5 V->V 1
## 6 V->V->C 2