---
title: "06 Signal Data Calculation and Extraction"
output:
  html_document: default
  pdf_document: default
word_document: default
---


Once again, let us create a temporary EMU-SDMS-database first:

```{r, echo=TRUE,results="hide",message=FALSE}
# load package
library(emuR)
# create demo data in directory
# provided by tempdir()
create_emuRdemoData(dir = tempdir())
# create path to demo database
path2ae = file.path(tempdir(), "emuR_demoData", "ae_emuDB")
# load database
ae = load_emuDB(path2ae, verbose = F)
```
We can use the `summary()` function to learn more about the given data-base. 
```{r, echo=TRUE,message=FALSE}
summary(ae)
```
In the last chapter, we were interested in querying the rather complex hierarchy that can be seen at "Link definitions". Today, we are interested in derived signals, i.e. in signals dervied from the signal, with which every utterance is associated with (this is usually an audio recording, but we could also have EMA-data, eletropalatographic data, and so on, possibly without any audio data at all).
In an EMU-SMDS-data-base, derived data is stored in the so-called SSFF file format. SSFF stands for Simple Signal File Format. 
In the so-called SSFF track definitions, we can see two definitions, one for so-called dft-data, one for fm-data. To see these definitions, we could also have typed:

```{r, echo=TRUE,message=FALSE}
list_ssffTrackDefinitions(ae)
```

Be informed, that these data are signals derived from the audio data, and represent spectral analyses in the `dft`-tracks and calculated formants (and their bandwidths) in the "fm"-tracks. Keep in mind that every signal except audio data is stored in this format, so e.g. the EMA-tracks in the demo-data-bases `ema` or the electropalatographic data in the demo-data-base `epgdorsal` are stored in the SSFF format. See e.g. the aforementioned demo-data-bases in http://ips-lmu.github.io/EMU-webApp/ (open demo -- ema/epgdorsal). 

As the SSFF track definition informs us, these tracks are obviously saved in extra files, defined by their `fileExtension`, e.g. 'dft' or 'fms'. If we open one of these files, we will notice, that we cannot see the data directly (because it is binary data), but we can read the header, e.g. in the case of the fms-files:

```{r, echo=TRUE,results="hide",message=FALSE}
## SSFF -- (c) SHLRC
## Machine IBM-PC
## Record_Freq 200.0
## Start_Time 0.0025
## Column fm SHORT 4
## Column bw SHORT 4
## Original_Freq DOUBLE 20000.0
## -----------------
```

This means, that every fms-file contains four columns with formant values (recorded every 5 ms - as can be derived from the Record_Freq information) alongside with 4 columns which contain the bandwidths of the first four formants. We can read in these data into R with the following commands. First of all, we have to do a query, e.g. for the vowel /i:/, and then use the command `get_trackdata()`:
```{r, echo=TRUE,eval=FALSE, message=TRUE}
# query loaded "ae" emuDB for all "i:" segments of the "Phonetic" level
sl = query(emuDBhandle = ae, 
           query = "Phonetic == i:")

# get the corresponding formant trackdata
ae_i_fm = get_trackdata(emuDBhandle = ae, 
                   seglist = sl, 
                   ssffTrackName = "fm")
# the following will fail, due to the missing SSFF track definition in our database
ae_i_bw = get_trackdata(emuDBhandle = ae, 
                   seglist = sl, 
                   ssffTrackName = "bw")
```
It will, however, not always be the case that we are served with pre-calculated derived data. Oftentimes we start with audio data (and accompanying segmentations) only. Recall e.g. the example from chapter 02:
```{r, echo=TRUE,eval=TRUE, results='hide', message=FALSE}
demoDataDir = file.path(tempdir(), "emuR_demoData")
# create path to TextGrid collection
tgColDir = file.path(demoDataDir, "TextGrid_collection")
# convert TextGrid collection to the emuDB format
convert_TextGridCollection(dir = tgColDir,
                           dbName = "myFirst",
                           targetDir = tempdir(),
                           tierNames = c("Text", "Syllable","Phoneme", "Phonetic"))
# get path to emuDB called "myFirst"
# that was created by convert_TextGridCollection()
path2directory = file.path(tempdir(), "myFirst_emuDB")

# load emuDB into current R session
dbHandle = load_emuDB(path2directory, verbose = FALSE)
```
```{r}
list_ssffTrackDefinitions(dbHandle)
```

We cannot read in pre-calculated derived signals, because there are not any. We therefore have to calculate these.
We can do this on-the-fly, or we can precalculate derived signals. In both cases, we will use the package `wrassp`. After calculating data with `wrassp`, we are able to read it into R with `get_trackdata()`. We will come back to this command later in this document.

# The R package `wrassp`^[This part of the chapter is nearly identical to Winkelmann, Raphael (to appear), "The EMU-SDMS Manual", Dokumentation zur Mediendissertation
zur Erlangung des Doktorgrades
der Philosophie an der Ludwig-Maximilians-Universität
München, chapter 7]
This passage gives an overview and introduction to the `wrassp` package. The `wrassp`
package is a **wrapper** for **R** around Michel Scheffers' *libassp* (**Advanced Speech Signal
Processor**). The libassp library and therefore the wrassp package provide functionality
for handling speech signales in most common audio formats and for
performing signal analyses common in the phonetic and speech sciences. As such, wrassp fills a gap in the R package landscape as, to our knowledge, no previous packages
provided this specialized functionality. The currently available signal processing
functions provided by `wrassp` are:

Command | Meaning 
:------------------ | :--------------------------------------------------------------------------------------
`acfana()`| Analysis of short-term autocorrelation function
`afdiff()`| Computes the first difference of the signal
`affilter()`| Filters the audio signal (e.g., low-pass and high-pass)
`cepstrum()`| Short-term cepstral analysis
`cssSpectrum()`| Cepstral smoothed version of `dftSpectrum()`
`dftSpectrum()`| Short-term DFT spectral analysis
`forest()`| Formant estimation
`ksvF0()`| F0 analysis of the signal
`lpsSpectrum()`| Linear predictive smoothed version of `dftSpectrum()`
`mhsF0()`| Pitch analysis of the speech signal using Michel Scheffers' Modified Harmonic Sieve algorithm
`rfcana()`| Linear prediction analysis
`rmsana()`| Analysis of short-term Root Mean Square amplitude
`zcrana()`| Analysis of the averages of the short-term positive and negative zero-crossing rates


The available file handling functions are:

Command|Meaning
:------------------ | :--------------------------------------------------------------------------------------
`read.AsspDataObj()`| read an SSFF or audio file into an AsspDataObj, which is the in-memory equivalent of the SSFF or audio file.
`write.AsspDataObj()`| write an AsspDataObj to file (usually SSFF or audio file formats).

See R's help() function for a comprehensive list of every function and object
provided by the wrassp package is required.

```{r}
help(package="wrassp")
# create path to bundle in database
path2bndl = file.path(path2ae, "0000_ses", "msajc003_bndl")
# list files in bundle directory
list.files(path2bndl)
```

One of the aims of `wrassp` is to provide mechanisms for handling speech-related files
such as audio files and derived and complementary signal files. To have an in-memory
object that can hold these file types in a uniform way the wrassp package provides
the `AsspDataObj` data type:

```{r}
# load the wrassp package
library(wrassp)
# create path to wav file
path2wav <- file.path(path2bndl, "msajc003.wav")
# read audio file
au <- read.AsspDataObj(path2wav)
# show class
class(au)
## [1] "AsspDataObj"
# show print() output of object
print(au)
```

The resulting `au` object is of the class
AsspDataObj. The output of print provides additional information about the object,
such as its sampling rate, duration, data type and data structure information.
Since the file we loaded is audio only, the object contains exactly one track. Further,
since it is a mono file, this track only has a single field. 

We could plot the audio:
```{r}
plot(seq(0,numRecs.AsspDataObj(au) - 1, 10)
/ rate.AsspDataObj(au),
au$audio[c(TRUE, rep(FALSE, 9))],
type = "l",
xlab = "time (s)",
ylab = "Audio samples (INT16)")
```

The export counterpart to `read.AsspDataObj()` function is `write.AsspDataObj()`.
It is used to store in-memory AsspDataObj objects to disk and is particularly useful
for converting other formats to or storing data in the SSFF file format. To show how this function can be used to write a slightly altered
version of the `au` object to a file, The following initially multiplies all the sample
values of au$audio by a factor of 0.5. The resulting AsspDataObj is then written
to an audio file in a temporary directory provided by R's tempdir() function.
```{r, echo=TRUE,eval=TRUE, results='hide', message=FALSE}
# manipulate the audio samples
au$audio = au$audio * 0.5
# write to file in directory
# provided by tempdir()
write.AsspDataObj(au, file.path(tempdir(), 'newau.wav'))
```

## Deriving signals by signal processing
This section will focus on demonstrating three of wrassp's signal processing functions
that calculate formant values, their corresponding bandwidths, the fundamental frequency
contour and the RMS energy contour.
Use `wrasspOutputInfos` in the following way to get more information aubout wrassp functions, e.g.:
```{r}
# show output info of forest function
wrasspOutputInfos$forest
```
###Formants and their bandwidths calculated by `forest()`
`forest()` is wrassp's formant estimation function. The
default behavior of this formant tracker is to calculate the first four formants and
their bandwidths.

```{r}
fmBwVals=forest(path2wav, toFile=F)
# show class vector
class(fmBwVals)
## [1] "AsspDataObj"
# show track names
tracks.AsspDataObj(fmBwVals)

# check dimensions of tracks are the same
all(dim(fmBwVals$fm) == dim(fmBwVals$bw))
# plot the formant values
matplot(seq(0, numRecs.AsspDataObj(fmBwVals) - 1)
/ rate.AsspDataObj(fmBwVals)
+ attr(fmBwVals, "startTime"),
fmBwVals$fm,
type = "l",
xlab = "time (s)",
ylab = "Formant frequency (Hz)")
# add legend
legend("topright",
legend = c("F1", "F2", "F3", "F4"),
col = 1:4,
lty = 1:4,
bg = "white")
```

###Fundamental frequency contours
```{r, message=FALSE}
# calculate the fundamental frequency contour
ksvF0(path2wav)
# create path to newly generated file
path2f0file = file.path(path2bndl,
paste0("msajc003.", wrasspOutputInfos$ksvF0$ext))
# read file from disk
f0vals = read.AsspDataObj(path2f0file)
# plot the fundamental frequency contour
plot(seq(0,numRecs.AsspDataObj(f0vals) - 1)
/ rate.AsspDataObj(f0vals) +
attr(f0vals, "startTime"),
f0vals$F0,
type = "l",
xlab = "time (s)",
ylab = "F0 frequency (Hz)")
```


###RMS energy contour
The `wrassp` function for calculating the short-term Root Mean Square (RMS) amplitude
of the signal is called `rmsana()`. As its usage is analogous to the above
examples, here we will focus on using it to calculate the RMS values for all the audio
files of the `ae` emuDB. The following example initially uses the `list.files()` function to
aquire the file paths for every .wav file in the `ae` emuDB. As every signal processing
function accepts one or multiple file paths, these file paths can simply be passed in
as the main argument to the `rmsana()` function. As all of wrassp's signal processing
functions place their generated files in the same directory as the audio file they
process, the rmsana() function will automatically place every .rms into the correct
bundle directory.

```{r}
# list all .wav files in the ae emuDB
paths2wavFiles = list.files(path2ae, pattern = "*.wav$",
recursive = TRUE, full.names = TRUE)
# calculate the RMS energy values for all .wav files
rmsana(paths2wavFiles,verbose=FALSE)
# list new .rms files using
# wrasspOutputInfos->rmsana->ext
rmsFPs = list.files(path2ae,
pattern = paste0("*.",
wrasspOutputInfos$rmsana$ext),
recursive = TRUE,
full.names = TRUE)
# read first RMS file
rmsvals = read.AsspDataObj(rmsFPs[1])
# plot the RMS energy contour
plot(seq(0, numRecs.AsspDataObj(rmsvals) - 1)
/ rate.AsspDataObj(rmsvals)
+ attr(rmsvals, "startTime"),
rmsvals$rms,
type = "l",
xlab = "time (s)",
ylab = "RMS energy (dB)")
```

So, ae we can see, it is rather complicated to pre-calculate derived signals with the package `wrassp` only. However, `emuR` delivers much shorter ways. The long path shown here is still to be used, however, if one wishes to introduce speaker-group specific parameter values (e.g. different settings for gender and the like).

#Using wrassp in the EMU-SDMS ^[This sub-chapter owes much to chapters 6 and partially 7 from Winkelmann's Dissertation]
In order to make the above created rms files readable for the EMU-SDMS system, we would have to define the rms track. In order to to so, do:
```{r}
# add SSFF track defintion
# that references the .rms files
# calculated above 
ext = wrasspOutputInfos$rmsana$ext
colName = wrasspOutputInfos$rmsana$tracks[1]
add_ssffTrackDefinition(ae,
name = "rms",
fileExtension = ext,
columnName = colName)
```
From now on, rms is defined and therefore available for emuR - we can now use `get_trackdata()` to read in the rms-data. 

`add_ssffTrackDefinition()` can, however, be used much more directly to precalculate derived signals, by means of the parameter `onTheFlyFunctionName`. The same parameter is available in the command `get_trackdata()` to calculate derived signals for a given seglist on-the-fly. 

##Extracting pre-defined tracks
To access data that are stored in files, the user has to define tracks for a database
that point to sequences of samples in les that match a user-specified file extension.
The user-defined name of such a track can then be used to reference the track in the
signal data extraction process. Internally, emuR uses `wrassp` to read the appropriate
files from disk, extract the sample sequences that match the result of a query and
return values to the user for further inspection and evaluation.

```{r}
# list currently available tracks
list_ssffTrackDefinitions(ae)
```

In `ae`, there are three tracks available, that can be read by `get_trackdata()`. We could e.g.

```{r}
# query all "ai" phonetic segments
ai_segs = query(ae, "Phonetic == ai")
# get "fm" track data for these segments
# Note that verbose is set to FALSE
# only to avoid a progress bar
# being printed in this document.
ai_td_fm = get_trackdata(emuDBhandle = ae,
                         seglist = ai_segs,
                         ssffTrackName = "fm",
                         verbose = FALSE)
# show summary of ai_td_fm
summary(ai_td_fm)
```

 So, we needed an emuDBhandle, a seglist, and the correct ssffTrackName to read the formant values from the fms-files. Being able to access data that is stored in files is important for two main reasons.

Firstly, it is possible to generate files using external programs such as VoiceSauce
(Shue et al., 2011), which can export its calculated output to the general purpose
SSFF file format. This file mechanism is also used to access data produced by EMA,
EPG or any other form of signal data recordings. Secondly, it is possible to track,
save and access manipulated data such as formant values that have been *manually
corrected*. It is also worth noting that the `get trackdata()` function has a predefined
track which is always available without it having to be defined. The name of this
track is MEDIAFILE SAMPLES which references the actual samples of the audio files
of the database. The next example shows how this predefined track can be used to
access the audio samples belonging to the segments in ai segs.

```{r}
# get media file samples
ai_td_mfs = get_trackdata(ae,
        seglist = ai_segs,
        ssffTrackName = "MEDIAFILE_SAMPLES",
        verbose = FALSE)
# show summary of ai_td_fm
summary(ai_td_mfs)
```

##Adding new tracks
The signal processing routines provided by the
`wrassp` package can be used to produce SSFF files containing various derived signal
data (e.g., formants, fundamental frequency, etc.). The following example shows how the
`add_ssffTrackDefinition()` can be used to add a new track to the `ae` emuDB. Using
the `onTheFlyFunctionName` parameter, the `add_ssffTrackDefinition()` function
automatically executes the `wrassp` signal processing function `ksvF0` (`onTheFlyFunctionName
= "ksvF0"`) and stores the results in SSFF files in the bundle directories.

```{r, eval=FALSE, echo=TRUE,results="hide",message=FALSE}
# add new track and calculate
# .f0 files on-the-fly using wrassp::ksvF0()
add_ssffTrackDefinition(ae,
                        name = "F0",
                        onTheFlyFunctionName = "ksvF0",
                        verbose = FALSE)
# show newly added track
list_ssffTrackDefinitions(ae)
# show newly added files
list_files(ae, fileExtension = "f0")
# extract newly added trackdata
ai_td = get_trackdata(ae,
                      seglist = ai_segs,
                      ssffTrackName = "F0",
                      verbose = FALSE)
```

In the command `add_ssffTrackDefinition()`, we could have also added parameter values to the `wrassp` function `ksvF0`.

```{r}
formals(wrassp::ksvF0)
```

We can see, that the parameter gender is by default on "u" (undefined). As we now, that the `ae` data-base consists of male speech only, we could set this to "m" (male):

```{r, message=FALSE}
add_ssffTrackDefinition(ae,
                        name = "F0",
                        onTheFlyFunctionName = "ksvF0",
                        onTheFlyParams = list(gender = "m"),
                        verbose = FALSE)
```

Pre-calculated dervied signals can be shown and corrected in the EMU-webApp. Pre-calculated formants may be overlaid to the spectrogram. Learn more about this in chapter 08.

One disadvantage of this method may not be withholded: it is -- until now -- not püossible to pre-calculate speaker-group-(e.g. gender)-specific data by setting different parameters for various speaker-groups. However, implementation of this feature is in the making.

##Calculating tracks on-the-fly

With the `wrassp` package, we were able to implement a new form of signal data
extraction which was not available in the legacy system. The user is now able to select
one of the signal processing routines provided by `wrassp` and pass it on to the signal
data extraction function. The signal data extraction function can then apply this
`wrassp` function to each audio file as part of the signal data extraction process. This
means that the user can quickly manipulate function parameters and evaluate the
result without having to store to disk the files that would usually be generated by the
various parameter experiments. In many cases this new functionality eliminates the
need for defining a track definition for the entire database for temporary data analysis
purposes. The following example shows how the onTheFlyFunctionName parameter of the
get trackdata() function is used:

```{r}
ai_td_pit = get_trackdata(ae,
seglist = ai_segs,
onTheFlyFunctionName = "mhsF0",
verbose = FALSE)
# show summary of ai_td
summary(ai_td_pit)
```


#The resulting object: `trackdata` vs. `emuRtrackdata`
The default resulting object of a call to `get_trackdata()` is of class `trackdata`
The `emuR` package provides multiple routines such as `dcut()`,
`trapply()`, `eplot` and `dplot()` for processing and visually inspect objects of this type (see Harrington, 2010, for the use of these functions). 
```{r}
iVu = query(ae,query="Phonetic==V|i:|u:")
iVu_fm = get_trackdata(ae,
                        seglist = iVu,
                        ssffTrackName =  "fm",
                        verbose = FALSE)
# show class of iVu_fm
class(iVu_fm)

iVu_fm05=dcut(iVu_fm,.5,prop = TRUE)
eplot(iVu_fm05[,1:2],label(iVu),centroid=TRUE,formant = TRUE)
dplot(iVu_fm[,1:2],label(iVu))
dplot(iVu_fm[,1:2],label(iVu),normalise=TRUE,average=TRUE)
```

A trackdata object consists of three parts (`$index`, `$ftime`, and `$data`). 

```{r}
#show the first two segments' values only
iVu_fm[1:2,]
```
As the fms-files have fout columns of formant data (F1 ... F4), `$data` has four columns as well. In f0-data, there will be only one column, called `T1`. In spectral data, however, there may by hundrets of columns:

```{r}
iVu_spectral = get_trackdata(ae,
                        seglist = iVu,
                        onTheFlyFunctionName =  "dftSpectrum",
                        verbose = FALSE)
colnames(iVu_spectral$data)
```

However, as the trackdata object is a fairly complex nested matrix object with internal
reference matrices, which can be cumbersome to work with, the `emuR` package introduces
a new equivalent object type called emuRtrackdata that essentially is a
flat data.frame or data.table object. This object type can be retrieved by setting
the resultType parameter of the get trackdata() function to `emuRtrackdata`:

```{r}
iVu_fm_new = get_trackdata(ae,
                        seglist = iVu,
                        ssffTrackName =  "fm",
                        resultType="emuRtrackdata",
                        verbose = FALSE)
# show class of iVu_fm
class(iVu_fm_new)

iVu_fm_new

names(iVu_fm_new)
```

The emuRtrackdata object is an amalgamation of both a segment list and a trackdata object. The
first `sl_rowIdx` column of the `iVu` object indicates the row index of the
segment list the current row belongs to, the `times_rel` and `times_orig` (and `times_norm` in the forthcoming emuR-version) columns
represent the relative time and the original time of the samples contained in the
current row and `T1` (to `T`n in n dimensional trackdata) contains the actual signal
sample values. It is also worth noting that the `emuR` package provides a function
called `create emuRtrackdata()`, which allows users to create emuRtrackdata from
a segment list and a trackdata object. This is beneficial as it allows trackdata
objects to be processed using functions provided by the emuR package (e.g., dcut()
and trapply()) and then converts them into a standardized data.table object for
further processing (e.g., using R packages such as `lme4` or `ggplot2` which were implemented
to use with data.frame or data.table objects).

#Alternative signal derivations
Some people believe, that e.g. `wrassp`'s formant tracker is not as good as the one used in `praat`. In order to use the one provided by `praat`, you could use Raphael Winkelmann's script here: https://gist.github.com/raphywink/2512752a1efa56951f04
This script allows for calculation of formant frequencies in praat and converts the result into the SSFF format.