#' ---
#' title: "06 Signal Data Calculation and Extraction"
#' output:
#'   html_document: default
#'   pdf_document: default
#' word_document: default
#' ---
#' 
#' 
#' Once again, let us create a temporary EMU-SDMS-database first:
#' 
## ---- echo=TRUE,results="hide",message=FALSE-----------------------------
# load package
library(emuR)
# create demo data in directory
# provided by tempdir()
create_emuRdemoData(dir = tempdir())
# create path to demo database
path2ae = file.path(tempdir(), "emuR_demoData", "ae_emuDB")
# load database
ae = load_emuDB(path2ae, verbose = F)

#' We can use the `summary()` function to learn more about the given data-base. 
## ---- echo=TRUE,message=FALSE--------------------------------------------
summary(ae)

#' In the last chapter, we were interested in querying the rather complex hierarchy that can be seen at "Link definitions". Today, we are interested in derived signals, i.e. in signals dervied from the signal, with which every utterance is associated with (this is usually an audio recording, but we could also have EMA-data, eletropalatographic data, and so on, possibly without any audio data at all).
#' In an EMU-SMDS-data-base, derived data is stored in the so-called SSFF file format. SSFF stands for Simple Signal File Format. 
#' In the so-called SSFF track definitions, we can see two definitions, one for so-called dft-data, one for fm-data. To see these definitions, we could also have typed:
#' 
## ---- echo=TRUE,message=FALSE--------------------------------------------
list_ssffTrackDefinitions(ae)

#' 
#' Be informed, that these data are signals derived from the audio data, and represent spectral analyses in the `dft`-tracks and calculated formants (and their bandwidths) in the "fm"-tracks. Keep in mind that every signal except audio data is stored in this format, so e.g. the EMA-tracks in the demo-data-bases `ema` or the electropalatographic data in the demo-data-base `epgdorsal` are stored in the SSFF format. See e.g. the aforementioned demo-data-bases in http://ips-lmu.github.io/EMU-webApp/ (open demo -- ema/epgdorsal). 
#' 
#' As the SSFF track definition informs us, these tracks are obviously saved in extra files, defined by their `fileExtension`, e.g. 'dft' or 'fms'. If we open one of these files, we will notice, that we cannot see the data directly (because it is binary data), but we can read the header, e.g. in the case of the fms-files:
#' 
## ---- echo=TRUE,results="hide",message=FALSE-----------------------------
## SSFF -- (c) SHLRC
## Machine IBM-PC
## Record_Freq 200.0
## Start_Time 0.0025
## Column fm SHORT 4
## Column bw SHORT 4
## Original_Freq DOUBLE 20000.0
## -----------------

#' 
#' This means, that every fms-file contains four columns with formant values (recorded every 5 ms - as can be derived from the Record_Freq information) alongside with 4 columns which contain the bandwidths of the first four formants. We can read in these data into R with the following commands. First of all, we have to do a query, e.g. for the vowel /i:/, and then use the command `get_trackdata()`:
## ---- echo=TRUE,eval=FALSE, message=TRUE---------------------------------
## # query loaded "ae" emuDB for all "i:" segments of the "Phonetic" level
## sl = query(emuDBhandle = ae,
##            query = "Phonetic == i:")
## 
## # get the corresponding formant trackdata
## ae_i_fm = get_trackdata(emuDBhandle = ae,
##                    seglist = sl,
##                    ssffTrackName = "fm")
## # the following will fail, due to the missing SSFF track definition in our database
## ae_i_bw = get_trackdata(emuDBhandle = ae,
##                    seglist = sl,
##                    ssffTrackName = "bw")

#' It will, however, not always be the case that we are served with pre-calculated derived data. Oftentimes we start with audio data (and accompanying segmentations) only. Recall e.g. the example from chapter 02:
## ---- echo=TRUE,eval=TRUE, results='hide', message=FALSE-----------------
demoDataDir = file.path(tempdir(), "emuR_demoData")
# create path to TextGrid collection
tgColDir = file.path(demoDataDir, "TextGrid_collection")
# convert TextGrid collection to the emuDB format
convert_TextGridCollection(dir = tgColDir,
                           dbName = "myFirst",
                           targetDir = tempdir(),
                           tierNames = c("Text", "Syllable","Phoneme", "Phonetic"))
# get path to emuDB called "myFirst"
# that was created by convert_TextGridCollection()
path2directory = file.path(tempdir(), "myFirst_emuDB")

# load emuDB into current R session
dbHandle = load_emuDB(path2directory, verbose = FALSE)

## ------------------------------------------------------------------------
list_ssffTrackDefinitions(dbHandle)

#' 
#' We cannot read in pre-calculated derived signals, because there are not any. We therefore have to calculate these.
#' We can do this on-the-fly, or we can precalculate derived signals. In both cases, we will use the package `wrassp`. After calculating data with `wrassp`, we are able to read it into R with `get_trackdata()`. We will come back to this command later in this document.
#' 
#' # The R package `wrassp`^[This part of the chapter is nearly identical to Winkelmann, Raphael (to appear), "The EMU-SDMS Manual", Dokumentation zur Mediendissertation
#' zur Erlangung des Doktorgrades
#' der Philosophie an der Ludwig-Maximilians-Universität
#' München, chapter 7]
#' This passage gives an overview and introduction to the `wrassp` package. The `wrassp`
#' package is a **wrapper** for **R** around Michel Scheffers' *libassp* (**Advanced Speech Signal
#' Processor**). The libassp library and therefore the wrassp package provide functionality
#' for handling speech signales in most common audio formats and for
#' performing signal analyses common in the phonetic and speech sciences. As such, wrassp fills a gap in the R package landscape as, to our knowledge, no previous packages
#' provided this specialized functionality. The currently available signal processing
#' functions provided by `wrassp` are:
#' 
#' Command | Meaning 
#' :------------------ | :--------------------------------------------------------------------------------------
#' `acfana()`| Analysis of short-term autocorrelation function
#' `afdiff()`| Computes the first difference of the signal
#' `affilter()`| Filters the audio signal (e.g., low-pass and high-pass)
#' `cepstrum()`| Short-term cepstral analysis
#' `cssSpectrum()`| Cepstral smoothed version of `dftSpectrum()`
#' `dftSpectrum()`| Short-term DFT spectral analysis
#' `forest()`| Formant estimation
#' `ksvF0()`| F0 analysis of the signal
#' `lpsSpectrum()`| Linear predictive smoothed version of `dftSpectrum()`
#' `mhsF0()`| Pitch analysis of the speech signal using Michel Scheffers' Modified Harmonic Sieve algorithm
#' `rfcana()`| Linear prediction analysis
#' `rmsana()`| Analysis of short-term Root Mean Square amplitude
#' `zcrana()`| Analysis of the averages of the short-term positive and negative zero-crossing rates
#' 
#' 
#' The available file handling functions are:
#' 
#' Command|Meaning
#' :------------------ | :--------------------------------------------------------------------------------------
#' `read.AsspDataObj()`| read an SSFF or audio file into an AsspDataObj, which is the in-memory equivalent of the SSFF or audio file.
#' `write.AsspDataObj()`| write an AsspDataObj to file (usually SSFF or audio file formats).
#' 
#' See R's help() function for a comprehensive list of every function and object
#' provided by the wrassp package is required.
#' 
## ------------------------------------------------------------------------
help(package="wrassp")
# create path to bundle in database
path2bndl = file.path(path2ae, "0000_ses", "msajc003_bndl")
# list files in bundle directory
list.files(path2bndl)

#' 
#' One of the aims of `wrassp` is to provide mechanisms for handling speech-related files
#' such as audio files and derived and complementary signal files. To have an in-memory
#' object that can hold these file types in a uniform way the wrassp package provides
#' the `AsspDataObj` data type:
#' 
## ------------------------------------------------------------------------
# load the wrassp package
library(wrassp)
# create path to wav file
path2wav <- file.path(path2bndl, "msajc003.wav")
# read audio file
au <- read.AsspDataObj(path2wav)
# show class
class(au)
## [1] "AsspDataObj"
# show print() output of object
print(au)

#' 
#' The resulting `au` object is of the class
#' AsspDataObj. The output of print provides additional information about the object,
#' such as its sampling rate, duration, data type and data structure information.
#' Since the file we loaded is audio only, the object contains exactly one track. Further,
#' since it is a mono file, this track only has a single field. 
#' 
#' We could plot the audio:
## ------------------------------------------------------------------------
plot(seq(0,numRecs.AsspDataObj(au) - 1, 10)
/ rate.AsspDataObj(au),
au$audio[c(TRUE, rep(FALSE, 9))],
type = "l",
xlab = "time (s)",
ylab = "Audio samples (INT16)")

#' 
#' The export counterpart to `read.AsspDataObj()` function is `write.AsspDataObj()`.
#' It is used to store in-memory AsspDataObj objects to disk and is particularly useful
#' for converting other formats to or storing data in the SSFF file format. To show how this function can be used to write a slightly altered
#' version of the `au` object to a file, The following initially multiplies all the sample
#' values of au$audio by a factor of 0.5. The resulting AsspDataObj is then written
#' to an audio file in a temporary directory provided by R's tempdir() function.
## ---- echo=TRUE,eval=TRUE, results='hide', message=FALSE-----------------
# manipulate the audio samples
au$audio = au$audio * 0.5
# write to file in directory
# provided by tempdir()
write.AsspDataObj(au, file.path(tempdir(), 'newau.wav'))

#' 
#' ## Deriving signals by signal processing
#' This section will focus on demonstrating three of wrassp's signal processing functions
#' that calculate formant values, their corresponding bandwidths, the fundamental frequency
#' contour and the RMS energy contour.
#' Use `wrasspOutputInfos` in the following way to get more information aubout wrassp functions, e.g.:
## ------------------------------------------------------------------------
# show output info of forest function
wrasspOutputInfos$forest

#' ###Formants and their bandwidths calculated by `forest()`
#' `forest()` is wrassp's formant estimation function. The
#' default behavior of this formant tracker is to calculate the first four formants and
#' their bandwidths.
#' 
## ------------------------------------------------------------------------
fmBwVals=forest(path2wav, toFile=F)
# show class vector
class(fmBwVals)
## [1] "AsspDataObj"
# show track names
tracks.AsspDataObj(fmBwVals)

# check dimensions of tracks are the same
all(dim(fmBwVals$fm) == dim(fmBwVals$bw))
# plot the formant values
matplot(seq(0, numRecs.AsspDataObj(fmBwVals) - 1)
/ rate.AsspDataObj(fmBwVals)
+ attr(fmBwVals, "startTime"),
fmBwVals$fm,
type = "l",
xlab = "time (s)",
ylab = "Formant frequency (Hz)")
# add legend
legend("topright",
legend = c("F1", "F2", "F3", "F4"),
col = 1:4,
lty = 1:4,
bg = "white")

#' 
#' ###Fundamental frequency contours
## ---- message=FALSE------------------------------------------------------
# calculate the fundamental frequency contour
ksvF0(path2wav)
# create path to newly generated file
path2f0file = file.path(path2bndl,
paste0("msajc003.", wrasspOutputInfos$ksvF0$ext))
# read file from disk
f0vals = read.AsspDataObj(path2f0file)
# plot the fundamental frequency contour
plot(seq(0,numRecs.AsspDataObj(f0vals) - 1)
/ rate.AsspDataObj(f0vals) +
attr(f0vals, "startTime"),
f0vals$F0,
type = "l",
xlab = "time (s)",
ylab = "F0 frequency (Hz)")

#' 
#' 
#' ###RMS energy contour
#' The `wrassp` function for calculating the short-term Root Mean Square (RMS) amplitude
#' of the signal is called `rmsana()`. As its usage is analogous to the above
#' examples, here we will focus on using it to calculate the RMS values for all the audio
#' files of the `ae` emuDB. The following example initially uses the `list.files()` function to
#' aquire the file paths for every .wav file in the `ae` emuDB. As every signal processing
#' function accepts one or multiple file paths, these file paths can simply be passed in
#' as the main argument to the `rmsana()` function. As all of wrassp's signal processing
#' functions place their generated files in the same directory as the audio file they
#' process, the rmsana() function will automatically place every .rms into the correct
#' bundle directory.
#' 
## ------------------------------------------------------------------------
# list all .wav files in the ae emuDB
paths2wavFiles = list.files(path2ae, pattern = "*.wav$",
recursive = TRUE, full.names = TRUE)
# calculate the RMS energy values for all .wav files
rmsana(paths2wavFiles,verbose=FALSE)
# list new .rms files using
# wrasspOutputInfos->rmsana->ext
rmsFPs = list.files(path2ae,
pattern = paste0("*.",
wrasspOutputInfos$rmsana$ext),
recursive = TRUE,
full.names = TRUE)
# read first RMS file
rmsvals = read.AsspDataObj(rmsFPs[1])
# plot the RMS energy contour
plot(seq(0, numRecs.AsspDataObj(rmsvals) - 1)
/ rate.AsspDataObj(rmsvals)
+ attr(rmsvals, "startTime"),
rmsvals$rms,
type = "l",
xlab = "time (s)",
ylab = "RMS energy (dB)")

#' 
#' So, ae we can see, it is rather complicated to pre-calculate derived signals with the package `wrassp` only. However, `emuR` delivers much shorter ways. The long path shown here is still to be used, however, if one wishes to introduce speaker-group specific parameter values (e.g. different settings for gender and the like).
#' 
#' #Using wrassp in the EMU-SDMS ^[This sub-chapter owes much to chapters 6 and partially 7 from Winkelmann's Dissertation]
#' In order to make the above created rms files readable for the EMU-SDMS system, we would have to define the rms track. In order to to so, do:
## ------------------------------------------------------------------------
# add SSFF track defintion
# that references the .rms files
# calculated above 
ext = wrasspOutputInfos$rmsana$ext
colName = wrasspOutputInfos$rmsana$tracks[1]
add_ssffTrackDefinition(ae,
name = "rms",
fileExtension = ext,
columnName = colName)

#' From now on, rms is defined and therefore available for emuR - we can now use `get_trackdata()` to read in the rms-data. 
#' 
#' `add_ssffTrackDefinition()` can, however, be used much more directly to precalculate derived signals, by means of the parameter `onTheFlyFunctionName`. The same parameter is available in the command `get_trackdata()` to calculate derived signals for a given seglist on-the-fly. 
#' 
#' ##Extracting pre-defined tracks
#' To access data that are stored in files, the user has to define tracks for a database
#' that point to sequences of samples in les that match a user-specified file extension.
#' The user-defined name of such a track can then be used to reference the track in the
#' signal data extraction process. Internally, emuR uses `wrassp` to read the appropriate
#' files from disk, extract the sample sequences that match the result of a query and
#' return values to the user for further inspection and evaluation.
#' 
## ------------------------------------------------------------------------
# list currently available tracks
list_ssffTrackDefinitions(ae)

#' 
#' In `ae`, there are three tracks available, that can be read by `get_trackdata()`. We could e.g.
#' 
## ------------------------------------------------------------------------
# query all "ai" phonetic segments
ai_segs = query(ae, "Phonetic == ai")
# get "fm" track data for these segments
# Note that verbose is set to FALSE
# only to avoid a progress bar
# being printed in this document.
ai_td_fm = get_trackdata(emuDBhandle = ae,
                         seglist = ai_segs,
                         ssffTrackName = "fm",
                         verbose = FALSE)
# show summary of ai_td_fm
summary(ai_td_fm)

#' 
#'  So, we needed an emuDBhandle, a seglist, and the correct ssffTrackName to read the formant values from the fms-files. Being able to access data that is stored in files is important for two main reasons.
#' 
#' Firstly, it is possible to generate files using external programs such as VoiceSauce
#' (Shue et al., 2011), which can export its calculated output to the general purpose
#' SSFF file format. This file mechanism is also used to access data produced by EMA,
#' EPG or any other form of signal data recordings. Secondly, it is possible to track,
#' save and access manipulated data such as formant values that have been *manually
#' corrected*. It is also worth noting that the `get trackdata()` function has a predefined
#' track which is always available without it having to be defined. The name of this
#' track is MEDIAFILE SAMPLES which references the actual samples of the audio files
#' of the database. The next example shows how this predefined track can be used to
#' access the audio samples belonging to the segments in ai segs.
#' 
## ------------------------------------------------------------------------
# get media file samples
ai_td_mfs = get_trackdata(ae,
        seglist = ai_segs,
        ssffTrackName = "MEDIAFILE_SAMPLES",
        verbose = FALSE)
# show summary of ai_td_fm
summary(ai_td_mfs)

#' 
#' ##Adding new tracks
#' The signal processing routines provided by the
#' `wrassp` package can be used to produce SSFF files containing various derived signal
#' data (e.g., formants, fundamental frequency, etc.). The following example shows how the
#' `add_ssffTrackDefinition()` can be used to add a new track to the `ae` emuDB. Using
#' the `onTheFlyFunctionName` parameter, the `add_ssffTrackDefinition()` function
#' automatically executes the `wrassp` signal processing function `ksvF0` (`onTheFlyFunctionName
#' = "ksvF0"`) and stores the results in SSFF files in the bundle directories.
#' 
## ---- eval=FALSE, echo=TRUE,results="hide",message=FALSE-----------------
## # add new track and calculate
## # .f0 files on-the-fly using wrassp::ksvF0()
## add_ssffTrackDefinition(ae,
##                         name = "F0",
##                         onTheFlyFunctionName = "ksvF0",
##                         verbose = FALSE)
## # show newly added track
## list_ssffTrackDefinitions(ae)
## # show newly added files
## list_files(ae, fileExtension = "f0")
## # extract newly added trackdata
## ai_td = get_trackdata(ae,
##                       seglist = ai_segs,
##                       ssffTrackName = "F0",
##                       verbose = FALSE)

#' 
#' In the command `add_ssffTrackDefinition()`, we could have also added parameter values to the `wrassp` function `ksvF0`.
#' 
## ------------------------------------------------------------------------
formals(wrassp::ksvF0)

#' 
#' We can see, that the parameter gender is by default on "u" (undefined). As we now, that the `ae` data-base consists of male speech only, we could set this to "m" (male):
#' 
## ---- message=FALSE------------------------------------------------------
add_ssffTrackDefinition(ae,
                        name = "F0",
                        onTheFlyFunctionName = "ksvF0",
                        onTheFlyParams = list(gender = "m"),
                        verbose = FALSE)

#' 
#' Pre-calculated dervied signals can be shown and corrected in the EMU-webApp. Pre-calculated formants may be overlaid to the spectrogram. Learn more about this in chapter 08.
#' 
#' One disadvantage of this method may not be withholded: it is -- until now -- not püossible to pre-calculate speaker-group-(e.g. gender)-specific data by setting different parameters for various speaker-groups. However, implementation of this feature is in the making.
#' 
#' ##Calculating tracks on-the-fly
#' 
#' With the `wrassp` package, we were able to implement a new form of signal data
#' extraction which was not available in the legacy system. The user is now able to select
#' one of the signal processing routines provided by `wrassp` and pass it on to the signal
#' data extraction function. The signal data extraction function can then apply this
#' `wrassp` function to each audio file as part of the signal data extraction process. This
#' means that the user can quickly manipulate function parameters and evaluate the
#' result without having to store to disk the files that would usually be generated by the
#' various parameter experiments. In many cases this new functionality eliminates the
#' need for defining a track definition for the entire database for temporary data analysis
#' purposes. The following example shows how the onTheFlyFunctionName parameter of the
#' get trackdata() function is used:
#' 
## ------------------------------------------------------------------------
ai_td_pit = get_trackdata(ae,
seglist = ai_segs,
onTheFlyFunctionName = "mhsF0",
verbose = FALSE)
# show summary of ai_td
summary(ai_td_pit)

#' 
#' 
#' #The resulting object: `trackdata` vs. `emuRtrackdata`
#' The default resulting object of a call to `get_trackdata()` is of class `trackdata`
#' The `emuR` package provides multiple routines such as `dcut()`,
#' `trapply()`, `eplot` and `dplot()` for processing and visually inspect objects of this type (see Harrington, 2010, for the use of these functions). 
## ------------------------------------------------------------------------
iVu = query(ae,query="Phonetic==V|i:|u:")
iVu_fm = get_trackdata(ae,
                        seglist = iVu,
                        ssffTrackName =  "fm",
                        verbose = FALSE)
# show class of iVu_fm
class(iVu_fm)

iVu_fm05=dcut(iVu_fm,.5,prop = TRUE)
eplot(iVu_fm05[,1:2],label(iVu),centroid=TRUE,formant = TRUE)
dplot(iVu_fm[,1:2],label(iVu))
dplot(iVu_fm[,1:2],label(iVu),normalise=TRUE,average=TRUE)

#' 
#' A trackdata object consists of three parts (`$index`, `$ftime`, and `$data`). 
#' 
## ------------------------------------------------------------------------
#show the first two segments' values only
iVu_fm[1:2,]

#' As the fms-files have fout columns of formant data (F1 ... F4), `$data` has four columns as well. In f0-data, there will be only one column, called `T1`. In spectral data, however, there may by hundrets of columns:
#' 
## ------------------------------------------------------------------------
iVu_spectral = get_trackdata(ae,
                        seglist = iVu,
                        onTheFlyFunctionName =  "dftSpectrum",
                        verbose = FALSE)
colnames(iVu_spectral$data)

#' 
#' However, as the trackdata object is a fairly complex nested matrix object with internal
#' reference matrices, which can be cumbersome to work with, the `emuR` package introduces
#' a new equivalent object type called emuRtrackdata that essentially is a
#' flat data.frame or data.table object. This object type can be retrieved by setting
#' the resultType parameter of the get trackdata() function to `emuRtrackdata`:
#' 
## ------------------------------------------------------------------------
iVu_fm_new = get_trackdata(ae,
                        seglist = iVu,
                        ssffTrackName =  "fm",
                        resultType="emuRtrackdata",
                        verbose = FALSE)
# show class of iVu_fm
class(iVu_fm_new)

iVu_fm_new

names(iVu_fm_new)

#' 
#' The emuRtrackdata object is an amalgamation of both a segment list and a trackdata object. The
#' first `sl_rowIdx` column of the `iVu` object indicates the row index of the
#' segment list the current row belongs to, the `times_rel` and `times_orig` (and `times_norm` in the forthcoming emuR-version) columns
#' represent the relative time and the original time of the samples contained in the
#' current row and `T1` (to `T`n in n dimensional trackdata) contains the actual signal
#' sample values. It is also worth noting that the `emuR` package provides a function
#' called `create emuRtrackdata()`, which allows users to create emuRtrackdata from
#' a segment list and a trackdata object. This is beneficial as it allows trackdata
#' objects to be processed using functions provided by the emuR package (e.g., dcut()
#' and trapply()) and then converts them into a standardized data.table object for
#' further processing (e.g., using R packages such as `lme4` or `ggplot2` which were implemented
#' to use with data.frame or data.table objects).
#' 
#' #Alternative signal derivations
#' Some people believe, that e.g. `wrassp`'s formant tracker is not as good as the one used in `praat`. In order to use the one provided by `praat`, you could use Raphael Winkelmann's script here: https://gist.github.com/raphywink/2512752a1efa56951f04
#' This script allows for calculation of formant frequencies in praat and converts the result into the SSFF format.
