---
title: "The EMU-webApp"
author: "Raphael Winkelmann"
date: "`r format(Sys.time(), '23 October %Y')`"
output:
  html_document: default
  pdf_document: default
  word_document: default
---


# The EMU-webApp 

![](pics/EMU-webAppEmu_emu-webApp.png)


The EMU-SDMS has a unique approach to its GUI in that it utilizes a web application as its primary GUI. This is known as the EMU-webApp \citep{winkelmann:2015d}. The EMU-webApp is a fully fledged browser-based labeling and correction tool that offers a multitude of labeling and visualization features. These features include unlimited undo/redo, formant correction capabilities, the ability to snap a preselected boundary to the nearest top/bottom boundary, snap a preselected boundary to the nearest zero crossing, and many more. The web application is able to render everything directly in the user's browser, including the calculation and rendering of the spectrogram, as it is written entirely using HTML, CSS and JavaScript. This means it can also be used as a standalone labeling application, as it does not require any server-side calculations or rendering. Further, it is designed to interact with any websocket server that implements the EMU-webApp websocket protocol (see Section \ref{subsec:emu-webAppTheProtocol}). This enables it to be used as a labeling tool for collaborative annotation efforts. Also, as the EMU-webApp is cached in the user's browser on the first visit, it does not require any internet connectivity to be able to access the web application unless the user explicitly clears the browser's cache. The URL of the current live version of the EMU-webApp is: [http://ips-lmu.github.io/EMU-webApp/](http://ips-lmu.github.io/EMU-webApp/).

## Main layout

The main screen of the EMU-webApp can be split into five areas. Figure \ref{fig:emu_web_emuWebAppLayout} shows a screenshot of the EMU-webApp's main screen displaying these five areas while displaying a bundle of the *ae* demo database. This database is served to the \texttt{EMU-webApp} by invoking the `serve()` command as shown in R Example \ref{rexample:webApp_serve}. The left side bar (area marked 1 in Figure \ref{fig:emu_web_emuWebAppLayout}) represents the bundle list side bar which, if connected to a database, displays the currently available bundles grouped by their sessions. The top and bottom menu bars (areas marked 2 and 5 in Figure \ref{fig:emu_web_emuWebAppLayout}) display the currently available menu options, where the bottom menu bar contains the audio navigation and playback controls and also includes a scrollable mini map of the oscillogram. Area 3 of Figure \ref{fig:emu_web_emuWebAppLayout} displays the signal canvas area currently displaying the oscillogram and the spectrogram. Other signal contours such as formant frequency contours and fundamental frequency contours are also displayed in this area. Area 4 of Figure \ref{fig:emu_web_emuWebAppLayout} displays the area in which levels containing time information are displayed. It is worth noting that the main screen of the EMU-webApp does not display any levels that do not contain time information. The hierarchical annotation can be displayed and edited by clicking the `show hierarchy` button in the top menu bar (see Figure \ref{fig:webApp_hierModal} for an example of how the hierarchy is displayed).

```{r, rexample:webApp_serve, rexample=TRUE, eval=FALSE}
# serve ae emuDB to EMU-webApp
serve(ae)
```


![Screenshot of EMU-webApp displaying the *ae* demo database with overlaid areas of the main screen of the web application (see text).](pics/emu-webAppLayout.png)


## General usage

This section introduces the labeling mechanics and general labeling workflow of the EMU-webApp. The EMU-webApp makes heavy use of keyboard shortcuts. Is is worth noting that most of the keyboard shortcuts are centered around the `WASD` keys, which are the navigation shortcut keys (`W` to zoom in; `S` to zoom out; `A` to move left and `D` to move right). For a full list of the available keyboard shortcuts see the EMU-webApp's own manual, which can be accessed by clicking the EMU icon on the right hand side of the top menu bar (area 2 in Figure \ref{fig:emu_web_emuWebAppLayout}).

### Annotating levels containing time information

#### Boundaries and events

The EMU-webApp has slightly different labeling mechanics compared with other annotation software. Compared to the usual click and drag of segment boundaries and event markers, the web application continuously tracks the movement of the mouse in levels containing time information, highlighting the boundary or event marker that is closest to it by coloring it blue. Figure \ref{fig:webApp_preSelBoundary} displays this automatic boundary preselection.


![Screenshot of segment level as displayed by the EMU-webApp with superimposed mouse cursor displaying the automatic boundary preselection of closest boundary (boundary marked blue).](pics/preSelBoundary.png)

Once a boundary or event is preselected, the user can perform various actions with it. She or he can, for example, grab a preselected boundary or event by holding down the `SHIFT` key and moving it to the desired position, or delete the current boundary or event by hitting the `BACKSPACE` key. Other actions that can be performed on preselected boundaries or events are:


- snap to closest boundary or event in level above (Keyboard Shortcut `t`),
- snap to closest boundary or event in level below (Keyboard Shortcut `b`), and
- snap to nearest zero crossing (Keyboard Shortcut `x`).

To add a new boundary or event to a level the user initially has to select the desired level she or he wishes to edit. This is achieved either by using the up and down cursor keys or by single-left-clicking on the desired level. The current preselected level is marked in a darker shade of gray, as is displayed in Figure \ref{fig:webApp_selectLevel}.


![Screenshot of two levels as displayed by the EMU-webApp, where the lower level is preselected (i.e., marked in a darker shade of gray).](pics/selectLevel.png)

To add a boundary to the currently selected level one first has to select a point in time either in the spectrogram or the oscillogram by single-left-clicking on the desired location. Hitting the enter/return key adds a new boundary or event to the preselected level at the selected time point. Selecting a stretch of time in the spectrogram or the oscillogram (left-click-and-drag) and hitting enter will add a segment (not a boundary) to a preselected segment level.

#### Segments and events

The EMU-webApp also allows segments and events to be preselected by single-left-clicking the desired item. The web application colors the preselected segments and events yellow to indicate their pre-selection as displayed in Figure \ref{fig:webApp_preSelSeg}.


![Screenshot of level as displayed by the EMU-webApp, where the /@/ segment is currently preselected as it is marked yellow.](pics/preSelSeg.png)

As with preselected boundaries or events the user can now perform multiple actions with these preselected items. She or he can, for example, edit the item's label by hitting the enter/return key (which can also be achieved by double-left-clicking the item). Other actions that can be performed on preselected items are:


- Select next item in level (keyboard shortcut `TAB`),
- Select previous item in level (keyboard shortcut `SHIFT` plus `TAB`),
- Add time to selected item(s) end (keyboard shortcut `+`),
- Add time to selected item(s) start (keyboard shortcut `SHIFT` plus `+`),
- Remove time to selected item(s) end (keyboard shortcut `-`),
- Remove time to selected item(s) start (keyboard shortcut `SHIFT` plus `-`), and
- Move selected item(s) (hold down `ALT` Key and drag to desired position).


By right-clicking adjacent segment or events (keyboard shortcut `SHIFT` plus left or right cursor keys), it is possible to select multiple items at once.

#### Parallel labels in segments and events

If a level containing time information has multiple attribute definitions (i.e., multiple parallel labels per segment or event) the EMU-webApp automatically displays radio buttons underneath that level (see red square in Figure \ref{fig:webApp_parTimeLevel}) that allow the user to switch between the parallel labels. Figure \ref{fig:webApp_parTimeLevel} displays a segment level with three attribute definitions.


![Screenshot of segment level with three attribute definitions. The radio buttons that switch between the parallel labels are highlighted by a red square.](pics/emu-webAppParTimeLevel.png)

#### Legal labels

As mentioned in Section \ref{subsubsec:emuDBlegalLabels}, an array of so-called legal labels can be defined for every level or, more specifically, for each attribute definition. The EMU-webApp enforces these legal labels by not allowing any other labels to be entered in the label editing text fields. If an illegal label is entered, the text field will turn red and the EMU-webApp will not permit this label to be saved.


### Working with hierarchical annotations [^5]

[^5]: This section is an updated version of the \textit{The level hierarchy} section of the \textit{General Usage} chapter that is part of the \texttt{EMU-webApp} own brief manual by Markus Jochim.

#### Viewing the hierarchy

As mentioned in Section \ref{sec:emu-webApp_mainLayout}, pressing the `show hierarchy` button (keyboard shortcut `h`) in the top menu bar opens the hierarchy view modal window[^6]. As with most modal windows in the EMU-webApp, it can be closed by clicking on the `close` button, clicking the X circle icon in the top right hand corner of the modal or by hitting the `ESCAPE` key. By default, the hierarchy modal window displays a horizontal version of the hierarchy for a spatially economical visualization. As most people are more familiar with a vertical hierarchical annotation display, the hierarchy can be rotated by hitting the `rotate by 90°` button (keyboard shortcut `r`). Zooming in and out of the hierarchy can be achieved by using the mouse wheel, and moving through the hierarchy in time can be achieved by holding down the left mouse button and dragging the hierarchy in the desired direction. Figure \ref{fig:webApp_hierModal} shows the hierarchy modal window displaying the hierarchical annotation of a single path (*Utterance* -> *Intonational* -> *Intermediate* -> *Word* -> *Syllable* -> *Phoneme* -> *Phonetic*) through a multi-path hierarchy of the *ae* emuDB in its horizontal form.

[^6]: The term modal window is used in user interface design to refer to pop-up windows that force the user to interact with the window before returning back to the main application.

![Screenshot of the hierarchy modal window level displaying a path through the hierarchy of the *ae* emuDB in its horizontal form.](pics/emu-webAppHierModal.png)



#### Selecting a path through the hierarchy

As more complex databases have multiple hierarchical paths through their hierarchical annotation structure (see Figure \ref{fig:annotStruct} for an example of a multi-dimensional hierarchical annotation structure), the hierarchy modal offers a drop-down menu to choose the current path to be displayed. Area 2 in Figure \ref{fig:webApp_hierModalTop} marks the hierarchy path drop-down menu of the hierarchy modal.


![Screenshot of top of hierarchy modal window of the EMU-webApp in which the area marked 1 shows the drop-down menus for selecting the parallel label for each level and area 2 marks the hierarchy path drop-down menu.](pics/emu-webAppHierModalTop.png)


It is worth noting that only non-partial paths can be selected in the hierarchy path drop-down menu.

#### Selecting parallel labels in timeless levels

As timeless levels may also contain multiple parallel labels, the hierarchy path modal window provides a drop-down menu for each level to select which label or attribute definition is to be displayed. Area 1 of Figure \ref{fig:webApp_hierModalTop} displays these drop-down menus.

#### Adding a new item

The hierarchy modal window provides two methods for adding new annotation items to a level. This can either be achieved by pressing the blue and white + button next to the level's name (which appends a new item to the end of the level) or by preselecting an annotation item (by hovering the mouse over it) and hitting either the `n` (insert new item before preselected item) or the `m` key (insert new item after preselected item).

#### Modifying an annotation item

An item's context menu[^7] is opened by single-left-clicking its node. The resulting context menu displays a text area in which the label of the annotation item can be edited, a play button to play the audio section associated with the item and a collapse arrow button allowing the user to collapse the sub-tree beneath the current item. Collapsing a sub-tree can be useful for masking parts of the hierarchy while editing. A screenshot of the context menu is displayed in Figure \ref{fig:webApp_hierContextMenu}.

[^7]: The term context menu is used in user interface design to refer to a pop-up menu or pop-up area that provides additional information for the current state (i.e., the current item).

![Screenshot of the hierarchy modal window of the EMU-webApp displaying an annotation item's context menu.](pics/emu-webAppHierContextMenu.png)

#### Adding a new link

Adding a new link between two items can be achieved by hovering the mouse over one of the two items, holding down the `SHIFT` key and moving the mouse cursor to the other item. A green dashed line indicates that the link to be added is valid, while a red dashed line indicates it is not. A link's validity is dependent on the database's configuration (i.e., if there is a link definition present and the type of link definition) as well as the *non-crossing constraint* \citep{coleman:lp1991a} that essentially implies that links are not allowed to cross each other. If the link is valid (i.e., a green dashed line is present), releasing the `SHIFT` key will add the link to the annotation.

#### Deleting an annotation item or a link

Items and links are deleted by initially preselecting them by hovering the mouse cursor over them. The preselected items are marked blue and preselected links yellow. A preselected link is removed by hitting `BACKSPACE` and a preselected item is deleted by hitting the `y` key. Deleting an item will also delete all links leading to and from it.



```{r, echo=FALSE, results='hide', message=FALSE}
# clean up emuR_demoData
unlink(file.path(tempdir(), "emuR_demoData"), recursive = TRUE)
```
