Validation Method

Once you have selected the validation data, you'll have to decide which validation method to follow. Basically there are four different methods (probably more):

• One-pass Check
  All selected annotations are presented together with the raw data to the validator and the validator decides for certain discrete categories, for instance phonemic label correct / wrong or deviation less than 10msec / deviation between 20 and 25 msec / deviation larger than 25 msec or gender correct / gender wrong^6.2 etc.
• Multiple-pass Check
  The same as the one-pass check but performed by a group of independent validators. The results of these checks have to be summarized accordingly. Very often concurring judgments across the group of validators are considered to be correct, while non-concurring judgments are double-checked by the supervisor who then decides, or - in the extreme - all non-concurring judgments are considered as errors. Needless to say, the multiple-pass check requires more man power.
• One-pass Re-annotation
  The annotation of the selected validation data is repeated under exactly the same conditions as during the speech corpus production by a different annotator (= validator). Then the results of both are compared to calculate deviations.^6.3

  This method is considered to be `more objective' because the validator is not biased by the results of the original annotation. This is certainly true for problematic linguistic items like `phonemic categories' or `segment boundaries'. However, this method has also its drawbacks: It is very hard to reconstruct the exact labeling conditions by the validator. You'll need at least one validator in your group that has the skills to do the annotation on the same level as the producer (or even better) and a supervisor who is able to ensure the quality of the annotation. Also, in most cases the inevitable discussion about ``what is to be considered correct'' ensues between the producer and the validator of the speech corpus.

  We consider this method to be very effective for the validation of annotations or meta data where the nature of the categorical system is well established and the validator has no problems to justify his/her annotations. These are typically: the gender of a speaker, the transcript, dialog act or word segmentation and simple linguistic and noise tagging. The following data types are considered to be problematic for re-annotation: phonemic, prosodic, syllable, morph segmentation, dialect, age, phrase accent and boundaries.

• Multi-pass Re-annotation
  Of course, as with the one-pass check, the method of re-annotation may also be extended to multiple-pass re-annotations. To make things short: everything gets much more complicated. In most cases this method is considered to be way too costly anyway.