Next: Questionnaires and Forms
Up: Recording Techniques
Previous: Field Recordings
Contents
Wizard-of-Oz Recordings
WOZ recordings may be designed in many different settings depending on
your needs or the features of the simulated application (`virtual
machine'); therefore it is difficult to give detailed instructions on
how to set up a WOZ recording site. Please also refer to the section
for general hints and tips as they probably
apply for WOZ recordings as well. Here are some general hints specially for the
WOZ technique that might be useful:
- Do not use a room with a one-way mirror. This technique is so
well-known by now that almost every speaker will become suspicious. If
you need to watch or video tape the speaker, use
small cameras instead, and if you cannot hide them successfully, simply
add them to the `virtual machine'5.9
- Take care that the location of the wizards (the `control room') is
acoustically insulated from the recording room, so that the wizard can
produce live speech output and/or communicate (possibly via an
intercom) with the recording supervisor.
- Since the microphones are further away from the recording
device in a WOZ setting than in a usual setting, make sure that you don't
get a hum on your signals. If you experience disturbances, try to set up
the microphone amplifier in the recording room and not in the control
room.
- In most WOZ recordings you will need a synthetic speech output of
the `virtual machine' instead of written feedback. According to our experience
it is next to impossible for a
wizard to type absolutely error free. On the other hand it is quite easy
to distort the human voice - for instance by using sound devices from
electronic music
stores - and to train the wizards to speak in a very controlled manner.
The only weak points in this setting are laughing or coughing (no machine
laughs or coughs!).
- Do a lot of training sessions with cooperative, non-naive users and
ask them what went wrong. Since WOZ recordings are very expensive because
of the manpower you need, it is better to do some extra off-line
training sessions before you recruit naive speakers.
- Clarify the legal aspects of recordings in which the recorded person is
unaware of the situation. For instance you should ask your legal
department whether it is ok not to tell the speakers immediately after the recording
that it was a fake. Because if you have to tell them the truth right after the recording,
very soon you will not be able to find naive speakers in the
vicinity any more.
- Simulating very complex (`intelligent') systems is difficult because
the wizard constantly has to decide whether a user input is
still within the capabilities of the `virtual machine'. A good way to
handle this is to prepare task flow maps. Task flow maps are like a
semantic map that shows the wizard which possible ways through a dialog
with the `virtual machine' are allowed and which are not. Since the
wizard has to think about it (time delay) it is also very helpful to have
some pre-recorded standard answer ready that can be played by just
pushing a button5.10. For instance there should be a general
``Sorry-I-did-not-understand-you'' button and probably others that fit
exactly to the weak points in the `virtual machine' setup.
- Keep the recording session short and keep the speakers occupied.
In long sessions it is more difficult to maintain the illusion that the
speaker talks to a machine. Bored speakers start to look around or think
about clever ways to trick the system.5.11
This is by no means a complete list of hints. Many problems you'll find
in the pre-tests might be solved by a better or more specific instruction
to the speakers as well.
Next: Questionnaires and Forms
Up: Recording Techniques
Previous: Field Recordings
Contents
BITS Projekt-Account
2004-06-01