Impacting the social presence of virtual agents by scaling the fidelity of their speech and movement

Julian Fietkau

February 19th, 2015

Master’s thesis

for the purpose of attaining the academic degree
Master of Science. (M.Sc.)

First supervisor: Prof. Dr. Frank Steinicke
Second supervisor: Prof. Dr. Martin Christof Kindsmüller

Fachbereich Informatik
Universität Hamburg


Contents

  1. Introduction
  2. Definitions
    1. Virtual Reality
    2. Virtual Agents
    3. Social Presence
    4. Fidelity
    5. Idle Motion
  3. Experiment
    1. Hypotheses
    2. Design
      1. Questionnaire
      2. Movement
      3. Speech
      4. Experimental Procedure
    3. Implementation
      1. Technical Components
      2. Assets
      3. Experimental Setup
      4. Experimental Procedure
  4. Results
    1. Evaluation
    2. Discussion
  5. Conclusion
  1. Questionnaire
  2. Data: Questionnaire
  3. Data: Experiment

Creative CommonsCC-BY-SA 4.0

This creative work is available in accordance with the terms of the Creative Commons Attribution Share-Alike 4.0 license. This means that, with very few restrictions, it may be freely copied, transferred and used for any purpose as long as the name of the author (Julian Fietkau) is clearly mentioned as the original creator and derivative works are made available under the same license. More information:

http://creativecommons.org/licenses/by-sa/4.0/

Abstract

Virtual agents are constructs that fulfill human or human-like roles in virtual environments, but are directly controlled by software instead of real humans. They have use cases such as presenting information, demonstrating actions or simulating a social environment. If a real person perceives them as sufficiently human-like, they may induce social phenomena like empathy, competition or conversational turn taking, even if the person is consciously aware that the agent is purely virtual.

This thesis explores the influence of technical fidelity on perceived social presence in terms of the virtual agents’ speech and movement. Both of these two variables were assigned different implementations of varying technical sophistication, from text-to-speech output to fully recorded voices and from a completely rigid idle body to a high-quality relaxed idle animation based on motion capturing data. The various combinations were tested in an experiment using a head-mounted virtual reality display in order to measure their influence on perceived social presence. This thesis describes the experiment and its results.

Keywords: avatars, head-mounted displays, social presence, virtual agents, virtual reality


1. Introduction

For several decades now, personal computers have been capable of producing real-time 3D graphics, predominantly used in games, that – even if they are not photorealistic – look convincingly enough like a spatial location to evoke a sense of immersion (Slater, Usoh, & Steed, 1994). In order to populate these environments, virtual agents (perhaps more commonly known as “non-player characters”) are commonplace. They are virtual humanoid characters controlled by software. Depending on the quality of their implementation, they may be a terrific addition to an immersive world, or they might feel artificial and jarring.

In the context of this thesis, we1 are concerned with the technical aspects of such implementations. Specifically, we investigate whether the technical quality of their voice or their animation has a strong influence on the user’s feeling of interacting with a person, even if they are aware that there is no real human behind the virtual agent. Some unconscious social actions might take place even in exchanges with virtual agents (Biocca, Harms, & Gregg, 2001).

We make a point of focusing on characteristics that are not easily communicated through a static screenshot. Voice and animation quality are perhaps not the first things to come to mind when we consider realism in virtual environments, but we think that neglecting them outright could have very negative consequences for the user’s feeling of social presence (a term we define in section 2.3).

On the other hand, if we know how that feeling interacts with the fidelity of our virtual agents’ voice and animation, then we would be better equipped to find compromises between it and development resources.

This, all in all, is why we decided to examine this specific area of VR research further, and conduct an experiment to produce some reliable answers.

We start out by defining a number of important concepts in chapter 2, relying on established knowledge wherever possible. In chapter 3 we describe our experiment in detail, from the initial idea through the design decisions and including a summary of the final implementation, before analyzing and interpreting the results in chapter 4. We close with a summary and conclusion in chapter 5. Bulk data can be found in the appendix.


1: Of course this is a master’s thesis, so any usage of the first person plural in the manuscript refers more or less exclusively to the author, who even has to certify that he wrote everything by himself. Still, we stick to this pronoun not only because it is the polite thing to do, but also as a respectful nod towards the friends, colleagues and advisors who contributed to discussions, talked about ideas or gave valuable feedback. Thank you!


2. Definitions

In order to create a common understanding of the core concepts of this work, it is vital that there be agreed-upon definitions. Wherever possible, we base our definitions on previous established works to increase the viability of this work as a stepping stone for future scientific progress.

2.1. Virtual Reality

The term virtual reality (VR) has historically often been defined in terms of the hardware used to convey a particular medial experience to a human user (Krueger, 1991). To alleviate the ties to concrete technological developments, Steuer (1992) proposes a definition based on the perception of the experience rather than the method of implementation, he defines virtual reality as “a real or simulated environment in which a perceiver experiences telepresence” (Steuer, 1992, p. 7), building upon his previously established definition for telepresence as “the experience of presence in an environment by means of a communication medium” (Steuer, 1992, p. 6).

Even though this definition might at first glance seem overly broad, the mandate of achieving telepresence using a communication medium (as opposed to natural human senses) covers a lot of past and future implementations, and Steuer makes a convincing case for not chaining the concept of VR to classes of hardware like head-mounted displays or data gloves, which is why we operate on the basis of his definition even though this work happens to have a concrete technical scope in that we focus on a VR experience using a head-mounted display (see section 3.3.1).

2.2. Virtual Agents

Even though virtual agents have been extensively studied in works such as Caridakis et al. (2008) or Kopp, Sowa, and Wachsmuth (2003), a systemic definition of the term is often not supplied and there does not seem to be an agreed-upon understanding of the term. We provide our own definition as follows.

We understand an agent (in the context of software programming) to be a software construct that possesses agency, i.e. something that distinguishes between its own behavioral autonomy and the environment in which it exists. An agent may have some perception of its environment, and its actions may have consequences within the environment. A virtual agent is then defined to be an agent that exists in a virtual reality.

Virtual agents are not virtual avatars, because the latter represent and are controlled by human users while the former are controlled by software (Blascovich & Bailenson, 2011). However, both belong to the overarching category of virtual actors. Some of the results of this experiment may be applicable to avatars as well as agents, but since we only tested agents, we do not wish to make any claims to that effect.

2.3. Social Presence

As mentioned above in section 2.1, Steuer (1992) provides a useful definition for the term “presence” (within the context of VR). While he also touches on telepresence, he does not talk about social presence. To find a good definition for this concept, we consult Biocca et al. (2001), who establish what they call “three dimensions of social presence”:

Co-presence
The degree to which the observer believes he/she is not alone and secluded, their level of peripherally or focally awareness of the other, and their sense of the degree to which the other is peripherally or focally aware of them.
Psychological Involvement
The degree to which the observer allocates focal attention to the other, empathically senses or responds to the emotional states of the other, and believes that he/she has insight into the intentions, motivation, and thoughts of the other.
Behavioral engagement
The degree to which the observer believes his/her actions are interdependent, connected to, or responsive to the other and the perceived responsiveness of the other to the observer’s actions.
From: Biocca et al. (2001, p. 2)

They further divide these three dimensions into various factors like awareness, attention, understanding, and interaction. However, their high-level overview is sufficient for our experiment.

2.4. Fidelity

The concept of fidelity (as it is understood in the context of technology) is etymologically rooted in “faith”/“faithful” and refers to “the degree to which something matches or copies something else” (Merriam-Webster Dictionary, 2015). As the presentation of our virtual agents aims to emulate the real world, we interpret the fidelity of a property of a virtual agent as something akin to a degree of closeness to the real-world counterpart.

We would further like to highlight the contrast between fidelity and realism. We understand fidelity to be an inherent property of the implementation of the virtual agent, the degree of fidelity is a design decision. Realism, on the other hand, is the (intended) result of a high degree of fidelity, it is inevitably influenced not only by the virtual agent, but also the rest of the VR experience, and it is dependent on a human observer.

Real-world applications have to make certain trade-offs when it comes to fidelity. Even though a higher degree of fidelity is helpful in achieving a more realistic experience, it also tends to be more difficult (and thus costly) to achieve than lower-fidelity alternatives. The examples given in sections 3.2.2 and 3.2.3 might illuminate the concept further.

2.5. Idle Motion

Even if a human being is not actively doing anything in particular, their body never stops moving completely. They unconsciously perform actions that we summarize as idle motion (Egges, Molet, & Magnenat-Thalmann, 2004), such as shifting their weight, slightly moving their arms, or mildly moving their head while gazing around. These actions are involuntary and require concentrated effort to be suppressed, which is why they are crucial for a virtual agent to appear convincingly “alive” instead of appearing to be a statue. So-called idle animations are commonplace for virtual agents in modern games (Starck, Miller, & Hilton, 2005). We hypothesize that the fidelity (or utter absence) of idle motion may have an influence on the virtual agent’s social presence.


3. Experiment

Our research is rooted in the question of how the technical fidelity of virtual agents influences their social presence in a VR setting. This chapter begins by formulating a number of hypotheses about the interrelations of speech and movement fidelity with user perception and behavior.

To evaluate the validity of our hypotheses, we conducted an experiment involving binary comparisons between pairs of virtual agents whose fidelity of speech and movement had been set to various preconfigured levels. The sections starting from 3.2 detail the design decisions that went into it as well as the technical execution. A summary of the results follows in chapter 4.

3.1. Hypotheses

The possible interactions between the kinds of fidelity that we intend to manipulate and the social presence of the virtual agents are manifold, but some ideas and hunches are certainly more obvious than others. For example, given that higher fidelity virtual agents tend to be more difficult to develop, and seeing that this development happens in real-world applications anyway, it is easy to assume that high-fidelity virtual agents are developed because they are better at producing the respective intended results (depending on the use case). If that is indeed the case, then it is also reasonable to look into whether stronger social presence may be a factor in their increased efficacy, which leads us to our first set of hypotheses:

Hypothesis 1a
A virtual agent with a higher technical fidelity in terms of speech will have a stronger social presence compared to one with a lower fidelity.
Hypothesis 1b
A virtual agent with a higher technical fidelity in terms of movement will have a stronger social presence compared to one with a lower fidelity.

These hypotheses imply positive correlations between the technical fidelity of the virtual agent in terms of one of the two properties speech and movement. To test them, we need to define how exactly we intend to manipulate their fidelity (which happens in sections 3.2.2 and 3.2.3) and we have to provide a measure for their social presence, which we outline at the beginning of the following section 3.2.

Experimental data that would substantiate the above two hypotheses would allow us to infer further details. For example, there could be interaction effects between the fidelity of the two properties – or for the sake of simplicity, it might make sense to assume that they act independently until proven otherwise.

Hypothesis 2
Changes in the fidelity of speech and changes in the fidelity of movement will independently influence the social presence of the virtual agent.

Since we have full control over the experimental software, we are at liberty to record the time that the participants take to make their choices. The next step would then be to draw conclusions from the decision duration time to the difficulty of the choice – it seems reasonable that someone would take more time to make a decision if the choice is extraordinarily difficult.

If we can define a metric for the difficulty of the choice between two of our virtual agents, then we might find a correlation to the duration of the choosing phase.

Hypothesis 3
Comparing virtual agents in terms of social presence is easier (faster) if they have a big difference in technical fidelity in terms of speech and/or movement.

For the sake of scope, it should be noted that any systematic analysis of social presence in virtual agents will have to make abstractions from real-world use cases. For example, our experiment can not feature a rich and complex VR scenario with large numbers of virtual agents interacting with different users and with one another. In order to be able to make empirically substantiated claims, we have to reduce the interaction between the virtual agents and the study participants to a clearly defined minimum to ensure clarity and reproducibility.

3.2. Design

Even though we are building upon an existing definition of social presence, there are no substantiated methods to measure it on a scale in an experimental setting. As a simple tool to enable comparisons between the different degrees of fidelity, we construct our experiment around singular binary comparisons. Pairs of differently configured virtual agents are presented to the participant, who judges them in relation to one another and points out the one with the stronger social presence (see figure 1). This process is repeated for all pairs of configurations.

Sketch
Figure 1: This is the original concept sketch of the virtual experimental setup. The camera is positioned in an otherwise unremarkable scene with two virtual agents who perform an action of speech one after the other, after which the participant decides which of the two has a stronger social presence.

As we are testing two different axes of technical fidelity, namely speech and movement, we design independent degrees of fidelity. Each of them gets implemented as three different realizations, which are described in detail in sections 3.2.2 and 3.2.3.

We also decide to focus our research on a setting where the participant uses a head-mounted display (HMD) instead of commodity display hardware. We do this in order to increase the participant’s sense of presence, since it has been established that head-mounted displays have that effect (Pausch, Proffitt, & Williams, 1997). It stands to reason that a higher sense of presence on the user’s part could also lead to a higher sensitivity for social presence of virtual agents, or at least it should not be detrimental – however, we are not aware of any empirical proof for this conjecture. At the very least, VR is a research field where we observe a healthy dialogue and openness to new ideas regarding the construction of virtual agents.

3.2.1. Questionnaire

We created a digital questionnaire to guide the participant through the experiment procedure. After the initial greeting, the participant sits down at the desk and finds the questionnaire displayed on the PC monitor in a fullscreen web browser window.

The questionnaire consists of several segments, their order determined by the structure of the experiment, which will be enumerated as follows.

The full questionnaire can be found in appendix A as a display variant optimized for printing.

Demographic and Biological Data

The first few questions cover standard demographic information such as age, gender and occupation. Since the experiment deals with the participants’ reactions to acts of speech, among other things, we also ask about their degree of familiarity with the German language, since people with less proficiency might interpret speech (even those that are only pseudo-German, see section 3.2.3) differently or more slowly than someone whose mother tongue is German.

To gauge the influence of medical issues regarding vision or hearing, we also inquire about known issues in those two areas as well as about any vision and hearing corrections that may exist.

Furthermore, we ask participants about their experience with 3D games, 3D stereoscopic displays, and head-mounted displays, as each of these could have an influence on the way that virtual agents are perceived.

Lastly, participants are asked to state their handedness (left- or righthanded, or ambidextrous) and their inter-pupillary distance, the latter of which is measured in the laboratory.

Hearing Assessment

Even though participants are asked about any issues with their hearing capacity, we strive to make doubly sure that there are no directional hearing problems, not even potentially unknown ones, that could jeopardize our reliance on directional audio signals during the experiment. To that end, we conduct a very brief directional hearing assessment of each participant using the Home Audiometer software by Esser (2012–2015). It tests both ears’ hearing capacity across the frequency spectrum typically audible to humans and displays the results graphically.

The questionnaire makes it abundantly clear to the participants that our hearing assessment is, for a number of reasons, not a substitute for any medical procedure. Our audio equipment is not professionally calibrated, we’re likely to have high levels of ambient noise (e.g. due to the technical equipment in the laboratory and the relative proximity to the Hamburg Airport), and our personnel are not trained to make any medical diagnoses. However, the results of the hearing assessment would give us the possibility to react to any detectable directional hearing issues that might occur.

Lateral Preference Inventory

The Lateral Preference Inventory – or, in full, the Lateral Preference Inventory for Measurement of Handedness, Footedness, Eyedness, and Earedness, and in short, the LPI – is a set of 16 questions developed by Coren (1993). It is intended to determine the four abovementioned lateral preference indices (hand, foot, eye, ear). We include it in our questionnaire to acquire some more detailed information than just the participants’ stated handedness, especially since any lateral preferences for vision and hearing might be relevant for our results even though the participants themselves might not even be consciously aware of them.

Simulator Sickness Questionnaire

The Simulator Sickness Questionnaire created by Kennedy, Lane, Berbaum, and Lilienthal (1993) is a standard tool to gauge the extent to which the participant might be affected by simulator sickness (also known as cybersickness), a set of short-term symptoms that can arise if a person spends a prolonged amount of time using VR hardware.

The SSQ is split into a pre- and a post-experiment half, each consisting of identical questions about the participant’s subjective well-being. It is designed to detect whether any temporary health effects (such as nausea, eyestrain, or dizziness) are produced or amplified by the experiment.

Post Questionnaire

The general post questionnaire consists of a small number of questions tailored to our experiment and the local circumstances. Specifically, we ask the participants about any outside distractions that might have occurred and about their opinion of the experiment, including opportunities for free-form answers and feedback.

Slater-Usoh-Steed Questionnaire

The Slater-Usoh-Steed Questionnaire intends to measure a VR system’s degree of immersion as defined by Slater et al. (1994). In the scope of this thesis, we are not overly concerned with the concept of immersion by itself, but the questionnaire still provides valuable data about how the participants perceived the experience and the extent to which they themselves had a sense of presence.

3.2.2. Movement

Since the social actions of our virtual agents are heavily based on speech, their movement might seem like a secondary concern. However, in order to create a convincing social presence, the usage of suitable idle motions (see section 2.5) is a big contributor to social presence (Egges et al., 2004).

For the highest degree of fidelity that is feasible, we use idle animations based on high resolution motion capturing data, a process that creates animations for virtual agents based on recording the movements of real actors (Moeslund, Hilton, & Krüger, 2006). In real-life applications, this is a costly approach compared to, for example, keyframe animation (which entails a 3D animator creating several “key” poses and interpolating in-between movements), but understandably provides more realistic results. For the purposes of our experiment, we rely on commercially available high-quality animations that surpass anything that we could produce in the local laboratory.

The obvious opposite end of the movement fidelity scale is the completely frozen virtual agent with no idle motions at all. This is trivially easy to implement, fulfilling our expectation that lower-fidelity approaches tend to have a smaller resource impact during development.

For the in-between step, a keyframe-based animation would be a possible middle ground in terms of fidelity, and the comparison between the social presence for keyframe animations versus motion capturing animations in the general case would certainly be of interest. However, for our specific experiment, such a comparison would be difficult to generalize, because any difference in perceived social presence may just as well be rooted in the specific movements that make up the two animations we would use instead of their overall categories. In other words: We would only be comparing one specific keyframe animation with one specific motion capturing animation. To mitigate this issue and permit a general inference, we would have to compare a large number of examples from each category so that we would be able to prove the presence of statistically significant differences, but this is too far beyond the scope of our experiment to be feasible.

Instead, we base our in-between step on the full motion capturing animation, but manipulate it in a way that reduces its fidelity. To that end, we exclude parts of the 3D model from the idle animation, namely the hands and the legs. For the hands, we simply ignore them altogether, leaving them non-animated. For the legs, instead of using the motion-capturing data, we enable a feature called inverse kinematics (Tolani, Goswami, & Badler, 2000), which describes a set of algorithms that are capable of making sure that the virtual agent’s feet stay connected to the ground, even if the upper body moves (or the ground becomes uneven, which is not applicable to our experiment). As a result, the legs no longer use the prerecorded idle animation, but instead do the minimal amount of movement that is needed to plausibly support the upper body. We believe that this approach is a suitable compromise to reduce the movement fidelity.

3.2.3. Speech

There are many kinds of acts of speech that could be considered viable for our experiment. Depending on the use case, virtual agents in different applications may be used to ask questions, deliver instructions, perform back-and-forth conversations or fulfill any number of communicative roles.

However, since the experiment specifically attempts to test for effects of the technical fidelity of the speech, our aim was to provide as little distraction through the content of the speech as possible. Since our experiment relied on direct comparisons, clearly both sides of any comparison would need to execute the same act of speech, so that any bias that might arise from the content of the speech would be symmetrically canceled out.

Finding Suitable Acts of Speech

Ideally, we would like to rely on being able to make comparisons even across the different trials, which is why the differences in terms of speech content between trials should also be minimized. One way of achieving this would be to reuse the very same sentence over and over for every single trial. However, we suspected that this approach would lead to increased monotony during the experiment, since a full run would encompass a large number of trials. This could produce a more tiring experience for the participants, which would in turn reduce the quality of the data. We also suspected that continued use of the same sentence could lead to semantic satiation, a psychological phenomenon by which a word or phrase seems to lose its meaning and appears increasingly alien if it is repeated a sufficiently large number of times (Esposito & Pelton, 1971). These problems could be mitigated by the use of a number of different sentences instead of a single one, but that introduces variance into the process of understanding and interpreting the speech that could also detract from our results.

This is how we arrived at the idea of using gibberish speech (speech that is more or less phonetically and/or syntactically plausible, but does not contain any discernable meaning) instead of real acts of speech. Ideal gibberish speech would enable us to use a variety of different acts of (pseudo-)speech to stave off boredom and semantic satiation, while also keeping all speech at a constant level of semantic contents, that being none at all.

This raises the question of how to generate “high-quality” gibberish, in the sense that it should be nonsensical enough to not contain any meaning, yet sound plausible and familiar enough not to appear overly foreign. Fortunately, solutions to this problem have already been developed. We used a pseudoword generator named Wuggy to create our gibberish, which is based on existing psycholinguistic research (Keuleers & Brysbaert, 2010). It is capable of creating polysyllabic pseudowords from any given list of real words while preserving the phonetic constraints of the source language. We used a dataset courtesy of the Wortschatz project (Institut für Informatik, Universität Leipzig, 2001) containing the 1000 most common German words, from which we had to filter 16 abbreviations2. The remaining 984 German words were fed into Wuggy to be used as the basis for our gibberish.

The resulting list of pseudowords was then shuffled randomly to produce sentences of 12 words each. When spoken out loud, each one of them is four to five seconds long, which we considered a reasonable length to enable the participants to judge the speech.

The eight sentences that we used in our experiment are as follows:

  1. Kie Verpreils Hopitie Phraxe metes scheches krumciespiel Dimen wor klück Mozualiin Zaß.
  2. Putaun ehte pflon veßten düfflich La Fing hürte Kopp geripten Südchen Daude.
  3. Lychte rafen Fahl toswenden lält luchsgans gorm dadee Spresten ebstbals vesses Newage.
  4. Sis fist Lab Wuderfet kühe Hamte veuten Läuen alny Bopie schäler belögte.
  5. Allerlochs spöbten stekken hanuß bes Beren Rie fal rereis Piedes lanter dabbte.
  6. Tonzerr for Turicht gopen Gander fürr jor nasen hührend rusband zusel Händern.
  7. Vorkau hind nirgst ehka ätmehin umhächst zondern zöln giesen kolst begids Belsallem.
  8. Gesprals Marf hillten fiesen Rottel zockte Jen arrhen peit rafe Wuloner zührend.

It should be noted that word capitalization is essentially random, although we made sure to manually capitalize the first word in each sentence.

From Written Words to Audio Signals

To go from the above pseudospeech to audio signals to be played during the experiment, we first had to define the degrees of fidelity to serve as a basis for the experiment.

A viable approach to low-fidelity speech is text-to-speech (TTS) software. This term describes software that is capable of taking pure text as an input and converting it to audible speech. Detailing the various approaches to this problem in general would be vastly beyond the scope of this work, but plenty of literature on the subject exists (Sproat, 1997). We are largely interested in the results that the current “state of the art” can produce, so we did a short preliminary analysis of free and commercial consumer-grade text-to-speech systems, with the constraint that they had to support German TTS, since our pseudowords were based on the German phonetic structure.

We evaluated the following applications:

After listening to some example output from each application and comparing them in terms of vocal fluidity, phonetic plausibility and sound quality (this was a subjective comparison without any quantified justification), we decided to use the IVONA software as our text-to-speech solution for the experiment. However, the differences between the various products were not glaring, and the research field of speech synthesis is bound to make further improvements in the upcoming years. IVONA was able to read our gibberish without issue and we got the corresponding sound files out of it.

At the opposite end of the fidelity scale, it seemed like the obvious choice to create a fully human-voiced set of recordings. We used an adult male voice for the TTS files, so we had a real adult male listen to them and recorded his voice in attempting to read the sentences at the same speed and with the same inflection. We were not able to create an exact match, but we got as close as we could within our constraints.

To create a third stage in between the previous two, a middle ground between text-to-speech and full voice recording, we took the recorded sound files and made some alterations to them. We duplicated the waveform and played it at a delay of 5 milliseconds, which is too short to be perceived as an echo, but produces a tinny, metallic sound. We also cut out most of the small portions of silence within the recordings (see figure 2), which creates “jumps” in the audio recording that would be impossible to achieve by a real human mouth, but that we observed to be reminiscent of the audible inaccuracies found in text-to-speech sound samples. This leaves us with a set of sound files that still sound somewhat like a real voice (at least more than the TTS output does), and yet differentiate themselves from the full recording enough to be slightly uncanny.

Waveforms
Figure 2: This is a visualization of the waveform of our first recorded gibberish sentence. The top one displays the unaltered recording, while the bottom one represents the modified recording with most of the silent parts (highlighted in green) cut out.

3.2.4. Experimental Procedure

As described above, we have chosen the two properties speech and movement as our variable degrees of technical fidelity, which we manipulate independently in three steps each. This means that we have 3 × 3 = 9 possible ways to combine the two properties for each of our virtual agents. To reduce confusion, we will call them configurations (of the virtual agent) in order to differentiate them from the pairs of configurations, which we will call constellations.

Since we ask our participants to compare the configurations in pairs, we would ideally want to pair every configuration with every other one (deliberately excluding constellations where both configurations would be identical), which leaves us with 9 × 8 = 72 constellations. This number already includes symmetrical constellations, i.e. if we understand a constellation to be a two-tuple of configurations, and (C1,C2) is part of our set of constellations, then so is (C2,C1). Even though this doubles the number of trials per participant compared to the hypothetical situation where we would exclude mirrored constellations, they are indeed a big help in reducing the impact of any (conscious or unconscious) lateral preferences on the part of our participants.

Furthermore, we have to keep in mind that our experiment displays the two configurations in each constellation sequentially. As a result, for each of the above 72 constellations, we include it twice: once starting with the left configuration and following with the right, and once starting with the right configuration followed by the left. From here on out, we will call them left-to-right and right-to-left constellations, respectively. This doubles the total number again, leaving us with our final number of 72 × 2 = 144 trials per participant.

With such a big number of trials, each single one has to be very short if the experiment is to be completed in one sitting. With each of the two configuration displays lasting five seconds, and the decision time expected to be between one and three seconds approximately, we expect a total duration of about 12 seconds per trial. At 144 trials in total, we arrive at an expected experiment length of just under 30 minutes, which seems adequate.

3.3. Implementation

This section describes in further detail how our experiment was put together. In particular, we describe the technologies we used, the location as well as other details of the experimental setup, and we explain some noteworthy problems and other occurances from the execution of the experiment.

3.3.1. Technical Components

The central hardware component of our experiment is the Oculus Rift DK27 head-mounted display. It has a 1920 × 1080 pixel display covering a 100° horizontal field of view as well as various internal sensors for directional and positional head tracking (Oculus VR, LLC, 2014-2015). It is connected to a standard desktop PC which also has mouse and keyboard for input as well as a traditional LCD monitor.

The beyerdynamic MMX 28 provides the sound component of the VR experience. It is advertised as a “gaming headset” and also contains a microphone, which was not used during the experiment. It is capable of reproducing sound in the range of 18 to 22000 Hz (beyerdynamic GmbH & Co. KG, 2012-2015).

We decided to use the Unity Game Engine9 (version 4.5) as the basis for our virtual reality experience, which not only has the capability of interacting with the Oculus Rift HMD, but also has a proven track record as a relatively easy to use basis for real-time 3D applications in scientific contexts (Craighead, Burke, & Murphy, 2007). It runs on modern PCs on top of Microsoft Windows and encapsulates many difficulties of multimedia (in particular real-time 3D graphics) programming behind a graphical interface coupled with freely available documentation. The Unity Engine handles the aspects such as camera projection, lighting, and texturing so that we were able to focus on integrating our assets and programming the experiment.

As mentioned in section 3.2.1, we use the Home Audiometer software written by Esser (2012–2015) to perform a brief non-medical hearing assessment. For an example of what the results of an assessment look like, see figure 3.

The questionnaire was delivered through Google Forms10 in a standard web browser.

Hearing assessment
Figure 3: These diagrams show an example result from a hearing assessment done with the Home Audiometer software (Esser, 2012–2015). For both ears individually, the application tests various frequencies for their audibility at increasing volumes (the higher the line, the lower the volume, the better the hearing). The results shown here are unremarkable because they stem from a young adult with a healthy hearing capacity.

3.3.2. Assets

We used the MakeHuman11 software to create the 3D model of our virtual agent. It is capable of producing highly detailed and textured 3D models of human bodies that can be adjusted according to various physiological parameters. Our virtual agent is based largely on the MakeHuman defaults with the gender set to 100% male, the race being caucasian, and the physique being slim/athletic. The nondescript black hair and suit are also part of the MakeHuman default assets and proved easy to integrate. See figure 4 for a visual representation.

Man
Figure 4: This is what our virtual agent looks like under ideal lighting and texturing conditions. The MakeHuman software makes it feasible to create human 3D models like this without much knowledge about 3D modeling. Please note that this is a high-resolution render image using idealized lighting and that the real-time 3D representation in the Unity Engine has distinctly lower visual fidelity.

During the initial implementation of the virtual agent and the integration of the sound recordings, it quickly became obvious that the connection between the virtual agent and the voice recordings was not readily apparent as long as there was no mouth movement. Naturally, a human’s mouth moves while they talk, so we decided to implement some primitive lip-synchronization into our virtual agent. There are some lip-sync solutions available for the Unity Engine, for example FaceFX12, but their complexity would have been prohibitive at that stage of implementation. Instead, we implemented a barebones lip-sync algorithm written by UnityAnswers forum user Naletto (2011) – see figure 5 – which reads the audio file’s spectrum data to poll the sound amplitude over a certain time interval and use it to manipulate (stretch, move, etc.) any Unity object.

function BandVol(fLow: float , fHigh: float ): float
{
    fLow = Mathf.Clamp(fLow, 20, fMax); // limit low...
    fHigh = Mathf.Clamp(fHigh, fLow, fMax); // and high frequencies
    // get spectrum: freqData[n] = vol of frequency n * fMax / nSamples
    audio.GetSpectrumData(freqData, 0, FFTWindow.BlackmanHarris);
    var n1: int = Mathf.Floor(fLow * nSamples / fMax);
    var n2: int = Mathf.Floor(fHigh * nSamples / fMax);
    var sum: float = 0;
    // average the volumes of frequencies fLow to fHigh
    for (var i=n1; i<=n2; i++){
        sum += freqData[i];
    }
    return sum / (n2 - n1 + 1);
}

var mouth: GameObject;
var volume = 40;
var frqLow = 200;
var frqHigh = 800;
private var y0: float;

function Start()
{
    y0 = mouth.transform.position.y;
    freqData = new float [nSamples];
    audio.Play();
}

function Update()
{
    mouth.transform.position.y = y0 + BandVol(frqLow,frqHigh) * volume;
}

// A function to play sound N:
function PlaySoundN(N: int)
{
    audio.clip = sounds[N];
    audio.Play();
}
Figure 5: This is the code supplied by Naletto (2011) on the UnityAnswers forum that accomplishes rudimentary automated lip synchronization. While an audio file is being played, this script analyzes the spectrum data and manipulates the y position of a predetermined game object accordingly.

We applied a suitable scaling to the value and used it to move the jaw bone of our virtual agent downwards synchronized to the audio signal. The result is obviously difficult to appreciate in print, but a pair of screenshots can be seen in figure 6. Thanks to the high quality of the 3D mesh produced by MakeHuman, the simple act of moving the jaw bone results in relatively plausible and visually pleasing facial deformations. Even though it would likely not fool a face-to-face observer, it is convincing enough for use with our HMD and VR scene, where there’s a constant distance between the participant and the virtual agents that renders small inaccuracies invisible.

Lip sync
Figure 6: This pair of images shows the impact of the lip-sync script on our virtual agent. The idea of simply moving the jaw bone downwards in proportion to the volume of the sound file is crude, but works surprisingly well.

3.3.3. Experimental Setup

We set up our experiment in a room within the main HCI laboratory (Fachbereich Informatik, Universität Hamburg). While the laboratory itself was partially in use during the experiment, our room was seperated by a wall and a door.

Every part of the experiment took place on or around a table that we placed in the middle of the room (see figure 7), with one chair for the participant positioned as if the table were a normal desk, and one chair off to the side for the researcher. The PC was positioned under the table towards the left, with keyboard, mouse and monitor on the tabletop. Participants completed the questionnaire facing the monitor, while for the hearing assessment it was turned to face the researcher and to make it impossible for the participant to read the results of the assessment while it was in progress.

Participants only wore the headphones and the HMD whenever each was needed for the experiment. For the rest of the time, they were kept on the left side of the table. The software setup made it feasible to have both the monitor and the HMD connected and running at the same time without interfering with each other.

Water and snacks were available to participants during break times, but were stored on a shelf behind the researcher while the experiment was in progress.

Experiment
Figure 7: This photo shows one participant sitting at the table, wearing the HMD and the headphones during the experiment. The keyboard, mouse and monitor are also visible, as is the researcher’s laptop. The screen shows the Unity Engine running the experimental VR scene. In the background of the photo, the mostly empty experimental room is visible, with the rest of the HCI laboratory behind the glass windows.

3.3.4. Experimental Procedure

Volunteer participants were acquired from the students at the Fachbereich Informatik as well as the research staff. There was no material compensation for participation, either financial or otherwise.

After being greeted and going over the experiment consent form, the participant would start by filling out the questionnaire page by page, with the measurement of the inter-pupillary distance, the hearing assessment, and the HMD phase in between.

The hearing assessment involved the participant pressing the Ctrl key on the keyboard whenever they heard a noise. The audiometer software would adjust the volume and the frequency and switch between the left and right ear. The results of the assessment were stored with the rest of the experimental data.

The HMD section of the experiment involved 144 trials per participant, as explained above. It started with an explanation how to choose between the two virtual agents with the arrow keys and how to advance using the spacebar (see figure 8). Participants were shown a short summary of the social presence definition by Biocca et al. (2001) (see section 2.3) in order to know how to make the comparisons and were given the opportunity for prior questions. The 144 trials were broken up into 12 blocks of 12 trials each, with opportunities to take a break between blocks.

As explained in section 3.2.4, we expected a length of about 30 minutes for the HMD phase, which turned out to be rather accurate. In addition to that, the hearing assessment took 10 minutes and the questionnaire about 20 minutes per participant, adding up to an hour in total, which was also within our expectations. A few participants took longer breaks than others, which led to a total time of up to 80 minutes in some instances.

There were no significant problems or distractions throughout the experiment. On a few occassions, the hearing assessment was momentarily disrupted by passing planes (the laboratory is geographically close to an airport), but this proved to not be a problem.

VR 1
VR 2
Figure 8: This pair of screenshots shows the experimental VR scene. In the top image, the two virtual agents are displayed and the one on the right is currently talking – the scene puts an additional highlight on the talking agent as an added visual focus cue. In the bottom image, both of them have finished talking and the program is waiting for user input. The participant has to press either the “←” or the “→” key. The instructional message is displayed in German if the participant’s mother tongue is German.

2: The following abbreviations were manually removed from the word list: AG, CDU, CSU, DDR, DM, dpa, Dr., EU, FDP, GmbH, Mio, Mrd, SPD, USA, WELT, z.B.

3: https://translate.google.com/

4: http://www.ivona.com/

5: http://www.linguatec.de/products/tts/voice_reader/vrs15

6: http://imtranslator.net/translate-and-speak/speak/german/

7: https://www.oculus.com/dk2/

8: http://www.beyerdynamic.de/shop/mmx-2.html

9: http://unity3d.com/

10: https://docs.google.com/forms/

11: http://www.makehuman.org/

12: http://facefx.com/


4. Results

In this chapter, we examine the results of our experiment and interpret the data we gathered in such a way as to evaluate the hypotheses from section 3.1.

To start off, it bears mentioning that we had n = 15 participants aged between 19 and 45 years (M = 26.65, SD = 6.76), which should be enough to infer some statistically significant results. However, some of the answers we received make it clear that any results gathered from this experiment are not certain to be applicable to the populace at large. For example, all of our participants had a computer science background (10 with an HCI specialization, 5 without), all of them were native speakers of German, none of them suffered from any notable disorders in vision or hearing, and all participants were right-handed. Any conclusions we draw from the experimental data should only be relied on with these caveats in mind until the experiment can be repeated with participants of a more varied background.

4.1. Evaluation

As we decided early on that our trials would be binary comparisons between different virtual agent configurations, we can now look at the “winner” of each trial (the configuration that was chosen). If we look at how often each value for speech was in the winning configuration (cf. table 1, figures 9 & 10), we observe mean counts for the text to speech condition of M = 33.53 (SD = 12.92), for the modified recording condition of M = 48.47 (SD = 14.40), and for the full recording condition of M = 61.40 (SD = 9.49). Analogously, for the different idle motion values (cf. table 2), we observe mean counts for the “no idle motion” condition of M = 34.20 (SD = 13.52), for the reduced idle motion condition of M = 52.33 (SD = 8.27), and for the motion capturing idle motion condition of M = 56.87 (SD = 8.06).

All of the value counts are normally distributed across subjects according to a Shapiro-Wilk test at the p < 0.05 level.

Using the Kruskal-Wallis rank sum test, we can not confirm at the p < 0.05 level that the winning counts for the different degrees of fidelity are based on underlying distributions with distinct location parameters.

A χ2 test did not assert any interdependence between the winning speech values and the winning idle motion values across all trials.

subject idtext to speechmodified recordingfull recording
1166662
2234477
3444456
4256356
5354960
6495045
7275562
8383472
9304167
10375551
11166365
12177453
13583351
14462375
15423369
Mean33.5348.4761.40
SD12.9214.409.49
Table 1: These are the absolute counts of how often each value for speech fidelity has been in the winning configuration per participant.
subject idtext to speechmodified recordingfull recording
1534447
2523854
3146565
4394263
5264969
6206262
7246060
8355257
9345153
10176165
11465048
12395154
13435742
14524349
15196065
Mean34.2052.3356.87
SD13.528.278.06
Table 2: These are the absolute counts of how often each value for idle motion fidelity has been in the winning configuration per participant.
Fidelity winners
Figure 9: These diagrams show the number of times, for each participant, when a particular value for the fidelity of speech or movement was part of the winning configuration. As the fidelity gets higher, the corresponding variable is chosen more often and more consistently.
Fidelity winners cumulative
Figure 10: These two diagrams show the cumulative number of times each value of the two variables appeared in the winning configuration across all participants. This makes the weight towards the higher-fidelity values more obvious.

We analyzed the effects of display order (left to right or right to left) and the randomly chosen gibberish sentence on the winning fidelity values with a repeated measures ANOVA and paired-samples t-tests. For the winning speech value, there are no significant interactions. For the winning idle motion value, there is a highly significant interaction between the winning value and the gibberish sentence, but no significant interactions with the display order, nor any interaction effects between the sentence and the display order.

For the time it took the participants to make their individual binary choices (from here on out “choice duration”), we measured delays between 189 and 14850 milliseconds (M = 1384, SD = 1303). Even though the value range spans two orders of magnitude, even the highest outliers exist within the realm of plausibility, which is why we do not discard any of the data points (see figure 11).

Duration boxplot
Figure 11: This box plot shows the spread of the choice duration (milliseconds). Even though a large majority of all data points are lower than 1500 milliseconds, there are outliers up to ten times as big. Altogether, 233 points exist outside the 1.5 × IQR distance (233 of 2151, 10.8%).

To create a useful measure for the difference in fidelity between two configurations, we have to set a fidelity value for each individual configuration. We do this by interpreting the three values of each of our two variables as integer values in {0, 1, 2}, with 2 being the highest fidelity value and 0 being the lowest. We then define the fidelity value of a configuration as the sum of the fidelity values of its two components. Lastly, we define the fidelity distance as fdAB = |fvAfvB| (visualized in figure 12).

Fidelity distance
Figure 12: This is a tabular visualization of the fidelity distance between two configurations, defined as fdAB = |fvAfvB|. The fidelity distance between the top left and the bottom middle configuration is given as an example.

If we examine the distribution of the choice duration in relation to the fidelity distance (see figure 13), we see that there are visual hints for a small negative correlation, and the Pearson product-moment correlation coefficient of the two variables is indeed −0.036 at the p < 0.1 level.

Duration distance boxplot
Figure 13: This is an array of box plots showing the spread of the choice duration depending on the fidelity distance. The plots largely resemble the independent one shown in figure 11. For fd ≤ 2 there is not much variation, but for fd = 3 and especially fd = 4 it is obvious that the choice duration has far fewer outliers and even a slightly lower median.

The remaining parts of the questionnaire did not lead to any interesting results. The Lateral Preference Inventory aligned very well with the stated handedness of the participants and did not offer any further insight, the Simulator Sickness Questionnaire (fortunately) gave no signs of any health problems more significant than mild fatigue.

4.2. Discussion

Going back to our hypotheses from section 3.1, we are unfortunately not able to substantiate hypotheses 1a and 1b (the existence of significant correlations between speech or idle motion fidelity and the social presence of the virtual agent) based on our experimental data. To the naked eye it seems apparent that the higher fidelity configurations were chosen more often, but it appears that our sample size compared with the relatively small difference between the values (between one to two standard deviations) is not big enough to prove it conclusively, at least not at any worthwhile level of significance. However, it does seem like a worthwhile avenue for further research.

It could be a boon to focus on only one of our two fidelity scales per experiment – perhaps it would have been easier to prove a correlation, even with the same number of participants, if all trials were geared towards one scale of comparison instead of mixing both. At this stage that is a wild guess though.

In reference to hypothesis 2, we were able to provide some evidence that there are no significant interaction effects between the effects that the fidelity of speech and the one of idle motion have on social presence. Even though true independence can not be statistically proven, the data suggests that it is a safe assumption that there are no interactions between the two.

Regarding hypothesis 3 (a negative correlation between the overall fidelity and the duration of the choice phase), we were indeed able to prove the existence of a negative correlation at the p < 0.1 level. The effect is small but noticeable. From a user perspective it is not very surprising that configurations with a greater fidelity distance are easier to compare, but it is reassuring that the data corroborates the hypothesis.

Although it is hard to say whether a larger sample size would have led to more significant results, as a suggestion for the future it seems prudent to say that a larger number of participants would likely benefit experiments of this kind. Furthermore, our conjecture is that the idea behind our experimental trials – the binary comparison of two configurations with each combining more than one variable – generates only a small amount of usable information per trial. Maybe a proper measurement scale for social presence would make it easier to draw reliable conclusions, even if it comes at the cost of increased experimental duration.


5. Conclusion

During the work on this thesis, we consolidated several different sources to create suitable notions of both technical fidelity and social presence and to relate them to one another. We conceived an experimental framework based around large numbers of fast-paced binary comparisons to measure the social presence of virtual agents based largely on participants’ instant reactions, implemented these ideas into a real-time VR scenario capable of using a state-of-the-art HMD, and executed our experiment with 15 participants, proceeding to analyze the gathered data statistically.

Our intention for this experiment was to investigate any interactions between the fidelity of virtual agents’ speech and movement (specifically idle motions) and their social presence in a virtual reality context – one based around a head-mounted display with directional tracking, in our case.

It is unfortunate that we were unable to prove whether higher technical fidelity of virtual agents leads to a stronger social presence, which would have helped explain and steer some of the current developments around virtual agents. In the absence of such statistical proof and with us only having been able to draw some incidental conclusions, the experiment would have to be considered a partial success at best. But of course not every experimental result has to be groundbreaking, and especially in its role as part of a master’s thesis, perhaps this lesson that not every experiment can lead to clean results every time is all the more fitting.13

Viewed from a more constructive perspective, it bears mentioning that our framework for virtual agents’ technical fidelity and how to manipulate it along different axes is a potentially useful tool that did not exist before we developed it in anticipation of our experiment. Now that at least two examples for three-step fidelity manipulation have been established, it will be much easier for future experiments in the same area to establish controlled and reproducible fidelity circumstances.


13: And needless to say, the author learned a lot about laboratory experiments, VR technology, real-time 3D engines and other methods and technologies throughout the preparation and execution of this thesis.


References

beyerdynamic GmbH & Co. KG. (2012-2015). beyerdynamic MMX 2. Retrieved February 18th, 2015, from http://www.beyerdynamic.de/shop/mmx-2.html
Biocca, F., Harms, C., & Gregg, J. (2001). The networked minds measure of social presence: Pilot test of the factor structure and concurrent validity. In 4th annual international workshop on presence (pp. 1–9). Philadelphia, PA, USA.
Blascovich, J., & Bailenson, J. (2011). Infinite reality: Avatars, eternal life, new worlds, and the dawn of the virtual revolution. New York, NY, USA: William Morrow & Co.
Caridakis, G., Raouzaiou, A., Bevacqua, E., Mancini, M., Karpouzis, K., Malatesta, L., & Pelachaud, C. (2008). Virtual agent multimodal mimicry of humans. Language Resources and Evaluation, 41 (3–4), 367–388.
Coren, S. (1993). The lateral preference inventory for measurement of handedness, footedness, eyedness, and earedness: Norms for young adults. Bulletin of the Psychonomic Society, 31 (1), 1–3.
Craighead, J., Burke, J., & Murphy, R. (2007). Using the unity game engine to develop sarge: a case study. Computer, 4552, 366–372.
Egges, A., Molet, T., & Magnenat-Thalmann, N. (2004). Personalised real-time idle motion synthesis. Computer Graphics and Applications, 121–130.
Esposito, N. J., & Pelton, L. H. (1971). Review of the measurement of semantic satiation. Psychological Bulletin, 75 (5), 330–340.
Esser, T. (2012–2015). Home Audiometer Hörtest. Retrieved February 18th, 2015, from http://www.esseraudio.com/de/home-audiometer-hoertest.html
Institut für Informatik, Universität Leipzig. (2001). Projekt Deutscher Wortschatz – Wortlisten. Retrieved February 18th, 2015, from http://wortschatz.uni-leipzig.de/html/wliste.html
Kennedy, R., Lane, N., Berbaum, K., & Lilienthal, M. (1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. International Journal of Aviation Psychology, 3 (3), 203–220.
Keuleers, E., & Brysbaert, M. (2010). Wuggy: A multilingual pseudoword generator. Behavior Research Methods, 42 (3), 627–633.
Kopp, S., Sowa, T., & Wachsmuth, I. (2003). Imitation games with an artificial agent: From mimicking to understanding shape-related iconic gestures. Gesture-Based Communication in Human-Computer Interaction, 5th International Gesture Workshop.
Krueger, M. W. (1991). Artificial reality 2 (2nd ed.). Reading, MA, USA: Addison-Wesley.
Merriam-Webster Dictionary. (2015). Fidelity – Definition and more. Retrieved February 18th, 2015, from http://www.merriam-webster.com/dictionary/fidelity
Moeslund, T. B., Hilton, A., & Krüger, V. (2006). A survey of advances in vision-based human motion capture and analysis. Computer Vision and Image Understanding, 104 (2–3), 90—126.
Naletto, A. (2011). UnityAnswers: Any way of “automatic” lip syncing? Retrieved February 18th, 2015, from http://answers.unity3d.com/questions/139323/any-way-of-quotautomaticquot-lip-syncing.html
Oculus VR, LLC. (2014-2015). The All New Oculus Rift Development Kit 2 (DK2) Virtual Reality Headset. Retrieved February 18th, 2015, from https://www.oculus.com/dk2/
Pausch, R., Proffitt, D., & Williams, G. (1997). Quantifying immersion in virtual reality. In Proceedings of the 24th annual conference on computer graphics and interactive techniques. New York, NY, USA: ACM.
Slater, M., Usoh, M., & Steed, A. (1994). Depth of presence in virtual environments. Presence: Teleoperators and Virtual Environments, 3, 130–144.
Sproat, R. W. (Ed.). (1997). Multilingual text-to-speech synthesis. Norwell, MA, USA: Kluwer Academic Publishers.
Starck, J., Miller, G., & Hilton, A. (2005). Video-based character animation. In Proceedings of the 2005 acm siggraph/eurographics symposium on computer animation (pp. 49–58). New York, NY, USA: ACM.
Steuer, J. (1992). Defining virtual reality: Dimensions determining telepresence. Journal of Communication, 42, 73–93.
Tolani, D., Goswami, A., & Badler, N. I. (2000). Real-time inverse kinematics techniques for anthropomorphic limbs. Graphical models, 62 (5), 353–388.

Please note that there are several mentions of commercial products and/or websites in this thesis, some of which we deliberately excluded from this reference list, so that it contains only the sources from which we cite information. If a product or website is merely mentioned in context, but not cited, a web link is included as a footnote in the main text instead of here.


Appendix

The experiment used a digital questionnaire to acquire subject data beyond the boundaries of the HMD experiment. The following is a complete reproduction of the questionnaire. It uses the “for print” view in lieu of the web-based version to facilitate inclusion in a print document, so any references that seem counterintuitive (for example directions to click a button) would make more sense in the interactive web-based version of the questionnaire.

The experiment included n = 15 participants in total. Each one was guided through the questionnaire and the experiment. Since the experiment contained 144 trials per subject, the total maximum number would have been 15 × 144 = 2160 trials. However, 9 trials were faulty and had to be discarded, mostly because of outside interruptions and short-term software failures, leaving 2151 trial data points available for interpretation.

Because the measurement data from the hearing assessments was not used in the evaluation of the experiment (partly because there was no need, partly because none of the results were at all surprising or interesting), and even though the participants consented to a full publication of all experimental data, we have decided not to include the hearing assessment results with this publication because we feel that the participants’ interest in keeping potentially medically sensitive data safe and anonymous weighs heavier than the interest of the public in fully open data access in this particular case.


A. Questionnaire

Experiment Questionnaires

All details are collected only in the context of the present study. Thank you for your participation!
* Required
Age *
 
Height *
 
Profession / field of study: *
 
Gender *
Mark only one oval.
  • OvalMale
  • OvalFemale
How would you rate your German language skill? *
Mark only one oval.
  • OvalNative speaker
  • OvalFluent
  • OvalProficient
  • OvalBasic
  • OvalNone
Vision correction: *
Mark only one oval.
  • OvalNone
  • OvalGlasses
  • OvalContact lenses
Do you have a known eye disorder?
Check all that apply.
  • SquareColor blindness
  • SquareNight blindness
  • SquareDyschromatopsia (red-green color weakness)
  • SquareStrong eye dominance
  • SquareOther:
     
Do you suffer from hearing loss?
Mark only one oval.
  • OvalNo (healthy hearing capacity)
  • OvalMild hearing loss (difficulties understanding speech)
  • OvalModerate to severe hearing loss (impossible to understand speech)
  • OvalProfound hearing loss (impossible to hear speech or most noises)
If you suffer from hearing loss, please check all that apply:
Check all that apply.
  • SquareAsymmetrical hearing loss, more pronounced on the left side
  • SquareAsymmetrical hearing loss, more pronounced on the right side
  • SquareSymmetrical hearing loss (both ears affected at about the same level)
  • SquareCongenital hearing loss (present since birth)
  • SquareAcquired/Delayed hearing loss (onset later in life)
Hearing correction:
Mark only one oval.
  • OvalNone
  • OvalExternal hearing aids
  • OvalCochlear implants
  • OvalOther:
     
Do you suffer from a displacement of equilibrium or similar? *
Mark only one oval.
  • OvalYes
  • OvalNo
Do you have any experience with virtual reality HMDs (such as the Oculus Rift)? *
Mark only one oval.
1 2 3 4 5
no experience
Oval
Oval
Oval
Oval
Oval
a lot of experience
Do you have experience with 3D computer games? *
Mark only one oval.
1 2 3 4 5
no experience
Oval
Oval
Oval
Oval
Oval
a lot of experience
How many hours do you play per week? *
 
Do you have experience with 3D stereoscopic display (cinema, games etc.)? *
Mark only one oval.
1 2 3 4 5
no experience
Oval
Oval
Oval
Oval
Oval
a lot of experience
Are you left- or right-handed? *
Mark only one oval.
  • OvalLeft-handed
  • OvalRight-handed
  • OvalAmbidextrous
Inter-pupillary distance (IPD) *
Please contact the experimenter to measure your IPD.
 

Hearing assessment

Please contact the experimenter for a short assessment of your hearing ability (approximately 10 minutes). Please note: This is a very broad test that serves only to highlight any obvious patterns in the context of our experiment. Our staff does not (and can not) perform medical diagnoses. This assessment is not a substitute for a hearing test conducted by trained personnel using calibrated equipment. If you suspect that your hearing may be impaired, please arrange further steps with your medical doctor.

The Lateral Preference Inventory

Simply read each of the questions below. Decide which hand, foot, etc. you use for each activity and then put a check mark next to the answer that describes you the best. If you are unsure of any answer, try to act out the action.
With which hand do you draw? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Which hand would you use to throw a ball to hit a target? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
In which hand would you use an eraser on paper? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Which hand removes the top card when you are dealing from a deck? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
With which foot would you kick a ball to hit a target? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
If you wanted to pick up a pebble with your toes, which foot would you use? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Which foot would you use to step on a bug? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
If you had to step up onto a chair, which foot would you place on the chair first? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Which eye would you use to look through a telescope? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
If you had to look into a dark bottle to see how full it was, which eye would you use? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Which eye would you use to peep through a keyhole? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Which eye would you use to sight down a rifle? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
If you wanted to listen in on a conversation going on behind a closed door, which ear would you place against the door? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Into which ear would you place the earphone of a transistor radio? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
If you wanted to hear someone’s heartbeat which ear would you place againsttheir chest? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither
Imagine a small box resting on a table. This box contains a small clock. Which ear would you press against the box to find out if the clock was ticking? *
Mark only one oval.
  • OvalLeft
  • OvalRight
  • OvalEither

Simulator Sickness Questionnaire (Pre)

General discomfort (DE: "Unwohlsein") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Fatigue (DE: "Ermüdung") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Headache (DE: "Kopfschmerzen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Eyestrain (DE: "Ermüdung der Augen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Difficulty focusing (DE: "Schwierigkeiten mit der Sehschärfe") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Increased salivation (DE: "Erhöhte Speichelbildung") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Sweating (DE: "Schwitzen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Nausea (DE: "Übelkeit") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Difficulty concentrating (DE: "Konzentrationsschwierigkeiten") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Fullness of head (DE: "Druckgefühl im Kopfbereich") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Blurred vision (DE: "verschwommene Sicht") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Dizzy (eyes open) (DE: "Schwindelgefühl bei geöffneten Augen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Dizzy (eyes closed) (DE: "Schwindelgefühl bei geschlossenen Augen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Vertigo (DE: "Gleichgewichtsstörungen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Stomach awareness (DE: "Magenbeschwerden") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Burping (DE: "Aufstoßen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe

Experiment Procedure

In the experiment you will be asked to perform a task in a virtual environment while wearing a head-mounted display as well as headphones. You will see and hear pairs of virtual actors performing an act of speech. You will then be prompted to decide, for each pair, which one has the stronger "social presence" (this term is defined on an introductory slide during the experiment). Each trial lasts about 12 to 15 seconds. The experiment will be conducted in blocks of 12 trials (about 2.5 minutes each) and will end once all 12 blocks have been completed. The experiment usually takes about 30 minutes. You may take short breaks between blocks, but please try to hold your concentration throughout each block, as the trials within a block happen consecutively. Thank you! (Please click "continue".)

You are now ready to start the experiment. Please contact the experimenter.

If you have completed the experiment, please click "continue".

Simulator Sickness Questionnaire (Post)

General discomfort (DE: "Unwohlsein") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Fatigue (DE: "Ermüdung") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Headache (DE: "Kopfschmerzen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Eyestrain (DE: "Ermüdung der Augen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Difficulty focusing (DE: "Schwierigkeiten mit der Sehschärfe") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Increased salivation (DE: "Erhöhte Speichelbildung") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Sweating (DE: "Schwitzen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Nausea (DE: "Übelkeit") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Difficulty concentrating (DE: "Konzentrationsschwierigkeiten") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Fullness of head (DE: "Druckgefühl im Kopfbereich") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Blurred vision (DE: "verschwommene Sicht") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Dizzy (eyes open) (DE: "Schwindelgefühl bei geöffneten Augen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Dizzy (eyes closed) (DE: "Schwindelgefühl bei geschlossenen Augen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Vertigo (DE: "Gleichgewichtsstörungen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Stomach awareness (DE: "Magenbeschwerden") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe
Burping (DE: "Aufstoßen") *
Mark only one oval.
1 2 3 4
None
Oval
Oval
Oval
Oval
Severe

Post Questionnaire

Did you feel immersed in the virtual world? *
Mark only one oval.
1 2 3 4 5
no
Oval
Oval
Oval
Oval
Oval
yes
Were you distracted from the virtual world by real-world ambient noise? *
Mark only one oval.
1 2 3 4 5
no
Oval
Oval
Oval
Oval
Oval
yes
Have you been able to see parts of the real laboratory during the experiment? *
Mark only one oval.
1 2 3 4 5
no
Oval
Oval
Oval
Oval
Oval
yes
Do you think the experiment task was too difficult? *
Mark only one oval.
1 2 3 4 5
no
Oval
Oval
Oval
Oval
Oval
yes
Do you think the experiment was too long? *
Mark only one oval.
1 2 3 4 5
no
Oval
Oval
Oval
Oval
Oval
yes
How would you subjectively describe your level of attention during the experiment? *
Mark only one oval.
1 2 3 4 5
very low
Oval
Oval
Oval
Oval
Oval
very high
Which strategy did you use (e.g., concentrating on certain signals, making a "decision from the gut", etc.)? *
 
 
 
 
 
Any observations regarding the difficulty of the task that you made during the experiment and would like to share?
 
 
 
 
 
Additional comments:
 
 
 
 
 

Slater-Usoh-Steed Questionnaire (SUS)

Please rate your sense of being in the virtual environment, on a scale of 1 to 7, where 7 represents your normal experience of being in a place. *
I had a sense of “being there“...
Mark only one oval.
1 2 3 4 5 6 7
not at all
Oval
Oval
Oval
Oval
Oval
Oval
Oval
very much
To what extent were there times during the experience when the virtual environment was the reality for you? *
There were times when the virtual environment was the reality for me...
Mark only one oval.
1 2 3 4 5 6 7
not at all
Oval
Oval
Oval
Oval
Oval
Oval
Oval
almost all the time
When you think back to the experience, do you think of the virtual environment more as images that you saw or more as somewhere that you visited? *
The virtual environment seems to me to be more like...
Mark only one oval.
1 2 3 4 5 6 7
images that I saw
Oval
Oval
Oval
Oval
Oval
Oval
Oval
somewhere that I visited
During the time of the experience, which was the strongest on the whole, your sense of being in the virtual environment or of being elsewhere? *
I had a stronger sense of...
Mark only one oval.
1 2 3 4 5 6 7
being elsewhere
Oval
Oval
Oval
Oval
Oval
Oval
Oval
being in the virtual environment
Consider your memory of being in the virtual environment. How similar in terms of the structure of the memory is this to the structure of the memory of other places you have been today? By ‘structure of the memory’ consider things like the extent to which you have a visual memory of the virtual environment, whether that memory is in colour, the extent to which the memory seems vivid or realistic, its size, location in your imagination, the extent to which it is panoramic in your imagination, and other such structural elements. *
I think of the virtual environment as a place in a way similar to other places that I have been today...
Mark only one oval.
1 2 3 4 5 6 7
not at all
Oval
Oval
Oval
Oval
Oval
Oval
Oval
very much so
During the time of your experience, did you often think to yourself that you were actually in the virtual environment? *
During the experiment I often thought that I was really standing in the virtual environment...
Mark only one oval.
1 2 3 4 5 6 7
not very often
Oval
Oval
Oval
Oval
Oval
Oval
Oval
very much so

B. Data: Questionnaire

subject idTimestampAgeHeightProfession / field of study:GenderHow would you rate your German language skill?
12014-12-19 15:19:4622190Human-Computer-InteractionMaleNative speaker
22014-12-19 16:18:0527180Student InformatikMaleNative speaker
32014-12-19 17:25:1421177Student InformatikMaleNative speaker
42014-12-19 18:34:4124172HCIFemaleNative speaker
52014-12-19 19:33:2419181MCIMaleNative speaker
62014-12-22 11:19:2525172Bachelor MCIMaleNative speaker
72014-12-22 14:26:1833182Post-doc CSMaleNative speaker
82014-12-22 16:11:5933185InformaticsMaleNative speaker
92014-12-23 11:35:2034168MCIFemaleNative speaker
102014-12-23 14:05:3024180MCIMaleNative speaker
112014-12-23 17:07:3945155computer scienceFemaleNative speaker
122014-12-23 19:01:5720192Computer ScienceMaleNative speaker
132015-01-12 15:18:2221167MCI StudentFemaleNative speaker
142015-01-12 18:00:2724183HCIMaleNative speaker
152015-01-12 19:12:4128173phd studentMaleNative speaker
subject idVision correction:Do you have a known eye disorder?Do you suffer from hearing loss?If you suffer from hearing loss please check all that apply:
1NoneNo (healthy hearing capacity)
2GlassesNo (healthy hearing capacity)
3GlassesNo (healthy hearing capacity)
4GlassesNo (healthy hearing capacity)
5NoneNo (healthy hearing capacity)
6GlassesNo (healthy hearing capacity)
7Contact lensesNo (healthy hearing capacity)
8NoneNo (healthy hearing capacity)
9GlassesNo (healthy hearing capacity)
10None
11NoneNo (healthy hearing capacity)
12NoneNo (healthy hearing capacity)
13NoneNo (healthy hearing capacity)
14NoneNo (healthy hearing capacity)
15GlassesMild hearing loss (difficulties understanding speech)Symmetrical hearing loss (both ears affected at about the same level)
subject idHearing correction:Do you suffer from a displacement of equilibrium or similar?Do you have any experience with virtual reality HMDs (such as the Oculus Rift)?Do you have experience with 3D computer games?
1NoneNo13
2NoneNo44
3NoneNo15
4NoneNo21
5NoneNo25
6NoneNo15
7NoneNo53
8NoneNo35
9NoneNo12
10NoneNo15
11NoneNo44
12NoneNo21
13NoneNo21
14NoneNo33
15NoneNo55
subject idHow many hours do you play per week?Do you have experience with 3D stereoscopic display (cinema, games etc.)?Are you left- or right-handed?Inter-pupillary distance (IPD)
112Right-handed4.4
2104Right-handed6.1
3102Right-handed7.2
443Right-handed5.8
5253Right-handed6.5
6123Right-handed6.6
705Right-handed6.5
834Right-handed6.0
903Right-handed6.2
10103Right-handed6.7
110.23Right-handed5.7
1264Right-handed6.5
1303Right-handed5.6
14103Right-handed6.5
15205Right-handed6.8
subject idWith which hand do you draw?Which hand would you use to throw a ball to hit a target?In which hand would you use an eraser on paper?Which hand removes the top card when you are dealing from a deck?With which foot would you kick a ball to hit a target?
1RightRightRightLeftRight
2RightRightRightEitherRight
3RightRightRightRightRight
4RightRightRightEitherEither
5RightRightRightEitherRight
6RightRightRightRightRight
7RightRightEitherRightRight
8RightRightRightLeftRight
9RightRightRightRightRight
10RightRightRightRightRight
11RightRightRightRightRight
12RightRightRightEitherRight
13RightRightRightEitherRight
14RightRightRightRightRight
15RightRightEitherEitherRight
subject idIf you wanted to pick up a pebble with your toes which foot would you use?Which foot would you use to step on a bug?If you had to step up onto a chair which foot would you place on the chair first?Which eye would you use to look through a telescope?If you had to look into a dark bottle to see how full it was which eye would you use?
1RightRightRightLeftLeft
2RightRightEitherEitherEither
3RightRightEitherRightRight
4RightLeftLeftRightRight
5RightRightLeftEitherEither
6EitherEitherRightRightRight
7EitherEitherLeftRightEither
8RightRightRightLeftRight
9RightEitherRightLeftLeft
10RightRightRightRightRight
11RightRightRightRightRight
12RightEitherRightRightRight
13EitherEitherRightLeftLeft
14RightRightRightRightRight
15EitherEitherEitherRightRight
subject idWhich eye would you use to peep through a keyhole?Which eye would you use to sight down a rifle?If you wanted to listen in on a conversation going on behind a closed door which ear would you place against the door?Into which ear would you place the earphone of a transistor radio?If you wanted to hear someons heartbeat which ear would you place againsttheir chest?
1LeftLeftRightEitherEither
2EitherEitherLeftRightLeft
3RightRightLeftLeftLeft
4RightRightRightRightRight
5RightLeftRightRightRight
6RightRightLeftLeftRight
7EitherRightEitherEitherEither
8RightLeftLeftLeftLeft
9LeftLeftRightRightLeft
10RightRightRightRightLeft
11RightRightRightRightLeft
12RightRightRightEitherEither
13LeftLeftLeftRightEither
14RightRightRightRightRight
15RightRightEitherEitherEither
subject idImagine a small box resting on a table. This box contains a small clock. Which ear would you press against the box to find out if the clock was ticking?General discomfortFatigueHeadacheEyestrainDifficulty focusing
1Right12111
2Left12121
3Left22221
4Right12111
5Right22322
6Right11111
7Left11121
8Left11121
9Left12123
10Right11211
11Either11111
12Right11121
13Left13121
14Right12131
15Either11111
subject idIncreased salivationSweatingNauseaDifficulty concentratingFullness of headBlurred vision
1111111
2111111
3121221
4111211
5111212
6111111
7111211
8111111
9111312
10121111
11111111
12111211
13211211
14111211
15111111
subject idDizzy (eyes open)Dizzy (eyes closed)VertigoStomach awarenessBurpingGeneral discomfort
1111111
2111112
3111112
4222121
5111111
6111112
7111111
8111112
9111111
10111111
11111111
12111113
13111112
14111111
15111111
subject idFatigueHeadacheEyestrainDifficulty focusingIncreased salivationSweating
1311111
2212111
3222211
4322211
5213111
6211111
7111111
8212111
9213211
10121111
11112111
12313111
13413121
14314211
15211111
subject idNauseaDifficulty concentratingFullness of headBlurred visionDizzy (eyes open)Dizzy (eyes closed)
1111111
2211112
3223112
4123122
5121211
6111211
7111111
8112111
9121111
10112111
11111211
12121111
13122111
14132111
15111111
subject idVertigoStomach awarenessBurpingDid you feel immersed in the virtual world?Were you distracted from the virtual world by real-world ambient noise?
111143
211132
321144
411242
511142
611141
711142
811131
911121
1011131
1111111
1211223
1311122
1411122
1511132
subject idHave you been able to see parts of the real laboratory during the experiment?Do you think the experiment task was too difficult?Do you think the experiment was too long?How would you subjectively describe your level of attention during the experiment?
11123
21234
31124
41124
51114
62124
72114
81124
91115
101134
113314
125132
132124
142114
151123
subject idWhich strategy did you use (e.g., concentrating on certain signals, making a "decision from the gut", etc.)?
1decision from the gut, voice
2decision from the gut, clean audio
3concentrating on audio and movement of the body, speach clearlyness
4teils spezielle Signale, teils Bauchgefühl
5I mainly concentrated on the voice of the actors, but didn't really have a strategy elsewise. ``from the gut'' describes it pretty well.
6Comparing the actor's pattern of movement, i.e. choosing the actor with the most natural movement while speaking his text.
7motion > no motion, actual voice > tts, rest from the gut
8at first hearing experince, then body language and facial expressions
9in erster Linie habe ich nach dem Ton ausgewählt, zu technische, zu klare und wie bei einem Außenreporter verzerrte Sprache. ist als erstes rausgeflogen. Ansonsten hab ich mich auf mein Bauchgefühl verlassen und keine richtige Strategie verfolgt.
10differentiate between moving and non-moving person, differentiate between natural speech and synthezid speech
11I thought of one of them being the real person and the other as a virtual language teacher. Still it was not easy to decide.
121. loudest speaker, 2. if equal, the one who moves, 3. generally what felt best
13- allgemeiner Eindruck - ob Stimme ``in den Raum'' passt - Aufmerksamkeitsrichtung des Sprechers (auf mich gerichtet oder sonstwohin) - bei gleichem Eindruck Ausfall nach dem Motto: ``Zu wem passt die Stimme besser''
14decision from the gut, clearer voice maybee
15Movement & computer voice vs recorded voice as hints
subject idAny observations regarding the difficulty of the task that you made during the experiment and would like to share?
1
2
3
4man konnte jeden einzelnen Bildpixel sehen, stört den ``Realismus''
5
6Slight difficulties fitting my normal glasses in the Oculus Rift, but nothing too complicated.
7
8
9
10
11When it was exactly the same recording I had difficulties to choose.
12the Oculus Rift has a too low resolution for prolonged watching -> the eyes feel severe pain
13Die beiden Personen blinzeln wenig/gar nicht/schlecht zu erkennen, was dazu beigetragen haben kann, dass ich selber weniger geblinzelt habe und dadurch die Augen mehr angestrengt wurden.
14
15Headtracking would be nice. Felt like the actors looked past me sometimes.
subject idAdditional comments:
1
2
3very nice setup (and chair ;-) )
4das neu laden der Szene nach jedem Vergleich hat das Bild manchmal gefühlt leicht springen lassen (gefühlt leichter Ruck nach rechts oder links) - führte zu leichten Schwindel-Attacken
5
6No.
7nice work!
8if the voice sounds ``metallic/robotic'' than the experience is reduced in naturalness
9
10
11
12
13
14
15
subject idPlease rate your sense of being in the virtual environment, on a scale of 1 to 7, where 7 represents your normal experience of being in a place.To what extent were there times during the experience when the virtual environment was the reality for you?When you think back to the experience, do you think of the virtual environment more as images that you saw or more as somewhere that you visited?During the time of the experience, which was the strongest on the whole, your sense of being in the virtual environment or of being elsewhere?Consider your memory of being in the virtual environment. How similar in terms of the structure of the memory is this to the structure of the memory of other places you have been today? (...)During the time of your experience, did you often think to yourself that you were actually in the virtual environment?
1477624
2655555
3522532
4445543
5422754
6522562
7657765
8445354
9344511
10311232
11444412
12111171
13315542
14223422
15444444

C. Data: Experiment

subject_idtrial_idbody_leftbody_rightspeech_leftspeech_rightsentenceorderrepetitiontrial_configchoicedurationbody_winnerspeech_winnerfid_distancefid_diff
10022030043014940240
11210110010515171120
122201610110114782111
13121161078012791111
142001410100120080131
15200010097115100022
1612215009305171220
17021161030035650122
18200250010118810240
19011181024128691111
110202250013308982222
11101105002107160120
11210013005115170120
113121150077043761111
11402104102809970131
115020041012121742022
116001241020128530211
11720013009917150131
1182100700103115111011
119122161094029191220
120201250011715170231
1212110810120016432122
122120030059014621011
1232100810104121241011
1242011410116017432122
125121281080117432222
126222050014109142222
127101141068121240111
128101261070018091120
12902115002901890122
130010221010115441233
13110002105005181011
132120270063111132233
133210230010715171231
134020161014122072133
135212081013605172233
13612127007917982222
137120281064141122233
13800213003515170111
1392210500125025702111
140212230013918811211
14112203009108971231
1422201500109137472111
14311018105615171111
14402127003115172233
14501222104209480211
146100141052126710120
147110170055119091111
1482212810128111962211
149101130067123420111
150012181040040790220
15110226108618480211
1522021410132122890133
153002141036118750111
1542210610126023232111
155020270015114112244
156211070011906162122
1572111100121119751111
158012061038032670231
159210241010815171231
16001018108110471122
1612011300115117940122
162020030011017100022
163022161046119582131
164011210025015110122
165210121010605172020
16601021009151231233
167212070013506992233
168001230019037140111
16911108107205171111
170201021011406322133
171122281096130362211
17201006106125711011
17320002109805172022
174222061014209642222
17500023003146260222
17610001004907821011
177111210073116271211
178022041044022410240
179012170039010800220
180110210057110801222
1812122410140129191211
18201106102209480120
1832202700111111302222
1842212700127013452111
18501205003705670231
1862202810112082862022
18700024104123390222
188121030075010471120
18900012102111130111
1902221700143013942211
19102228104818152222
1922112300123123561220
1930100500505660011
19411107007106161111
19511122107408651111
1960101700705170022
197201010011304382133
19810202108206821233
1992121100137035482222
1100110221058146251222
110110025005318650231
1102102130083125540122
1103022150045110472131
110410225008517490211
1105002010033010800222
1106102010081026371233
110702015001315172133
1108202010012905172244
110912104107608811120
11102012610118015282131
11112021300131117260133
11122221810144023892211
1113211121012205172111
111401117002305170111
111511208108805171222
111612204109208481231
1117102141084125540122
1118112110089127701111
111902103002707330131
112012016106218002122
11212002610102119080240
1122212121013816831122
1123002021034017760222
1124112121090113451111
1125120150061146432122
1126100261054114450231
1127001021018020910111
112810102106615170022
11292022610134141790222
1130202021013005172244
1131011221026015770122
113212004106015172011
113302227004715832222
11342112410124018912120
1135101250069018591120
1136122270095133662211
11370001100117500111
1138020281016132352244
1139021281032011300133
1140001010017038790111
1141012210041163151211
114210101006505171122
1143112070087017091222
20011061022026220120
21011221026014770122
222210610126013962111
23022281048112292222
24012050037011630231
252112300123162491220
26011210025112631222
2701005005013620011
28202250013306992222
2900012102112460111
2102020100129018422244
21120013009909312031
21211018105617991111
2132121210138013952222
214221281012815212211
21500023003113620222
21611208108805171222
21721108101200112552122
21820126101180111902131
219200021098117430022
220012061038010130231
22101221004115171211
22212104107608811120
2232021300131021072233
22412028106415182233
225211241012416661220
22600213003505170211
22701018108011470022
22812203009105171231
229121270079110132222
2300101700719801122
23102003001105180022
232122161094110472120
233211121012208822111
234012170039028690220
235222170014304052211
236222181014409142211
237102010081029351233
23802004101218822022
2392100700103126701011
2400001100105180011
24110214108409151222
242021161030010300122
24300102101809310111
244021041028010630131
245022150045021900231
2462001410100110320131
247220270011119142222
24802028101619142244
249222061014205332222
250210121010615171120
25102127003107490133
252101250069112290220
253021030027010460131
25410002105011900011
2552120810136011142233
25611122107405171111
2572101100105010142020
258210241010808652031
2592020210130010142244
260002010033010130222
2612022610134112790222
262210081010406172011
263211070011906502122
264112070087010311222
26502016101417322133
26611212109005831211
267120150061032501022
26811017005515171111
26911021005716991222
27010101006519640022
27100101001705170111
272001241020124060211
273122281096010801211
27411108107209481111
27511121007315171211
27612215009305171220
2772201610110010472011
27812128108019482222
27902227004705170222
28001118102419321111
281112110089010971211
28210126107015170220
283220150010916162111
28412204109206991231
28501022101015201233
286022030043040780240
287102021082011871233
28812227009509991211
289212070013505172233
29001021009114281233
291120161062110142122
29212115007718482111
293011170023017430111
29401222104205170211
29510102106618980022
296120270063111962233
297200010097010962022
298221270012715172211
29901218104007980220
2100202141013207982233
210112116107819472111
2102211110012109312111
2103220281011218822222
21042102300107116771231
210500024104110140222
21062010100113010802133
2107200250010116170240
210800123001919640211
2109100261054110140231
211010213008309471222
2111120041060112622011
211211022105815181222
2113222050014109812222
2114102261086110800211
211502115002915672122
2116002141036017090211
211710013005107151020
21181110700710115881111
2119020270015119092244
212010014105219310120
2121100010049111290011
212210114106818980111
2123212110013708152222
21240100610607990011
212502204104415202040
21262011410116027042122
2127201250011718480231
21282010210114010302133
212902128103215212233
2130022161046010300231
2131200261010215180240
213212003005905171011
213302015001316002133
2134212230013909812211
21352011300115017432122
213612103007509151120
213710025005319800231
213801105002109480120
2139221050012509312111
214010225008516990211
214100202103408980222
214210113006715670111
2143212241014015171211
3012128108015182222
31120150061016431022
322001410100011132031
33221270012715172211
3402216104608150231
352201500109012452011
36012221042150721211
3712227009517162211
38221281012808482111
392202700111110142222
31000213003505180211
3111110810720112201111
3122122300139111801211
3132101210106110641120
31401022101015181233
315110170055112291111
31612115007717162111
317111221074112131211
318101130067029201111
319020041012127692022
320121030075010471120
321112110089010981211
322020161014112292133
323112070087011461222
324022041044111802040
325022281048110642222
3262022500133011132222
32702127003119142233
328212121013809812222
329122041092010641231
330222061014218322022
331121161078010141111
332102010081010971233
333200021098012122022
3342021410132012132233
335002010033036640222
33600102101815170011
337002141036010150211
338021030027112132031
3392011410116010812122
3402020100129012292244
34111018105609981011
3422021300131012632233
343020270015110482244
34400023003110470222
34502215004515172131
34611022105818811222
347021041028111632031
3482112410124011302120
349200130099012292031
350111070071113121011
351101010065011961122
352112081088118431022
353122150093112302120
35400024104110650222
35512028106419812233
35601121002519971222
357012050037110961031
3582102410108016762031
359102130083013121222
360102021082028861233
361100010049022731011
362100261054010471031
363100130051115940120
3642020210130010652244
365200010097011632022
366012210041110471211
367201010011309642133
368011181024110971111
3692112300123112471220
37002003001119642022
37112127007918982222
372100250053010131031
373111210073114611211
374012170039113631120
375120270063010651033
37600012102110130111
377120030059110962011
37802015001319312133
379020281016111312244
38000101001718640011
3812100700103010462011
38201005005110311011
383011061022111131020
384102141084013781222
3852221700143013792211
3860001100109810011
387212070013508652233
38801117002318151111
3892122410140112951211
390222050014118662022
391012061038112121031
392221050012518482011
393002021034110800022
394122030091014111231
395121041076018921120
396122161094125212120
39700124102017650211
398220281011209982022
399200250010105682040
3100102250085015441211
310110114106809141111
31022002610102010972040
3103211121012218981111
3104012181040111631120
310510002105008151011
31062012610118012792131
310700123001918650211
310812228109605171211
3109112121090010961211
311001006106111961011
311110125006908321120
31122121100137110141122
31132221810144110142111
3114211081012009142122
311510126107009811120
311602128103219312233
31172100810104011472011
3118021150029116102122
31192120810136084852233
3120110210057115601222
312101105002118981020
3122022270047111462222
3123022030043112622040
3124102261086011461211
3125100141052012461020
3126221061012619972011
3127210230010718981231
31280101700719971122
3129220161011008322011
313001018108113461122
313110102106608831122
31320102100918651233
31332012500117010472131
3134211110012118481111
3135211070011908652122
313612016106217992122
3137202261013407982222
3138201130011508482122
3139011221026111951222
3140210110010518321120
314112004106009811011
314202116103018482122
31432010210114011142133
40022161046114122131
412122300139014782211
42100021050130680011
432022500133164970222
4412216109415182120
452011410116114610122
462020210130021912244
47011050021010140120
48101010065010801122
492111100121033332111
41011018105618981111
411120270063111132233
412100130051110470120
413102010081012791233
414020030011011630022
415201250011709142131
416011210025011130122
417101141068110960111
418121041076111802020
419022281048110152222
420121150077025881111
421011170023116291111
422102021082110140033
4232111210122012122111
424012210041112291211
4252210500125011962111
426212081013609982233
427110221058112131222
428021150029020240122
42900024104110140222
4302212810128110632211
4312010100113010312133
432211070011908822122
433220270011119482222
434200010097041282022
43511122107419971211
43611208108819481022
4372020100129010152244
43802015001319642133
4390001100119470111
440101021066011631122
44110014105219810120
4422221810144016272211
44302004101219482022
444001010017011800111
445100010049110310011
4462002500101110140240
447210121010618821120
44802104102818812031
449200130099111800131
450001241020152060211
45111212109015171111
45212028106419642233
4532011300115012132122
45411108107208811111
455001021018013450111
45602128103219642233
4572110810120010472122
458010221010112461233
4592012610118131020231
46012204109219972031
461112110089013951211
4622121100137012292222
463021030027012120131
464001230019114280211
4650002300319640222
4662010210114011792133
467122281096112292211
468022270047020900222
469101261070110960220
4702022610134112300222
4712220500141010472222
472212070013508642233
4732112410124010472120
47412128108019322222
47510213008319810122
476022041044116262040
477100261054046081031
4780101700718811122
479100250053011791031
48010226108609141211
48100202103409470222
48211107007108811111
483122270095147592211
484021161030110812122
485112070087011631222
4862021300131017092233
48702127003119482233
48801205003709480231
489122150093011961220
49012103007508811120
491220161011018812111
49202028101618982244
4932001410100012452031
494220281011218812222
49520002109809812022
49601218104018811120
49700201003308810222
498111210073021731111
49901005005012120011
410012015006119162122
410112016106219982122
410201206103809140231
4103220150010918982111
410402215004518972131
4105212121013819811122
410601118102418821111
410701018108110471122
4108200261010219980240
41092210610126119412011
4110211230012309142120
411101222104219471211
411202027001519642244
411300012102110140111
411411017005518981111
4115210241010809642031
411602203004309640240
4117120030059118592011
411812203009118972031
4119102141084111460122
412001122102619151222
4121222061014219972022
41222102300107111631231
4123210081010409992011
41240100610618981011
4125002141036111790111
412610113006708821111
4127210110010518811120
4128202141013208982233
412911021005717821222
41302212700127010802111
413101106102209480120
4132212241014019641211
4133120041060111792011
413410225008508821211
4135002130035013280211
413610125006908811120
41370102100918811233
4138121270079013781122
413901217003918651120
414002016101419142133
41412221700143110652111
4142210070010309642011
414312116107817992111
5010026105415180231
51002021034011790222
52222181014405182211
53201010011305172133
5410201008109641233
55122281096132672211
5610225008505171211
57210121010615171120
58212241014006662211
5910226108615210211
51010202108205171233
511221061012605172111
51201117002315171111
5130101810815991122
514121030075118752020
51520002109805172022
516121041076130352020
517210070010315831011
51820013009906992031
51910114106806651111
5202112410124014612120
52111022105815171222
522021281032117422233
52311108107205171111
5240101700715171122
525122030091125212031
526020041012112972022
5270100500515171011
528101021066116600022
5292111210122010802111
53011021005715191222
53102116103016822122
53202204104415832040
533201021011405172133
5342112300123020572120
53511212109005661211
536222170014315172111
537211110012115171111
53811208108816161022
539020161014114442133
540220281011211902222
541122041092117272031
542201250011705172131
54312216109409641220
54412004106015182011
545201141011605662122
54602103002715832031
5470002300309310022
54801105002115171020
549200141010008152031
550200010097020262022
55101206103805170231
55202216104615172131
553011181024116601111
554121161078121572111
555012170039115601120
556210110010515181120
55700102101816660011
558021150029118752122
559200261010215170240
56010013005115180120
56120213001311148470133
56212015006115182122
56300123001905330111
564202010012906332244
56502215004515172131
5660102100917811233
567022281048117432222
568201130011505172122
56902227004715172222
570012221042117261211
57110214108405171222
5722022500133018592222
573212230013915171211
57410014105215990120
57500201003307650222
57612028106416162233
577210241010815171231
578212070013507822233
579110170055115111111
58012227009515172211
581020030011114112022
58211121007315171211
58302127003114382233
5842100810104021902011
58500213003505330211
586100250053018261031
58710213008308981222
5882120810136115441033
58901106102215171020
590121150077114622111
591121281080111302222
5920002410415180222
59301122102615171222
594120030059123072011
595020281016110312244
59612127007915332222
59712215009317322120
598211070011908822122
5992220500141129852022
5100111070071013291111
5101210230010705172031
5102100021050118410011
510312016106205171022
5104010221010119751233
5105220150010915172111
510602027001516162244
5107221281012816332211
5108022030043012950240
510912027006316162233
511001221004116491211
51112002500101143110240
511210001004905171011
511301218104016161120
511410126107018480220
511510101006506011122
511600214103615990111
5117101130067010631111
5118021041028111302031
5119112070087031351222
51202202700111121742222
5121202021013015180044
5122202141013208652233
5123112110089012781211
51242012610118017432131
512501121002518311222
5126222061014205992222
5127202261013406492222
51280001210218810111
51292210500125111132011
513011018105615171111
513110125006905171120
513202015001319472133
51332212700127114282211
5134012050037035480231
5135220161011017502111
5136111221074115111211
51370001100119480111
51380100610617001011
5139001010017015280111
5140212121013805172222
51412121100137013622222
51422110810120113121022
5143001241020122720211
602021410132099662233
61012170039025600220
622121100137117311122
63022150045110882131
64120161062016651022
652011300115011352122
66110221058113501222
6700011001133050111
68011221026016980122
692120810136121121033
610020150013110522133
611200130099013172031
612111221074011351111
613022270047115162222
614002141036012180211
615010221010110021233
616001230019039180111
617002130035112670111
618012210041110851211
619021270031110862233
620022041044113342040
6212220500141110022022
622212230013909692211
623001010017128090011
6242100810104112841011
625022281048112512222
626022030043126282040
62711108107219361011
628110170055110851111
629111210073028251111
630122270095014001211
6312012610118012182131
6322022610134021622222
6332221810144140502111
634020270015113672244
635101141068026761111
63610126107009701120
637011050021111351020
638120041060127592011
639121030075030901120
640100261054111350231
641002010033014830222
64202104102819032031
64300024104110020222
6442010100113013502133
64501205003719691031
64612116107819362111
64711208108818531022
64802128103219032233
6492112410124118961220
650100250053023441031
6512120700135010852233
652212121013818871122
653200010097011032022
65412215009318372120
6550101700719531122
6562101100105010362020
65701206103819861031
65810214108408541222
659102010081013001233
660122281096115332211
661221270012719202211
6622020100129010522244
66310001004909691011
664101021066011841122
6652001410100113500131
66610225008508541211
66700023003019960022
6680100500518531011
669102130083023441222
670210070010309192011
6712122410140121951211
672201250011709362131
673100141052010021020
67411107007109361111
675100021050116480011
67600012102110020111
6772110700119110191022
678200021098010032022
6792210500125111852011
6802011410116010522122
681022161046111192131
682011210025110681222
683121281080010181122
684121270079034061122
685221281012819692211
686121150077111182111
6872112300123011352120
688021161030110852122
68901021009111021233
690122030091014991231
6912102300107119141231
692120030059123612011
6932201500109011682011
694110210057011851022
695112070087111021022
696120150061012041022
6972220610142017482222
698020041012113502022
6992022500133019302222
6100020161014110522133
6101120270063110852233
6102002021034117640022
610301106102219031020
61042102410108011512031
61052202810112113342222
6106120281064024101033
6107001021018112670011
6108122041092111522031
6109101010065012671122
6110101130067013671111
6111020030011019160022
61122110810120111681022
61132202700111113662222
6114200250010108042040
6115220161011008862011
6116012221042113841211
611710202108209201233
61182002610102014662040
611901117002319531111
61202010210114013172133
61212210610126113672011
6122112121090112011111
6123102261086117800211
6124012181040114001120
6125202021013008362244
612601006106129071011
6127011181024113171111
6128100130051019141020
612901018108110851122
6130222170014309362211
613100124102019350211
613211018105619031111
6133021030027014990131
6134122161094110032120
61352021300131012672233
6136112110089134871111
61372111210122117311111
613802028101619362244
6139210121010609682020
61402111100121121781111
614110125006909531120
6142021150029114992122
6143121041076111352020
70101010065121620022
71001010017119800011
72210081010407712011
732002610102125590240
74022281048015660222
752121100137119461122
762221810144118482111
77100021050138680011
78120270063141172233
792012610118013672131
710011221026127081222
71101017007129571122
7120102100915221233
713020270015113332244
714021161030041840122
715020030011110682022
716220270011119532222
717212081013609692233
71810102106609201122
719202010012908692244
72001117002318031111
7212212700127019962111
722210070010315221011
72302016101417202133
7242221700143115992111
725012170039115821120
72612004106009531011
727001230019011180111
728212230013909692211
729020041012116322022
7302021300131012342233
73112104107609521120
7322201610110110522111
73311212109008701211
734112081088127911022
735100141052011851020
736002010033038530222
737102261086013011211
7382101210106112681120
7392112300123022782120
740021270031116982233
74101006106111681011
742112110089111511111
743010221010112351233
744102141084010021222
74502028101619532244
746020150013110352133
74711017005519191111
748120150061111512122
74900023003142660222
750200130099011192031
75111207008708861222
75200202103409850222
753121030075011511120
7542120700135013682233
7552121210138018972222
756002141036011510211
757012221042024100211
758110221058110691222
759011210025111711222
760121281080110022222
76111021005718861222
762100130051021111020
76301221004118871211
76402128103219192233
765111070071012171111
7662122410140014492211
7672020210130010022244
76810125006919520220
769022270047110682222
770210110010518371120
7712111100121110191111
7722010210114010852133
773021030027012010131
7742022500133121790222
775002130035020620211
776111081072010371111
7772110700119046482122
778122041092011021231
7792002500101122120240
780201141011609192122
781012050037110021031
78212003005909851011
7832012500117116480231
784122150093034871220
785121161078011841111
7860001210219190111
787201010011309862133
7882102410108110191231
78902215004519852131
79001218104018031120
791102130083010681222
7922112410124019962120
79310114106809371111
7942102300107128911231
795011061022123441020
796221281012819032211
797012061038010850231
798222061014209532222
799111210073116641211
710000011001110360111
7101122030091011511231
7102022161046124452131
710310225008509521211
7104022041044126092040
7105022030043119642040
7106100250053026421031
7107120161062110862122
7108101130067010691111
710920001009708702022
7110110181056112001111
7111102010081011511233
7112001021018012010111
7113121270079121302222
71140002410418030222
711501018108110021122
7116122270095024101211
711720002109808872022
7118200141010009192031
7119221061012609352111
7120001241020126580211
7121211081012009032122
712210026105409031031
71230100500519021011
712410001004909521011
7125011050021110021020
712612028106419032233
712712115007718862111
7128122281096112002211
712910202108208691233
71302022610134016312222
71312210500125014322111
7132222050014108532222
713301118102418861111
7134220281011219362222
713510126107009861120
71362011300115010682122
7137021041028110192031
7138021150029111682122
7139122161094111842120
7140220150010918862111
7141111221074010351111
71422021410132010182233
71432111210122111351111
80012210041024440211
812021410132024452233
82121030075016821120
832022610134114990222
8412115007705221111
852221700143029572211
8612004106016912011
872220500141017652222
88121281080121792222
89020161014112682133
8102010100113119640033
8112120810136017992233
812011210025112181222
8132122300139018472211
8142201610110024282011
815210121010615231120
816100261054114000231
8170001210215230111
818021281032113832233
819220150010905222011
820002130035013170211
8212020100129011352244
82212027006315232233
82301017007110191122
8242012500117125600231
82511211008905231211
826222061014206882222
82701018108011510022
82802215004505230231
8290102100915231233
83012127007916872222
83110225008508201211
83200023003112840222
83302003001105220022
834100021050032721011
83510213008309861222
83620001009709702022
8372111210122020132111
838210081010415221011
8392121100137010682222
84012015006108861022
84100011001110190111
842012181040010680220
84311107007105231111
844220270011119192222
8452011410116111850122
84610214108405221222
84710125006916870220
8482110700119013342122
849122030091012351231
85011108107217041011
8512022500133015992222
85212216109415232120
853110181056110361111
8542011300115013512122
855021270031114822233
856220281011216052222
85720013009908212031
8582001410100031722031
859201261011805232131
86001217003909530220
861002141036115000111
862001241020114160211
863110170055125431111
864002021034014670222
865100141052011191020
866200250010105222040
8672111100121111681111
8682210610126014002111
86900024104111350222
870020041012114172022
871021030027132232031
87210202108205391233
873001010017118480011
8742002610102012172040
8752121210138020632222
87602115002916712122
87702228104807040222
8782021300131013182233
879101021066014341122
880020281016021620044
8812122410140011352211
882100010049018151011
88301205003719191031
884210110010505552020
885001230019115160211
88611212109005221211
887120281064113332233
888120161062124442122
889112081088013511222
890100250053121620231
89110113006705221111
892022041044010190240
893011181024115661111
8942210500125129572011
89502203004315232040
896121161078110192111
8972112300123115331220
898011170023113511111
899200021098173640022
81002102410108110361231
810112228109615222211
8102022161046113672131
810302227004718702222
8104111210073115991211
810501022101019701233
810611022105815231222
8107112070087053091222
810801105002119381020
810912104107616222020
8110012221042111351211
8111122270095114002211
8112100130051013671020
8113221270012718862211
8114120030059021121011
8115110210057013841022
8116122150093010851220
811701005005113841011
8118101141068011681111
811901122102615221222
812012204109215232031
8121211241012418861220
8122102261086115490211
8123101010065123440022
81242120700135011852233
812510126107005221120
8126202021013004572244
81272010210114012342133
8128020150013110852133
8129210070010319361011
8130021041028011680131
8131021161030017480122
8132111221074110851211
81332102300107111191231
8134012061038021130231
8135211081012018871022
8136222181014409362211
8137001021018127920011
8138002010033012670222
81390100610616711011
81402212810128111852211
814102027001519862244
8142102010081012181233
8143011061022011520120
90201261011805222131
91110221058116321222
92021150029019470122
9311018105618871111
94020030011012180022
95210081010407542011
9122021410132018142233
913002021034029730222
914120161062116492122
9152111210122011352111
916022030043042490240
917012050037112211031
9182022500133111510222
91901018108012340022
9202220500141111522022
921111081072111351011
9222221810144010522211
923010221010128741233
9242202700111016182022
925002130035018150211
926101010065124270022
927122030091014661231
928100250053012841031
9292122410140012172211
9302212700127120632211
9312112300123114991220
93200024104112010222
933102021082010191233
9342010100113038182133
935020150013115172133
9362221700143014842211
937002141036011020211
9382202810112114662222
939102010081020951233
940020281016112682244
941111070071013331111
942022270047120792222
943200010097110350022
944112070087013661222
94512028106419852233
9462110700119110521022
947012170039010350220
948101141068010031111
949202021013008702244
950011170023111351111
951012210041020790211
952200250010118860240
953111221074115811211
954101021066019631122
955011221026116981222
956021281032029570133
957001241020116330211
958001021018112340011
9592201610110114502111
96011121007319851211
96110225008519700211
962121281080122962222
9632121210138015492222
96420002109809702022
9652120810136016982233
966100261054113000231
967110170055019311011
968012221042013330211
9692201500109011352011
97010214108406871222
9712122300139012842211
972120270063122292233
97301021009110851233
974112110089112341111
9752001410100111510131
9762011300115017482122
977020161014110192133
97800101001709370111
979121270079110352222
980100141052011021020
981122150093032061220
982122270095110522211
9832102300107110511231
984011050021111511020
985021041028110362031
98602027001519202244
987110210057110511222
9880002300319360222
989120041060110682011
990002010033011360222
99100012102112670111
9922010210114012342133
9932012500117111340231
994121030075014501120
995011210025113671222
996212110013718861122
997112081088012011222
9982100700103112341011
999122041092011691231
910001017007010020022
91012021300131014332233
9102022281048112342222
9103121161078112342111
9104021161030110352122
9105100130051110690120
91062220610142013842222
910710125006919200220
9108012061038010700231
91092020100129010362244
9110100010049013831011
91112210610126111182011
9112101261070014831120
9113011181024110851111
91142111100121029242111
9115120150061110022122
9116121041076010351120
9117100021050020801011
91180100500519201011
91192102410108113011231
9120012181040110351120
912100123001919520211
91222002610102112170240
91232101210106112351120
9124022041044110352040
9125221281012809362111
91262011410116010352122
912701106102219531020
912800011001010850011
9129101130067011021111
91302112410124010192120
913112115007705221111
9132202261013409202222
9133112121090036191211
9134200130099012182031
9135221050012509702111
91362101100105110711120
9137122281096011841211
91382120700135014662233
9139102130083013681222
9140022150045110192131
9141102261086112010211
91422110810120114501022
9143122161094114172120
100012181040114171120
101012210041111341211
102011061022119801020
103021270031114502233
104101250069011011120
1052100700103014172011
1062122410140010852211
107220150010908862011
108202021013009192244
1092111210122149121111
101001217003915221120
101100011001010690011
10120002300305230022
10131120700870113241222
10140101700719031122
1015121150077011511111
1016201250011708702131
1017102141084015151222
101812003005918402011
101912203009118862031
102001118102419021111
10212210610126022112111
1022020150013010190033
102310114106808371111
1024110170055014661011
1025210230010714731231
102602204104418372040
1027001230019099320111
1028210121010615221120
102920025001010113732040
103012103007508861120
103102215004517542131
1032201010011309032133
103312027006317872233
1034200010097012512022
103602104102808860131
1037212110013718361122
103802115002918202122
103900214103609020211
1040201021011409362133
104112204109218532031
1042202250013307872222
10430100610617871011
104410126107008691120
104510102106609521122
1046211081012008712122
104702128103217372233
104802203004317872040
104911208108817211022
105002227004717372222
1051202261013408202222
105201222104215221211
105321123001230112242120
1054011050021010520120
10552011300115039882122
105602016101418032133
105710025005319020231
105801122102618531222
105911122107408361111
106000102101815230011
106101206103818031031
1062212081013607702233
106302228104819032222
1064012050037117641031
1065200261010205222040
10662001410100011182031
10672220500141010852222
106810013005105221020
106912215009308701220
1070020281016113012244
107102004101215222022
107212116107818532111
1073220281011208202022
107412104107609191120
1075202130013108862233
107610201008108701233
1077212230013918361211
107800202103407540222
1079110210057010181022
1080021030027111182031
108102003001115222022
10820111700231114381111
1083111070071091371111
108402216104618702131
1085110181056111191111
108610026105405221031
108720002109808862022
108800213003508860211
1089221281012808862111
1090222181014408202211
10910002410418030222
109201022101017871233
109312216109418702120
109412015006118202122
109512128108008041122
1096201261011807872131
10972201610110010362011
109801018108117311122
109912227009508371211
1010010226108608031211
1010110014105218530120
1010210225008508861211
1010310213008307871222
1010411022105817531222
10105220270011117542222
1010612127007917382222
1010702027001519862244
1010811212109018541111
10109222061014208702222
10110222170014308532211
1011112028106417702233
101120100500519031011
1011310202108207861233
1011411121007317531211
1011512228109607371211
1011602116103016212122
10117210081010417531011
10118221270012717702211
10119201141011607362122
10120211241012417711220
1012111108107208531111
101222111100121112501111
10123210110010505222020
10124001010017081590111
101252102410108084252031
10126221050012508202111
10127202010012908202244
101282001300990114582031
101292021410132012512233
101300102100918371233
1013101121002519361222
1013210001004909861011
1013300201003309200222
1013410113006708861111
101352110700119010352122
1013600124102008030111
101370001210219190111
10138212070013509522233
1013910002105008691011
1014010101006508371122
1014111211008918371111
10142212121013819851122
1014312016106219042122
110200261010215230240
111001021018053420111
11200023003121120222
113022270047165202222
11401022101016381233
115122041092022281231
116020161014111022133
11710125006905221120
1182212810128112182211
1192111210122112511111
1110012061038117311031
111100012102026090011
111201018108134381122
1113102021082013501233
1114011221026129401222
11152012610118122450231
1116100010049142160011
11172210500125015852111
1118110181056110191111
1119102261086121950211
1120111221074114831211
112120013009919200131
1122222181014409032211
1123101261070071651120
1124120270063119632233
1125001010017014990111
1126012181040011020220
1127122161094115652120
11282122410140124931211
1129121161078037691111
1130200021098126920022
11312122300139020972211
113211121007309201111
11332112410124011182120
1134201010011308872133
11352021410132114660133
1136112081088012351222
11372220500141011182222
1138101130067079101111
113900202103415220022
1140012050037012670231
1141120030059160052011
1142020150013111022133
1143121270079020311122
1144122030091014001231
1145122281096131062211
11462021300131046802233
114702115002905220122
11482020210130011182244
11492120700135018302233
115002128103217202233
1151002130035148630111
11522110700119051772122
11532022500133083592222
11542102410108044162031
1155112070087027421222
1156011210025146301222
1157222061014205232222
1158102250085169820211
115901005005178441011
1160100021050022281011
116111108107209361111
116200024104111180222
116312028106415222233
11640121700391112241120
116502127003114922233
116612215009305051220
1167112110089113161111
1168020041012028090022
11690100610615221011
1170102141084014321222
1171002141036117810111
11722020100129018142244
1173022150045015320231
11742202700111118652222
117501117002306380111
1176021161030117812122
1177102010081012341233
11782121100137114341122
1179021030027011360131
1180120150061113172122
1181111070071017981111
11822111100121110681111
118310026105415220231
118410014105215220120
1185200010097052922022
11862201500109125932111
1187210230010717711231
1188112121090012181211
1189122270095116822211
1190001230019120300211
1191110210057116161222
1192020281016110692244
11932010210114010522133
1194121281080115162222
1195022161046121132131
1196021041028011360131
1197100130051111680120
11982100700103167501011
1199211081012005232122
1110001017007127411122
111012101210106154581120
111022221700143015992211
11103022030043041710240
11104121150077015321111
111052120810136015492233
111062012500117112350231
11107011061022010520120
1110800124102015230211
11109011181024015650111
111102210610126012022111
1111101105002107380120
11112200141010017530131
11113022041044013500240
11114102130083128250122
11115022281048127422222
111162101100105112841120
11117002010033014330222
111182100810104116651011
1111911022105819531222
1112001222104215221211
11121100250053111510231
1112200011001111010111
11123012210041014990211
1112410114106817210111
11125121030075014821120
111262022610134115170222
11127120161062111022122
1112802027001518042244
11129020030011032390022
11130201130011507042122
11131120041060128252011
11132110170055125111111
11133211230012305232120
111342002500101164520240
111352121210138114501122
11136101021066015491122
11137101010065048791122
11138201141011615230122
111392201610110126422111
1114012104107605231120
111412212700127112832211
1114201021009113161233
111432202810112114662222
1202122300139123621211
12100101001705220111
12201106102205900120
12312203009105221231
12411017005516211111
125201250011705232131
126112081088013841222
12700124102016220211
128200250010116710240
12920001009709542022
121002028101619702244
12110100610615221011
1212210121010619521120
1213121161078125762111
121411018105619861111
1215201021011408532133
1216102141084037031222
121710114106819530111
121811108107208881111
1219210081010409532011
122012003005908871011
1221002141036012350211
122200024104112020222
122301018108014330022
12242111100121034722111
1225021161030011180122
1226111210073010031111
122710013005119030120
122801217003919361120
122912015006119362122
12300001100119200111
1231222050014108872222
123202203004309060240
123301005005010030011
123400201003319360022
1235222170014319362111
1236020030011122112022
12372202700111118652222
123812115007718702111
123910001004909531011
124010201008109361233
1241220150010918712111
1242101010065012011122
124302027001518372244
1244102130083111020122
1245210070010308552011
1246011210025118811222
1247211081012008702122
1248221050012509362111
124902004101208700022
125011122107408371111
125100213003519520111
125212028106418542233
1253010221010114661233
12540101700718371122
125512227009508701211
125610113006719200111
125712016106219032122
1258200141010019360131
1259221270012709062111
126012216109418702120
1261201130011519530122
126212215009318212120
1263120041060118482011
1264012210041111021211
1265211070011908202122
12662122410140117481211
1267201141011608042122
126810226108609031211
1269202250013318210222
1270212070013508542233
1271210110010518531120
127210225008508041211
1273202141013209202233
12742201610110110852111
1275202010012909702244
1276121281080015001122
1277100250053010351031
1278200130099010862031
127902127003108030133
1280112070087010031222
128110102106608701122
1282222181014418372111
128311211008919201111
128401222104208040211
128502015001318202133
128602228104819052222
128711022105818201222
128802128103208370133
128910014105218370120
1290200261010218370240
12910002300318700222
129212104107608031120
1293202261013408532222
129411021005719531222
1295222061014207872222
129601205003709360231
129712027006318202233
1298202130013108042233
129901206103808030231
12100221281012808202111
121012121210138113671122
12102220281011218372222
1210302016101418872133
1210420002109808532022
1210512103007508861120
1210602103002709530131
1210710202108208031233
12108122041092010191231
12109201010011307882133
1211001118102418371111
1211102216104617872131
121120001210218530111
1211300202103408700222
1211412127007908041122
1211501122102608030122
1211601218104018031120
1211702104102807870131
1211810126107008531120
12119201261011808532131
12120210241010818701231
1212102115002918042122
12122100021050012181011
12123211230012318041220
12124202021013008042244
12125212081013608212233
1212600123001917870211
12127211121012208372111
1212802204104408700240
1212912228109608371211
121302210610126017482111
1213111212109017871111
12132011050021011850120
1213302215004518362131
12134212110013718371122
1213511107007108201111
1213610125006908371120
12137211241012408702120
1213800102101807540111
1213902227004718542222
1214001117002318211111
121412102300107027752031
1214201021009111681233
1214310026105418040231
13010126107011920220
131110210057125931222
132020270015012170044
133111081072111681011
1342212810128111032211
135202250013315220222
136122270095012831211
13701222104217381211
138022281048158072222
13901221004101910211
131002015001305220033
1311112081088112841022
131210025005305221031
13132122300139110851211
13142202810112017152022
13150121700390108270220
131612028106405231033
1317010050051114561011
13180220300430111980240
131902116103008030122
13202221810144111842111
1321220161011005232011
132202227004715222222
132311211008904891211
132400214103605220211
1325122161094112352120
1326200250010105232040
1327121161078011021111
132801118102415221111
1329012181040119301120
1330202261013411920222
1331202010012915230044
1332122041092111352031
1333202130013105222233
1334210081010409362011
133500123001916050211
13362110810120134541022
1337210121010609692020
1338211110012105222111
1339100021050049461011
134001106102215221020
13412120700135020132233
1342120150061012011022
1343021281032111682233
134412016106215232122
13452011300115110680122
1346220150010905222011
134712003005915222011
13480101810805220022
1349022041044116982040
1350120041060133222011
135101205003719191031
1352102141084012171222
135310226108605231211
135402004101206380022
1355111221074113501211
13562100700103120791011
135711207008705231222
13582212700127011522111
135901105002118361020
136002028101609850044
136112027006305221033
136201117002315221111
136300024104110850222
1364100010049012511011
13652220500141111682022
136601122102618701222
1367200261010206712040
136800012102016650011
1369101141068010031111
1370102130083011521222
1371100261054134710231
13722102410108110521231
13732112300123118651220
137401021009028410033
1375200010097122110022
13762121210138115991122
13772111210122119141111
137801017007131061122
137920002109815220022
1380101021066134060022
1381110221058113171222
1382200130099021132031
13832210500125121292011
13842010210114110190033
13852001410100158390131
138602216104615222131
138900102101815220011
1390121270079110852222
1391122150093111182120
1392101130067011191111
1393011210025016820122
13942122410140110371211
139502103002705220131
139612103007505261120
13972102300107115321231
139802003001105230022
139910014105209861020
1310000202103418360022
131012220610142115822022
13102102021082110680033
131032012610118113380231
1310401006106116651011
13105201141011619690122
13106202141013209362233
13107102010081117160033
1310801206103818361031
13109002010033112170022
13110001010017113670011
13111101250069011181120
1311211017005505231011
13113001241020111510211
13114121150077021621111
13115110181056110521111
131162010100113124440033
131172101100105111181120
13118121041076110202020
1311911212109017701111
13120002130035012680211
131212121100137025592222
13122020161014110062133
1312312203009105221231
131242221700143013832211
13125211241012416881220
1312600023003110530222
1312710225008509861211
131282120810136111181033
13129021150029010190122
1313010101006509711122
1313111121007308371111
1313202215004508200231
13133201250011719030231
131342020210130010352244
131352110700119011352122
13136021270031110032233
13137220270011118202222
1313801022101018531233
1313912128108019522222
13140221061012619032011
13141122281096110022211
1314202104102818692031
1314311107007119191011
140110170055013011011
141110221058111351222
1422011410116012682122
14312204109205391231
144012061038011350231
145121161078015491111
146120161062111012122
147022041044011350240
148211241012419521220
14912127007915912222
1410011170023110701111
141110013005118690120
1412222050014108532222
14130001100109520011
1414102250085111180211
1415001010017112360011
141610101006515220022
1417202261013419530222
141802016101415222133
1419100250053110680231
142020001009717700022
1421122281096010031211
142211212109008701211
142310002105009201011
1424211070011905712122
14252100810104012012011
1426012210041112011211
1427212110013708702222
1428111210073110691211
142902004101209030022
14302001410100010352031
143101021009112851233
1432021270031110182233
1433022161046010030231
14340001210209850011
1435201130011515550122
1436010221010120621233
1437101130067014331111
14382120810136010522233
14392002500101110200240
144011021005708201022
14412111100121010022111
1442020270015112012244
14432122410140110201211
144410102106618870022
1445210070010308532011
1446102261086011351211
144700213003509030211
144800214103605230211
1449021150029022280122
145010126107019390220
14512022500133113170222
1452020030011111342022
145302103002718532031
1454112070087010691222
14552220610142011352222
145612028106419032233
145700123001919350211
145800201003319690022
1459111221074123611211
146001217003905220220
146101121002519361222
146210014105219190120
1463022150045010690231
146412104107618532020
1465011061022014650120
146600124102018860211
146700202103408860222
14682102410108110521231
14692110810120111191022
1470220161011009692011
14710002410408700022
1472201021011416880033
14730101700709200022
14742021300131012182233
1475001021018014990111
147610201008108701233
1477122270095112012211
147801005005110851011
1479112081088110521022
1480221281012818042211
1481202010012919040044
1482200261010209692040
1483022030043010190240
148412103007518702020
1485021041028110692031
1486201010011319860033
1487200021098110190022
14882122300139111181211
148910026105409201031
149012015006108701022
149110202108208701233
149210214108409031222
1493212121013808362222
1494012221042015660211
149512115007708371111
149601122102617371222
149712027006318372233
1498210121010609692020
149901006106010690011
14100222181014418862111
1410112203009108701231
1410212128108019532222
1410302028101618532244
1410410125006918700220
14105211230012319361220
14106020150013010190033
1410701218104008370220
141080002300315220222
1410901105002108860120
1411010114106818200111
14111210110010509212020
141120101810809030022
1411302128103218862233
14114202021013009202244
1411512003005918862011
1411610213008309361222
14117221270012718042211
1411802228104819362222
1411910001004919030011
14120220281011215222222
141212012610118110180231
1412202227004718362222
141232021410132112010133
1412411107007109031111
14125222170014308362211
14126210230010719521231
1412711211008908361211
1412812216109409031220
14129212070013508532233
1413001205003709200231
1413112004106008701011
1413211018105608371011
14133220270011118702222
14134221050012518862011
141352201500109010362011
1413601118102408870111
141372012500117110020231
14138211121012208702111
14139221061012619362011
1414011108107219031011
1414120013009908702031
1414212215009308201220
1414302116103016372122
1502001410100156250131
151200021098015992022
152222170014305222211
1532212810128193522211
154121281080110352222
1552020100129012182244
156012050037017480231
15702015001305230033
158122161094022281220
159102021082029571233
151002128103215232233
15112121100137131061122
1512011061022132221020
151301018108015820022
151420001009705232022
1515010221010129231233
151611121007315221211
15172002500101047792040
151800213003505220211
1519201021011409362133
152010226108606871211
152110125006905221120
1522022161046113002131
1523112081088010521222
152402104102817042031
1525022270047116652222
1526220281011217862222
15272210610126117642011
152801118102418701111
152912027006315882233
15302012610118017812131
153102204104409880240
1532202130013108362233
1533110181056116151111
1534100141052022611020
15350100500518041011
153601105002117871020
153702016101418702133
1538120030059011841011
15392210500125116322011
1540122030091012521231
154110214108408041222
154200102101809690111
154312028106418532233
15442101210106019302020
1545011210025121291222
1546220270011118032222
1547022030043112352040
15482201500109011852011
154920013009907872031
155002028101618362244
1551111070071010521111
1552221270012718532211
155301122102618041222
15542002610102114160240
15552220500141142992022
15562022610134011522222
1557002141036010190211
155811207008708201222
155911021005718861222
15602102300107110521231
156102004101218372022
1562021270031116652233
15630100610618041011
156401206103819351031
156510102106608531122
156602115002918712122
1567201141011609692122
15682111210122010052111
1569122150093012681220
15702100700103012192011
1571210241010815231231
157210114106808861111
157300124102019860211
157401221004118531211
1575121270079110522222
157610002105008371011
1577110221058117311222
1578122041092012671231
1579121030075110362020
15802120810136011182233
158102027001519032244
1582001230019113830211
158312228109609861211
158410025005319190231
15850001100108370011
158612227009518692211
15872120700135016492233
1588211070011908702122
15892101100105024772020
159001017007119801122
1591211241012418701220
1592202021013009532244
159310213008309531222
1594101261070010851120
159511108107208701111
1596022150045110362131
159710201008108861233
159810225008508861211
1599110170055036531011
15100202250013308202222
1510102228104818532222
1510200101001717870011
15103202141013207702233
1510402116103016212122
15105201130011507712122
151060002300319870222
15107212230013918201211
15108212241014008372211
1510912004106018702011
1511012016106206871022
151112201610110010362011
15112112110089010191211
1511301021009126101233
1511411212109009521211
15115222181014408212211
1511601117002318861111
15117212121013808532222
1511810113006709031111
151192112300123132401220
1512001218104019861120
1512112015006108201022
1512201217003917871120
15123100261054040501031
15124121041076123272020
15125211110012108542111
1512600012102034210011
15127100010049014341011
15128020030011113342022
15129222061014205232222
15130002010033126420022
15131211081012015231022
1513202103002719532031
151332010100113018132133
1513401222104216381211
151352100810104018642011
1513600024104111010222
15137002021034013840222
15138111221074111021211
151392012500117012192131
1514012116107815222111
15141100130051011351020
1514210101006509031122
15143121150077113842111