Transformed social interaction

Transformed Social Interaction (TSI) is a research paradigm and theoretical framework related to social interaction in virtual environments.

Overview of TSI

Virtual reality allows us to break normal rules of physical interaction because users do not actually have to share the same "reality". For example, in a Collaborative Virtual Environment (CVE), a presenter can program their digital avatar to maintain eye contact with every person in the audience at the same time. Because each member of the audience has their own view of the world, they would each think that the presenter was indeed looking at them all the time even though there are in fact many different versions of "reality" co-occurring at the same time. Three categories of TSI have been identified.

Self-Representation TSIs

Dramatic and subtle changes to appearances or behaviors can be made to our avatars for social advantage. For example, a digital avatar could incorporate 20% or 40% of someone else's face. Studies have shown that both behavioral and visual mimicry can make a person more persuasive.

Changes in digital self-representation can also be used to modify a person's own attitudes and behaviors. For example, users in tall avatars become more aggressive in a negotiation task than users in short avatars. And users placed in avatars of an elderly person held fewer negative stereotypes of the elderly in general.

Social Sensory Abilities

A virtual environment can provide information to a person that would be considered super-human powers in real life. For example, virtual classrooms could provide teachers with updated interest/boredom level of students by tracking their eye gaze patterns. Or it could alert teachers when they have not maintained eye contact with certain students in the virtual classroom for a long period of time.

Environmental Transformations

Because every user sees their own version of the virtual space, different users could be in different or exactly the same spatial setting. For example, in a virtual classroom, every student could be sitting right in front of the teacher. And because a digital environment can store information, students could also "rewind" to hear part of the material again.

Examples of Specific TSI Applications

Facial Similarity

Human beings are biologically driven to prefer faces similar to them. Naturally, facial cues convey more than a person's gender, race, or age; they also evoke strong affective responses. Over the years, researchers have found that similarity between two people instills altruism and trust. Biological explanations for this effect argue that phenotype matching (implicit recognition of subtle physical cues) is a mechanism organisms use to identify genetically-related kin. Indeed, different areas of the brain process facial images morphed with the self than images morphed with familiar others. Social explanations argue that people use physical similarity as a proxy for compatible interests and values.

In the context of political campaigns, a candidate's face could, by itself, influence voters' impressions of the candidate, especially in situations when substantive information is not available. Simply put, voters may prefer candidates whose faces resemble their own. It is inevitable that political candidates, advertisers, educators, and others who seek social influence will resort to methods of dynamically transforming appearance. This is especially true in most state and local elections where voters possess very little information about the candidates on the ballot. In such 'low-information' races, voters will resort to visual affective cues as the dominant basis for electoral choice.

A study carried out by the Virtual Human Interaction lab at Stanford University demonstrated that the outcome of the 2004 Presidential election could be manipulated by digitally altering the pictures of Kerry and Bush. Other studies have also proven that this preference holds true with other lesser known, generic faces as well. Results indicated that in a low-information context, a candidate could increase electoral support by as much as 20 percentage points simply by incorporating elements of individual voters' faces into his or her campaign photograph.

To test the effect of facial identity capture on vote choice, digital photographs of a national random sample of voting aged citizens were passively acquired. One week before the 2004 presidential election, participants in the study completed a survey of their attitudes concerning George Bush and John Kerry while viewing photographs of both candidates side by side (See Figure 1). For the morphing procedure, a software application called Magic Morph was used to digitally blend two images. A random one-third of the subjects had their own faces morphed with Kerry while unfamiliar faces were morphed with Bush. For a different one-third, their own faces were morphed with Bush while unfamiliar faces were morphed with Kerry. The remaining one-third of the sample viewed un-morphed pictures of the candidates.

File:Vhil brevia.jpeg Figure 1: Two subjects, (Panels A and B), the morph of Subject 1 and Bush (Panel C), the morph of Subject 2 and Kerry (Panel D), and the vote intention score by condition (Panel E). The difference in vote intention for Bush and Kerry by condition was statistically significant (p < .05).

Interviews with the participants of these studies demonstrated that not a single person had detected that his or her image had been morphed into the photograph of the candidate. Participants were more likely to vote for the candidate morphed with their own face than the candidate morphed with an unfamiliar face. The effects of facial identity capture on candidate support were concentrated among weak partisans and independents; for ‘card carrying’ members of the Democratic and Republican parties, the manipulation made little difference.

The use of facial identity capture was sufficient to change the outcome of the presidential election by a double-digit margin, according to a national random sample. In the case of presidential elections, it is well documented in earlier research that the candidates’ party affiliation, positions on major issues, personal traits, as well as the state of the economy affect vote choice. Results from more recent studies demonstrate that implicit facial similarity should be added to this list.

Given the widespread availability of digital photographic representations of candidates in the media, it will only be a matter of time before political candidates begin to take advantage of TSI. TSI makes it possible for a candidate to modify his or her appearance in order to achieve optimal levels of social influence. For example, in political advertisements or press conferences executed simultaneously in ethnically different districts, candidates can take on appearances that closely resemble the ethnicity-base of specific districts by applying TSI filters to the video feeds of his/her images in real-time. By shedding a new light on the importance of visual cues, the techniques and theories of TSI suggests innovative paradigms for modern day political games.

Non-Zero Sum Gaze

Non-zero-sum gaze (NSZG) is directing mutual gaze at more than a single interactant in a CVE at once. Previous research has demonstrated that eye gaze is an extremely powerful tool for communicators seeking to garner attention, be persuasive and instruct. People who use mutual gaze increase their ability to engage an audience as well as to accomplish a number of conversational goals. In face-to-face interaction, gaze is zero-sum. In other words, if Person A looks directly at Person B for 65 percent of the time, it is not possible for Person A to look directly at Person C for more than 35 percent of the time. However, interaction among avatars in CVEs is not bound by this constraint. The virtual environment as well as the other avatars in the CVE is individually rendered for each interactant locally. As a result, Person A can have his avatar rendered differently for each other interactant, and appear to maintain mutual gaze with both B and C for a majority of the conversation. Three separate projects (Bailenson, Beall, Blascovich, Loomis, & Turk, 2004; Beall, Bailenson, Loomis, Blascovich & Rex, 2003] have utilized a paradigm in which a single presenter read a passage to two listeners inside a CVE. All three interactants were of the same gender, wore stereoscopic, head-mounted displays, and had their head movements and mouth movements tracked and rendered, and the presenter’s avatar either looked directly at each of the other two speakers simultaneously for 100 percent of the time (augmented gaze) or utilized normal, zero-sum gaze. Results across those three studies have demonstrated three important findings: 1) participants never detected that the augmented gaze was not in fact backed by real gaze, 2) participants returned gaze to the presenter more often in the augmented condition than in the normal condition, and 3) participants (females to a greater extant than males) were more persuaded by a presenter implementing augmented gaze than a presenter implementing normal gaze.

Academic Articles about Transformed Social Interaction

Bailenson, J.N., Yee, N., Patel, K., & Beall, A.C. (2007, in press). Detecting Digital Chameleons. Computers in Human Behavior.

Yee, N. & Bailenson, J.N. (2007, in press). The Proteus Effect: Self Transformations in Virtual Reality. Human Communication Research.

Yee, N., & Bailenson, J.N. (2006). Walk A Mile in Digital Shoes: The Impact of Embodied Perspective-Taking on The Reduction of Negative Stereotyping in Immersive Virtual Environments. Proceedings of PRESENCE 2006: The 9th Annual International Workshop on Presence. August 24 – 26, Cleveland , Ohio , USA

Bailenson, J. N. (2006). Transformed Social Interaction in Collaborative Virtual Environments. In Messaris, P. and Humphreys, L. (Ed.) Digital Media: Transformations in Human Communication. 255-264. New York: Peter Lang.

Bailenson, J.N., Yee, N., Blascovich, J., & Guadagno, R.E. (2006, in press). Transformed Social Interaction in Mediated Interpersonal Communication. In Konijn, E., Tanis, M., Utz, S. & Linden, A. (Eds.), Mediated Interpersonal Communication, Lawrence Erlbaum Associates.

Bailenson, J.N., Garland , P., Iyengar, S., & Yee, N. (2006). Transformed Facial Similarity as a Political Cue: A Preliminary Investigation. Political Psychology, 27, 373-386.

Bailenson, J.N. & Beall, A.C. (2006). Transformed Social Interaction: Exploring the Digital Plasticity of Avatars. In Schroeder, R. & Axelsson, A.'s (Eds.), Avatars at Work and Play: Collaboration and Interaction in Shared Virtual Environments, Springer-Verlag, 1-16.

Bailenson, J.N., Beall., A.C., Blascovich, J., Loomis, J., & Turk, M. (2005). Transformed Social Interaction, Augmented Gaze, and Social Influence in Immersive Virtual Environments. Human Communication Research, 31, 511-537.

Bailenson, J. N. & Yee, N. (2005). Digital Chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological Science, 16, 814-819. (see video)

Bailenson, J.N., Beall, A.C., Loomis, J., Blascovich, J., & Turk, M. (2004). Transformed Social Interaction: Decoupling Representation from Behavior and Form in Collaborative Virtual Environments. PRESENCE: Teleoperators and Virtual Environments, 13(4), 428-441.

Beall, A.C., Bailenson, J.N., Loomis, J., Blascovich, J., & Rex, C. (2003). Non-Zero-Sum Mutual Gaze in Collaborative Virtual Environments. Proceedings of HCI International, 2003, Crete.

Bailenson, J.N., Beall, A.C., Blascovich, J., Weisbuch, M., & Raimmundo, R. (2001). Intelligent Agents Who Wear Your Face: Users' Reactions to the Virtual Self. Lecture Notes in Artificial Intelligence, 2190, 86-99.

---