One Measure of Happiness

One Measure of Happiness

An interactive Video Installation

Content
Content
Objectives
Concept

1. Installation
2. The Video
3. System Architecture
4. Setup
5. Project state (January 2003)
6. Who we are

Detailed System Architecture

1. Data flow and timing

Figure 1: General Data Flow and Timing

2. High level Roles & Responsibilities

2.1 Sensation TBD
2.2 Perception TBD
2.3 Cognition
2.4 Emotion
2.5 Sprite Manager
Table 1: Sprite Manager Selectors
Table 2: Sprite Manager Events

3. Detailed Roles & Responsibilities

3.1 Sensation Details TBD
3.2 Perception Details TBD
3.3 Cognition Details
3.4 Emotion Details TBD
3.5 Sprite Manager Details
3.5.1 Scene Selection
Table 3 - Story Skeleton Phases
Table 4 - Story Scene representation
Table 5 - Sequence representation
3.5.2 History
Table 6 - History
Table 7 – Memory Clips

Objectives

1. To tell a coherent story interactively, fluently and cinematically through the interplay of three actors: a database-driven story model, an autonomous synthetic character, and an involved and active interactor.

2. To create a compelling experience of intimacy between an interactor and a synthetic video character using familiar gestures through an affordable touch-screen interface.

Concept

1. Installation

The installation is situated in a physical four-wall compartment. Interactors enter the compartment one at a time through a path of obstacles. Each obstacle involves a prohibition that must be transgressed in order to proceed. At the end of the path, a touch screen is embedded into a wall, with a prohibitive “Don’t Touch!” sign above it.

2. The Video

The Video is composed of two composite layers: Foreground and Background. In the foreground layer we see a Close-Up shot of the protagonist, a young woman whose, face we can touch. The background layer is a back-projection of the protagonist’s thoughts, memories and feelings.

The protagonist invites us to touch her face. The way we touch, the location, manner and duration determines the level of intimacy and confidence we inspire in the protagonist. This, in turn, affects the way we progress through the story.

3. System Architecture

The foreground and background layers are generated in real time by Macromedia Director, from an external clip database. The clips are assembled on the fly according to the program algorithm, which is comprised of three elements: A narrative and presentation manager, which is in charge of maintaining narrative progression, coherence and presentation; the protagonist, with her autonomous goals, emotions and behaviour; and an interaction manager, which monitors and interprets the interactor’s activity.

4. Setup

1 Mac G4 computer with 512 RAM, at least 10 Gigabytes of free Hard-disk space

External speakers

MAC OS 9.2 or higher

QuickTime 5.0 software

Monitors: 21 inch; Touch Screen; RGB input

Resolution: DV Pal 720 X 360

5. Project State (January 2003)

Software:
We are progressing in a modular mode. We’ve already completed the overall system architecture and algorithms and are now writing the first modules of the Sprite Manager in Macromedia Director Lingo. Cuncurrently, we are writing the pseudo-code of other modules, to be executed later.

Creative:
We have completed a synopsis and are now in an advanced stage of adapting it into a 30-page treatment. We expect to be casting by March 2003, in production by April 2003 and in a gallery near you by May 2003.

6. Who we are

Investigators, producers and creative directors:
Noam Knoller & Udi Ben-Arie

Research Supervisor:
Amnon Dekel

Story and screenplay:
Galit Reuchman and Nelle Schaeffer

Architecture Engineer:
Maya Lotan

Software Manager:
Mirit Tal

Lingo Team Manager:
Gal Tushia

Lingo Team:
Udi Ben Arie & Noam Knoller

Detailed System Architecture

1. Data flow and timing

Figure 1: General Data Flow and Timing

2. High level Roles & Responsibilities

2.1 Sensation TBD

2.2 Perception TBD

2.3 Cognition

Evaluate and react to a gesture.

Evaluation - set the Reaction (Tag, Type, Emotion) according to Gesture (Type and Area) from the Perception, Attitude (Level) from the Emotion and Clip (Map) from the Sprite Manager.

Reaction – Send the Reaction to the Sprite Manager. The timing of the reaction is conducted according to the Reaction (Type):

Regular– update the Mood (Level) via Emotion (send Reaction (Emotion) to Emotion and receive Mood (Level)) and send the updated Mood (Level) (x) along with Reaction to the Sprite Manager in the same step (x)

Extreme – Send the Reaction directly to the Sprite Manager with the proper (x-1) Mood (Level) and then send the Reaction (Emotion) to Emotion

2.4 Emotion

Evaluates the user interactivity level, maintains the mood of the character, its attitude towards the user and the level of intimacy between them.

Maintain the Mood - set the Mood (Level):

Every X tics according to an internal process, decaying Mood (Level) over time, aiming at a balanced mood. The internal process decays to 0

According to Reaction (Level) from the Cognition

According to endMood (Level) from Sprite Manager

Maintain the Attitude - set the Attitude (Level):

Every Y tics (more slowly than the mood) according to an internal process decaying Attitude (Level) over time, aiming at a balanced attitude.

According to Reaction (Level) from the Cognition

Maintain the Intimacy level – Set the Intimacy (Level)

Every Z tics according to an internal process

According to Knowledge (Level) from Sprite Manager

According to attitude level (absolute value)

Maintain the Interaction State – Regular/Brooding

Every T tics according to an internal process

According to LastTouched of the Sensation comparing the time period the user hasn't touched the screen to a threshold. When there is no interaction for a long time (TBD) – Brooding

2.5 Sprite Manager

All selectors are responsible for collecting all necessary data for their selection process.

Select Memory and Character Speech Scene - select a scene authentic to character mood, the story skeleton and the intimacy between the user and the character. A scene is composed of a memory sprite scene with or without a character sprite speech scene.

Every end of scene

(Interrupt) - When triggered by an extreme Reaction (Type) from the Cognition

Select Mood Clip – select a clip authentic to the character mood only if no character sprite speech scene is played.

Every end of clip

Select Reaction Clip – select a clip reflecting physiological responses along with acts of the user according to Reaction (Tag) Gesture (Area, Type). Only if user is active.

(React) - When triggered by any Reaction (Type) from the Cognition

Play sprites – play the scenes and clips in the relevant sprite:

The Memory Sprite (BG) – play Memory scene

The Character Sprite (FG) – play Speech scene or play Mood clip

The Reaction Sprite UFG – play reaction clip

Update Cognition of current settings.

Send the current clip map to Cognition

Update Emotion with each scene’s end mood and K(knowledge)

Table 1: Sprite Manager Selectors

Memory Sprite	Character Sprite	Reaction Sprite
BG	FG	UFG
Select Memory & Speech Scene

		Select Reaction Clip

		Select Reaction Clip

Select Memory Scene	SelecMood Clip	Select Reaction Clip
	Select Mood Clip
	Select Mood Clip
	Select Mood Clip	Select Reaction Clip
	Select Mood Clip
	Select Mood Clip

Table 2: Sprite Manager Events

	Memory Sprite	Character Sprite	Reaction Sprite	User Activity
	BG	FG	UFG	User Activity
a scene with out a FG	Memory Scene	Mood clips	-	User Activity = No (from Emotion)
	Memory Scene		Reaction Clip Regular Opacity<100%	Reaction (Type) = Regular (from Cognition)
	Interrupt		Reaction Clip: Extreme Opacity=100%	Reaction (Type) = Extreme (from Cognition)
a scene with a FG	Memory Scene	Speech Scene	-	User Activity = No (from Emotion)
			Reaction Clip Regular Positive Opacity<100%	Reaction (Type) = Regular, Reaction (Tag) = Positive (from Cognition)
			Reaction Clip Harassed reaction (u r bugging me!) Opacity=100%	Reaction (Type) = Regular, Reaction (Tag) = Negative/Impertinent (from Cognition)
	Interrupt	Interrupt	Reaction Clip Extreme reaction Opacity=100%	Reaction (Type) = Extreme (from Cognition)

3. Detailed Roles & Responsibilities

3.1 Sensation Details TBD

3.2 Perception Details TBD

3.3 Cognition Details
TBD reaction Connotation is the GestureArea + neg/pos

3.4 Emotion Details TBD

3.5 Sprite Manager Details

3.5.1 Scene Selection

Scenes are selected according to all the below, each weighted – the weight of each aspect can be changed by us.

Authentic to the character – The scene with highest authentic level derived from connotation to user gestures and the Association to character memories. TBD Might be a weight between different aspects listed below.

Connotation – A scene that has a connotation that fits the Reaction (Tag) and the Gesture (Area).
(Reaction (Tag) à XXX Area Connotation)

Association – A scene that has association that fits the association of the last scene. TBD representation and process.

Start Mood - The scene Start Mood fits the Mood (Level)

Authentic to the story skeleton – according to the story phase.

Authentic to the Intimacy between the user and the character

Depth – The scene depth fits the I (intimacy Level). High depth scenes are presented only when high intimacy is achieved.

A drop in intimacy, reflected in low user activity, triggers Brooding Mood.

Table 3 - Story Skeleton Phases

Story Phase	Description	In	out	K
Introduction	Single Intro scene to the system played in loop.	user enters system	user touches start.	K1
Exposition	Several scenes exposing 4 aspects of the character and the story in no specific order. Selected out of several scenes, each covering at least one aspect.	end of Introduction	all aspects are covered exactly once or eject	K5
Plan	The plan scene of the story. Selected out of several scenes.	end of Exposition	scene ends or eject.	K6
Body	Several interlaced sequences. Each sequences is composed of a procession of stages (crisis, local peak etc.) Each stage is represented by a selected scene. Several sequences may be played in parallel. There is an incentive to close open sequences for the sake of coherence.	end of Plan	eject	K?
Peak & Relaxation	The peak and relaxation sequence of the story and interaction. The scene is selected according to the K level and preceding scenes played, the mood and the Attitude Level towards the user.	Eject

Sprite watches
K – User knowledge of the character (story information) – not each scene contributes to the K level (e.g. scenes that open a scene-loop – contribute nothing, while closing scenes contribute the whole-lot).
I - Intimacy (Level)

Eject – triggered when either:

The K sprite watch reaches a threshold

The Character’s Intimacy level drops below a certain threshold

Table 4 - Story Scene representation

Property	Description	Value	example
ID		number
Phase	story phase of the scene	phases described above
Depth	How much intimate knowledge about the character does this scene provide?	number
Precedent	Specifies which other scenes must be presented before the scene can be presented (only for peak scene)	list of number (scene ID)	12
StartMood(Level)		number (same as Mood(Level) of Emotion)	2
EndMood(Level)		number (same as Mood(Level) of Emotion)	-2
XXX Area Connotation (XXX = face area from clip map)	The connotations of this area of the face in the scene	connotation	positive negative, neutral (or null)
XXX Area Connotation Score		weight number (1-?)	2
Association	A list of the association of a scene with other scenes. Each has a weight.	list of associations (keyword, weight) weight number (1-?)	mother, 4

Table 5 - Sequence representation

Property	Description	Value	example
ID		number
Sequence skeleton	The stages within a sequence	List of strings
Sequence Order	the order of the stages within the equence	List of strings	2
Sequence stage scenes	The possible scenes in every stage	List of numbers

A sequence is selected indirectly when its 1^st scene is selected.
A sequence may be played in more than one order of stages, but only once (except for Brooding Mood scenes).
A stage within a specific order of a scene loop is filled by a specific scene, selected according to mood.

3.5.2. History

History maintains the order and frequency of the scenes, and monitors the incentives to close sequences.

Table 6 - History

scene.played	The number of times a scene has been played through	Variable
scene.repeat	the number of times a scene may be repeated.	Variable
Last scene	The last scene played	Number (of scene.ID)
open sequences	which sequences’ 1^st scene or more has been played	List of Numbers (of sequence.ids)
Open sequence last scene	Which scene was last played in a sequence	Number (of scene.ID)
Open sequence incentive	How many scenes have been played since open sequence last scene	Number

General clip – composing the scenes (not reaction clips) - remember clip for more than one scene TBD

Table 7 – Memory Clips

Property	Description	Value	example
ID
Scene