Space, Time and Play in Multimedia Environments

Heide Hagebölling-Eisenbeis
Raum, Zeit und Spiel in multi medialen Environments
(Space, time and play in multimedia environments)
In: Kilger, Gerhard ed.
Raumerfahrung und Erlebnispark / Experiencing Space and Adventure Parks
Ed. Klartext, Essen 2007

  Time and space are two peculiar phenomena: they form the foundations for childhood memories and adventurous dreams of the future, are the measure of history and evolution, basic data for exercising power and wars, stages for our aspirations and poetry. It is—regarded objectively—the construct made up of time and place that makes orientation possible in the first place. Yet beyond that, the actual relevant core of their concurrence is apparent in the interactions between time and space: they create the prerequisites for human perception and recognition, they are the parameters that define the field of our conditions of perception.

  This also goes for the core of every creative engagement with time-based and space-related works. Space and time—as abstract as they may at first appear—are the material from which artistic works emerge, at least they are factors that clearly mark the creative work, and this goes—alongside space, moving image, stage and event oriented works—equally for the dramaturgy, staging and scenography of exhibitions in the museal context and beyond. Media works in particular cannot escape from a reflection on, and a conscious relationship to, spatial-temporal constellations. Electrification and digitalisation have not only changed traditional perceptions fundamentally, but also the everyday presence of space and time. Poly-dimensional constructs, cybernetic cosmoses, which redefine the experience of space and time, appear behind the Euclidean concept. The twenty-first-century cybernaut navigates through intelligent environments.

  Three points in particular should be emphasised in relation to creative work—also exhibition work—with electronic and digital media:
  1. Space and time as parameters of human perception;
  2. Temporal-spatial constellations in hybrid and multimedia scenographies;
  3. The playful-emotional integration of the viewer/participant in environments based on time and space.


Space and Time
Parameters of human perception
  Time is, in itself, not determined by a beginning or an end. Time becomes manifest in processes that lay claim to a spatial expansion and is perceived linearly, therefore directionally. All perception processes are based on time. In this a differentiation is made between objective and subjective time. Thus Isaac Newton described “absolute, true, and mathematical time” as a phenomenon that, due to its own nature and from and of itself, flows evenly without relation to anything external and then sets this objective time against subjectively experienced time. “Relative, apparent, and common time,” according to Newton, “is some sensible and external (whether accurate or unequable) measure of duration by the means of motion.”

  That subjective time flows continuously—if not necessarily always evenly—corresponds with our immediate experience. However, we also perceive processes, which are divided or segmented, as continuous. For example, life or—barely consciously—the paradox of seeing, in which action is experienced as a continuum, but is, in fact—due to the saccadic movements of the eyes—erratically perceived fragments that are first joined together to form a whole by the brain. Seeing and hearing are not comparable to scanning, which systematically transcribes an object, one to one, in a predetermined rhythm and a fixed resolution. Visual as well as audial reception requires anchors or characteristics, which transport a phenomenon into consciousness and thus allow them to contribute to cognition.

  Richard Feynman’s amusing statement that time is when something happens that doesn’t happen otherwise is aimed at these characteristics. One could also speak of the deviation from the familiar: the caesura, the invasion, the exception, the unexpected, breaking away from the beat are all markers of temporal perception and anchors of cognition. Time is perceived through rhythms or their deviation. “Duration” is only perceptible through the rhythm. Thus the meaning of rhythm in Greek was not only that of flowing, but also of a structured movement, for example that of sea waves. A movement in flux. Geißler describes rhythm as an evenly repeating, systematically structured sequence of motion. It is order and movement and free flow all at the same time. As such, rhythms are not inflexible, they structure the time, dividing it, but not in the sense of a rigid beat. The organisation of rhythm and time is a fundamental condition of the perceptibility of time-based art. This can, however, be generally assumed for all perceptive phenomena. Language, poetry, film and music are shaped by rhythm. But rhythmic structuring principles also continue in the visual arts and architecture as spatial and temporal phenomena. One speaks, for example, of monotone or structured facades in architecture. The arrangements of three-dimensional structures also fluctuate between function and rhythmical elements. Points of contact can easily be found in the visual arts, for example with Klee and Kandinsky, who engaged with the universal laws of rhythm and music in great detail.

  If the “Zeit-Raum” (literally “time-space” or “space in time,” “timespan”) describes the linear measure of a period of time, a duration, which is of importance in the registration of signals and contents, then “Raum-Zeit” (“space-time”) illustrates the spatial navigation or exploration of a physically defined area subject to time. Time and space are, in the end, two sides of the same coin. Even if these interdependencies are—regarded theoretically—analysed and described separately, human perception only takes place in temporal-spatial synchronisation at the interface of here and now.

  Our perceptual references to space and time have become more complex and abstract in the past decades. Increasing mobility, the invigoration of space, scientific discoveries about spatial-temporal interrelations and the stretching of the universe, and not least the growth in virtual parallel worlds, among others, contribute considerably to this. In this way, space and time condense to form multi-layered poly-dimensional experience clusters, which break free of immediately experienceable frameworks. Not only these shifts, but especially the interrelationship of physical and virtual worlds in hybrid experiential spaces, give grounds for new spatial-temporal concepts. This not only leads to significant changes to our cognitive environment, but equally influences our rational behaviour, our power of imagination, emotions and our symbolic anchoring.

  Hybrid situations generate complex multidimensional patterns of perception and expand well beyond the physically-temporally determined space. Simultaneity and parallelity, continuous availability, selection, and arbitrary repetition are only some of the elements that organise time based on media and the media determined space. In addition, hybrid processes are imperceptibly becoming a part of everyday life. They permeate work processes, education and information, entertainment and play, and are long since part of the public and private spheres. If the first phase of digitalisation mirrored real phenomena and circumstances in a virtual space as simulation, this process is now noticeably beginning to reverse. Virtual characteristics now lie over the real world like foils. They change the quality of communication and information exchange, become part of the most varied of materials and objects and permeate the urban environment, which is expressed in the coining of terms such as “smart materials,” “responsive environments” and “intelligent architecture.” The hybridisation of our living areas, i.e. the exchange and permeation of physical and virtual characteristics, will grow into the future and will not stop at the borders of our own bodies.

  With the increasing technologisation of exhibitions and presentations, questions around orientation and perception in hybrid exhibition situations also grow in importance. Dealing with multidimensional spatial-temporal constructs indubitably requires new strategies and concepts, a task that media scenography and dramaturgy in particular should be dedicated to. Questions to be answered include: how is perception organised temporal-spatially and which structural conditions are prerequisite for this; how do these prerequisites present themselves in hybrid situations and non-linear presentations; are there dramaturgic or other characteristics that support the comprehension of multidimensional clusters?

  Some of these considerations can be illustrated very vividly in the following examples of work by young artists, architects and designers. The purpose is the exploration and discovery of new references and constellations, the transcending of real and virtual areas, the invention of a different aesthetic and sensuality as well as contents that engage in these changed parameters. The handling of time and space—as varied as this may be presented—plays a special role in these examples. The same goes for the involvement of the viewer in the action, the way they are addressed and their active integration. Numerous, especially interactive, works only find their expression through the participant and their willingness to become involved with it.


Condensation
Time, space and rhythm as narrative construct
  Time, space and montage are the classic form-giving and organising principles of film. They serve the orientation in terms of space-time and content, and have developed a comprehensive and independent set of rules over the course of film history, especially in narrative film. In a certain way there are numerous correlations with exhibition scenography, which develop their contents in a linear fashion, for example historically, related to people or themes. However, the creative handling of these principles opens up completely new worlds of association and perception.
Black_Lead, a triptych-like, space-appropriating 3-channel video installation by the young Swiss artist Corine Stübi, scrutinises the importance and effect of these parameters: time, space and cinematic montage are materials that she rejects semantically and converts connotatively. On the narrative level, two young girls, two older women, a little girl, a little boy, two shady-looking young men and a luxurious black car meet in an underground garage. Further accessories include a golden ball and a lifeless rat. This collection of people and objects, which first appears random, is endowed, by means of the montage, with an extremely complex network of meaning, which plays with the symbolism of filmic genres, mythical imagery and structures, social and in particular mass-media stereotypes, gender themes and psychoanalytical interpretations. A work emerges under the surface that is exceedingly disconcerting, disturbing and very difficult to grasp.

  Black_Lead, a multifaceted semantic mesh, finds its counterpart in aesthetic expression. The very specific dealing with space and time is the essential characteristic of this work and is part of the content-related, scenographic and dramaturgic total concept. Simultaneously, the extensive repertoire of an author familiar with electronic media is reflected in it. Corine Stübi works with repetitions and electronic loops, dissects spatial cohesion into parallel, temporally synchronous worlds and windows, works with breaks and rhythmical markers, sets up a narrative strand and promptly resolves it again. She steers the perceptual interest of the viewer in ever-new directions and interpretations: what is anticipated is rejected again in the next moment, the unexpected strikes against apparent rules. This impression is reinforced by the triptych of the triple projection: complexly designed contents and structures are not only positioned along one narrative axis, but rather on three narrative strands that correspond with each other in parallel. Space and time transfer into inter-media spheres between reality, virtuality and dream. Black_Lead is a work about the effects and possibilities of time based on media and the media determined space. Simultaneously it is an impressive example of how perceptual experiences are evoked and guided by time-shaping means and a headstrong dramaturgy.


Transgressions
Spatialisation and temporalisation of energetic states
  cognitive_housing, a pneumatic sculpture by the architect and media designer Mohamed Fezazi, is an example of the spatialisation and temporalisation of bio-physiological processes. Fezazi feeds activity data from his brain, logged digitally in six sections, into six corresponding pneumatic elements of his spatial sculpture. Each element thus represents a certain part of his brain and becomes a visible indicator of hidden bodily processes, which occurred at certain points in time.

  The aesthetic transcription of this process is determined by materiality, size, volume and form. However, of prime importance is the transfer of the data into a spatial construct as well as the behaviour of this “data-space” through time. The sculpture would be nonexistent without this parameter. The stretching, tightening, coming to rest again and falling in on itself of the individual elements proceeds in an organic kinetic architecture, largely abstracted from the original processes, or in a physio-temporal store. The recorded brain activities become the basis of an aesthetic game defined through time and space, with ever-new constellations and views. With this work, Fezazi has managed to transfer immaterial phenomena artistically into a time- and space-based structure: an accommodation of cognitive energies.


Time objects
Materialisation and dematerialisation of space and time
  The concept of the time-object already contains several phenomena: firstly a spatialisation of time and secondly its materialisation. Time to touch.

  Vera Doerk’s interactive installation z2-TimeObjects was preceded by various artistic phases. The starting point of the work cycle was the phenomenon of movement in space. Vera Doerk produced video recordings of people in public spaces, who crossed through the space along different axes to the direction the camera was pointing. Time, however, was not only measured in the distance covered from A to B. The recording medium itself became the metric measure of time. The human eye perceives 25 images, played in one second, as a continuous flow of movement. Vera Doerk removed one second from the stream of time, by taking away one image every five frames. These images, enlarged and printed on plexiglass, were then staggered one after the other in equidistant intervals in space and thus installed as time-objects. The intervals correlated to a visual rhythmisation or metric. Sculptures of a temporal equalisation were created.

  However, it is not the frozen images that are thematised, but rather the changes between them that cannot be represented. Vera Doerk visualised the striking changes in the people’s recorded movement using taut nylon threads. The volumes of movements created then formed architectural spaces, arisen from movement and equalised time.

  In a further step, these sculptures were transferred into virtual, interactive real-time objects. People and dancers, who moved on a stage, were recorded in five steps a second by an electronic camera and the pictures fed directly into a PC, which calculated the changes in movement as a virtual network of routes between the states of movement. The participant now described virtual time-objects in a virtual space with their movements and dances.

  z2-TimeObjects is an interesting example of the transcending of traditional ideas of space and time on many levels: an action performed in space and time passes over directly into a virtual space-time construct and is reflected back into the real situation in a loop. The participants experience a wondrous mirror: through making otherwise invisible interrelations visible—in this case through the movement architectures created by the body—a complex counterpart arises, which can be explored playfully and reintegrated in spontaneous actions. The invention of ever-new rhythms, sequences of movement and gestures becomes the actual object here, playful and yet analytical.


Immersive environments
Virtual spaces of experience and projection shells
  Flüssige Musik (Fluid Music), by Chinese artist and designer Zhe Li, is a virtual playing field, whose media presence takes possession of the physical space. While Vera Doerk projects the real space into virtuality and generates a feedback loop, Zhe Li expands artificial space into the real, architectural exhibition situation and overlays it with digitally generated information. Optically facilitated by mirrors and double projections, the visitor now stands in the middle of a fantastical world of water sounds and their visual associations.

  Ink-like paintings and animations can be called up and playfully altered via an intuitive touch interface. Drops are dabbed with the fingertip, sea waves formed by hand movements and audio impressions and rhythms, as well as how to deal with time, are left to the experience of the visitor. The physical space only serves as a shell here, as the anchor for an immersive experience, made up of sound, light and digital data.

Poly-dimensional worlds of data
The digital alter ego in cybernetic space
  Mirror_Space by Brigitta Zics is composed of a complex system of visual strands of data taken from various sources. The viewer becomes part of the virtual installation. First they enter a kind of antechamber at the transition to a data space. Bodily features, for example proportions or temperature, are registered here using a scanner or a camera. This personal data is then merged proportionally with data taken from the Internet to form a joint stream of information and passed on to a graphical alphabet of shapes. A kind of avatar, or better a hybrid being, is created using this personal and Internet data. The form, colour and modes of movements of this virtual entity are determined in real time by the incoming data.

  As such, each avatar is, in terms of the people present in the space, unique. An infrared camera installed in the playing space now registers the movements of the visitors and returns this data as kinetic input to the respective avatar. The hybrid being now follows the movements of its real model being like a mirror image. Up to five visitors can act and interact together playfully in the fantastical world of this virtual space. When the visitor leaves the field of play, their avatar or mirror image also expires after a certain amount of time. The installation is a successful example of a hybrid playing field based on space and time, which can be accessed by the viewer spontaneously and intuitively. As in Mohamed Fezazi’s cognitive housings, Mirror_Space transforms data generated from people or bodies into a kinetic driving force in a space-related design. Mirror_Space however expands this approach to include interactive experience components, which take on a temporary mythical character through the emergence and disappearance of virtual alter egos.

Paideia
The role of emotional appeal and playfully intuitive access in multimedia environments
  All of the examples presented are characterised by a strong emotional appeal. This becomes a quality especially in the interactive installations. Only through the active involvement of the viewer, or their behaviour as a participant, do the works open up in all their dimensions.

  This is where the communicative and narrative potentials of dramaturgy, which transpose temporal structures into experience events and thus decide on the course and interest trajectories, become relevant. The “hook”—the dramaturgic introduction and emotional involvement of the viewer at the beginning of the action—takes on a key role. It corresponds to the emotionally intuitive access to objects and facts in an interactive setting or in the exhibition context. As a playful, eventful beginning, the desire to make contact and a continuing occupation with the contents and exhibits at hand is often decided here.

  Thus a closer consideration of playful components goes side by side with an engagement with dramaturgic characteristics. What is interesting in this is the sheer range between the poles of Ludus (game) and Paideia (play). Whereas Ludus comprises playful activities that are characterised by a self-contained system of rules, actions to be carried out, procedures and gratifications or also possible defeats, the Greek term Paideia describes, in this context, physical or mental activities with a focus outside strict systems of rules and more on joyful experience, invention and experimentation.

  Although Flüssige Musik, Zone2 and Mirror_Space all have a defined playing field, like in constructed games, with their own mainly virtual laws, they do not necessarily contain a fictive world that operates according to its own rules. They are open offerings, which function without strict guidelines on how to act or behave. They can also be accessed spontaneously via intuitive interfaces and integrate the viewer into the action from the first moment of contact. This is also the key to their success. They are virtual playing grounds, somewhere between mimicry and simulation, Ilinx and movement or exploration play, which can largely be assigned to the area of Paideia. Their particularity lies especially in the fact that they refer to complex interrelationships beyond the playful event. The playful approach practiced here does not therefore exhaust itself in the direct experience of the object but rather refers, with a playful gesture, to a network of connotative references, which emerges from the complex polymorphism of the individual pieces.

  The often discussed model of non-linear storytelling reaches its limits, especially in the mediation of non-linear cooperative contents, because perceptual phenomena are often disregarded. Interactivity, playful deviations, as well as hard to predict reactions in human-machine communication can, however, take on the position of markers within a perceptual chain or that of a drastic feature in a rhythm sequence, which aids or advances conscious learning and experience, in order to bind attention and interest. Regarded in this way, the playful approach, with its emotional and open components, could take on the function of a structural element within the communication and cognition process and could thus influence, more decisively than ever before, the design of interactive processes both as part of individual projects as well as in complex exhibition scenarios.


Installations

Corine Stübi: Black_Lead; music: Death in Vegas; 3-channel video installation, 2006 (Prize at the International Short Film Festival Oberhausen 2006; Prix de la Création Video/Vidéoformes Clérmont Ferrand 2006)

Mohamed Fezazi: cognitive_housing; pneumatic sculpture, 2006
Altitude Cologne, 2006

Vera Doerk: z2-Zeitobjekte; interactive installation for performance and public spaces, 2004 (Traumzeit Festival, Duisburg, 2004; Imagina, Monte Carlo, 2004; Em Transito, Lisbon 2004; Popcom, Cologne, 2004)

Zhe Li: Flüssige Musik; interactive immersive playing field, 2006
Aquarius Wassermuseum/RWE, Mühlheim Ruhr, 2006

Brigitta Zics: Mirror_Space; interactive installation 2004/05
Nominated for the international media art award, Karlsruhe, 2004; Siggraph, Los Angeles, 2005; nominated for TTA Top Talent Award Europix, Vienna, 2006

All projects have been developed at the Academy of Media Arts Cologne,
LAB of media-scenography /Interactive Media & dramaturgy,
Faculty: Prof Heide Hagebölling-Eisenbeis
www.media-scenography.org

Literature

Caillois, Roger. Die Spiele und die Menschen. Maske und Rausch. Frankfurt/Berlin/Vienna, 1982.

Darley, Andrew. Visual digital culture. London, 2002.

Geißler, Karlheinz A. Vom Tempo der Welt. Freiburg/Basel/Vienna, 1999.

Hagebölling, Heide, ed. Interactive Dramaturgies: New Approaches in Multimedia Content and Design. Heidelberg/New York: Springer, 2004.

Huizinga, Johan. Homo ludens. Reinbek, 1997.

Metzinger, Thomas, ed. Bewusstsein. Beiträge der Gegenwartsphilosophie. Paderborn/Munich/Zurich, 2nd edition, 1996.

Pöppel, Ernst. Grenzen des Bewusstseins. Frankfurt, 1997.

top ↑