Synopsis

 

This proposal is to implement virtual space for a new form of multi-sensory (visual, auditory, and tactile) experience based on immersive content and brain-computer interfaces based on the theoretical considerations of immersive content, simulacrum, user experience data, etc. with ‘picturesque English garden’. Even after the proposal is completed, I will continue to study ‘human nature, mind, and brain’ and ‘real content’ And the interpretation of the viewer’s sensory experiences will help researches to understand how the viewer perceives and responds to the work. Also, I will see how this experience transcends the everyday perception of the world around us.

 

Introduction

 

1. Background and purpose

This proposal began with the questions of “Can future immersive content, such as virtual reality, be used as a form of art?”, “How can we use reactions such as the mind and the emotion of human nature to provide a different experience in art?” These questions were based on the reality that the existing viewer experience was not a two-dimensional interaction on the screen, but a realistic content using the viewer’s five senses, and the application of virtual reality technology to the art field was essential in the era of individual data and the fourth industrial revolution, and that the subject could increase interest in human nature and the importance of nature.

For a long time, people dreamed of an ideal and good society. And they tried to make progress in the development and technology of the humanities to achieve that. Especially in the West and the East, this world is Utopia, Arcadia, and Parada describing this as an ideal world, a heavenly world, and the efforts to embody it in this world appeared in the form of a “Garden.”

“Garden” with human intent as a cultural landscape involved has been recognized as a valuable cultural element that shows the unique cultural attributes of the mankind, while having global, humanistic attributes beyond time and space, as a cultural device for humans to adapt to nature to enjoy survival, prosperity, and pleasure. The designed garden represents the ideal and value of owner as a cultural product, is located in its social environment, and the process in which the garden is created by the combination of natural and artificial elements in different times and places inspires modern designers and affects their work.

This proposal is to implement virtual space for a new form of multi-sensory (visual, auditory, and tactile) experience based on immersive content and brain-computer interfaces based on the theoretical considerations of immersive content, simulacrum, user experience data, etc. with ‘picturesque garden’.

 

2. The content of the proposal

 

This proposal was a link between individual emotional reactions and curiosity in the reflection of the times of experience with the expectation that the link between realistic content and Brain-Computer Interface would come to us in a new form of art in the future, and with “Virtual Garden” and “Multi-sensory Experience” with the potential to develop virtual art. It targets those who have the potential for the development of art and technology, those who seek a new form of art in fine art, and the public who are not related to art, and all audiences interested in spatial art, such as gardens, environment, media art, architecture, and urban design.

 

3. Proposal task progress

Theoretical Background

 

1. Immersive content

1) Definition

‘Virtual Reality’ is a technology that allows users to experience really difficult situations in the real world by extending and sharing sensory information within a computer-generated virtual environment, while ‘Augmented Reality’ is a technology that combines virtual content with real- world and things to provide users with information on the environment. If virtual reality allows users to experience content disconnected from the real world through immersive devices, augmented reality is distinguishable by the fact that it presents content that is fused with reality. Recently, with the emergence of Mixed Reality, where the combination of the virtual and real worlds becomes more natural, technology is being developed to maximize the immersion experience of users.

This immersive experience has evolved into an online multi-user environment, which is collectively referred to as “immersive content,” expands the scope of virtual information on the current visual base to five senses, and allows multiple users to share and communicate the same virtual space without limits on multiple distances within a single user environment per device.

 

2) Characteristic

The goal of realistic contents is to embody ‘the practically same space as the real world’ through various sensory stimuli such as human senses of five senses and balance, with three characteristics: ‘Immersion,’ ‘Interaction’ and ‘Presence,’ the main contents are as follows:

(1) Immersion

Immersion means the feeling of digging or falling out of an object. Usually, people who use virtual reality forget their consciousness and perception and react only to their goals when they become immersive because most of the senses that people perceive are concentrated in sight. Therefore, visual requirements should be satisfied to enhance immersion, which is valued in realistic content.

(2) Interaction

Interaction means interacting with a device within a virtual space. As the input and output of the user’s information are instantly responded to within the space, the process of interaction is critical. When a viewer takes action within a virtual reality, it can increase the viewer’s sense of immersion by inducing the objects to respond. Interactions can be implemented in conjunction with external sensors and controllers

(3) Presence

Presence means ‘feeling in a certain environment’ Heeter (1992) used the expression “being there” to indicate the feeling that a user is in a virtual environmentv, while Slater and Wilbur (1997) defined presence is believing that the user exists in an environment other than the real space and the feeling as being remote.

 

3) Forms and types of virtual reality

Virtual reality is a real-life experience of difficult situations in the real world, and types are divided into three levels depending on how the interface or system is implemented, as follows:

First, non-immersive virtual reality is a virtual reality that is implemented with traditional PCs, such as keyboards and mouse monitors, and can be produced at minimal cost. Although popular, I feel a low level of immersion and presence.

Second, semi-immersive virtual reality is a virtual device that is immersive given by multiple screens and large monitors. A typical example is a movie theatre and a game.

Third, Fully-immersive is a virtual reality that uses separate hardware devices to block the external environment and reproduce the highest level of immersion. Various senses can be felt within virtual reality as hardware advances. Examples include ‘HTC Vive’ and ‘Oculus Rift.’ As an era of experience in the future, immersive virtual reality was used in this proposal, because immersive virtual reality would lead to a familiar and high level of user experience to the public.

Fully-immersed virtual reality is implementable in two different ways: Output Devices to reproduce virtual reality and Input Devices to receive input from viewers. In particular, output devices include a head-mounted display (HMD) that allows viewers to experience virtual space by being mounted on the head and an input device (controller) that receives viewer feedback. And the current virtual reality device automatically calculates the location, direction and height of the viewer by borrowing a method called Six degrees of freedom (6DoF), and changes the point of sight as the viewer moves. This has been changed so that the viewer can move beyond the space limit and enjoy what he or she wants.

 

4) The spatial significance of virtual reality

The viewer experiences a data-driven space designed by the operator in virtual reality. Realistic content is non-realistic and non-material space, and viewers experience immersion, empathy, behavioural identity, and transformation in virtual space. In particular, the release of consumer virtual reality devices by ‘HTC Vive’ and ‘Oculus Rift’ since 2016 has aroused the general public’s desire to experience virtual space and enabled viewer involvement in multiple areas, not the passive story experienced by past two-dimensional monitors. Thus, if the physical space of reality is formal and limited, including the concept of time, then virtual space is not formal, and time can be manipulated and amplified. These fabricated experiences trick the viewer’s brain into creating an illusion of emotion. For these reasons, the designer’s intention is important when creating a virtual space.

 

5) Future prospects

In ‘Top 10 Strategic Technology Trends for 2019’ released by Gartner, technologies for immersion experience were selected as key future technologies for building next-generation digital business ecosystems, while IDC forecast that the global AR/VR SW market will grow 44 times from about $2.1 billion in 2016 to about $92.6 billion in 2021.

 

2. Spatial properties of simulacrum

 

1) Definition

Simulacrum refers to an artefact that has created an object that does not exist. According to Plato, the human world is made up of a prototype (Idea), a replica (reality), and a replica of a replica (Simulacrum). The reality here is that human life itself is a replica, and we saw Simulacrum as the “imitation of imitation,” which is an inferior subordinate concept of “imitation of idea.” In other words, it was God itself that was created by a whole nature and a hostile being, and Simulacrum is considered as a form of chasing God by imitating its essence. Later, the concept of simulacrum began to take notice after the 20th-century post-structuralism’s representative philosopher Gilles Deleuze presented a spatial concept that was distinct from structuralism. The simulacrum that Gilles Deleuze thinks is not just a dichotomous replica of Plato, but an important concept that has self-identity creating a new area of its own without stopping to clone the cloned model, and later developed into an important concept in modern philosophy.

 

2) Simulation and Virtual Space

From Plato’s ‘Model Simulacrum’ to Gilles Deleuze’s common feature of ‘Simulacrum, where matching the original is not important’ is ‘reappearance,’ and Jean Baudrillard well highlights the importance of original replication before the concept of ‘Simulacrum’ and well explained about ‘virtual replicas that are more practical than original.’ Jean Baudrillard talked about simulation of a new image in his book [Simulacres et Simulation], published in 1981, which defined modern society as being dominated and replaced by simulation, and he claimed that simulation is the movement of Simulacrum which is the simulated condition. In other words, the core concept of Jean Baudrillard was called a real artefact as ‘Simulacra’ that was transformed into a derived entity, not an actual one, and created as if there were objects that did not exist in real life.

Simulacrum exists together as an infinite source, not as a dichotomous reproduction of the simulation. Many people think of Simulation as Imitation; however, it is wrong. There’s a real object for imitation, and if people copy this real one, that’s what imitation is. These imitations belong only to the image of a traditional reproduction system, far from ” Simulation.” Simulation is an image that has nothing to imitate, and unoriginal images replace reality in themselves, and images dominate reality, therefore It’s more realistic than it is.

In other words, ‘Simulation’ is a more realistic and one independent reality. What we thought was real so far comes from simulation, which we said was unrealistic. In ” Simulation,” realism is nothing more than an “effect” or a “fantasy,” and all re-enactments are made through a code of re-enactment, the relationship between structural terms that are exchanged and generated at all times, rather than a reflection of the existing reality.

Previously, frivolous replicas such as unrealistic, imitative, and unrealistic ones are believed to be more dominant than real, according to Jean Baudrillard. Clones are presentable and are an infinite source. This is because there is no longer any distinction between the “reality” of the real-world and the “replications” of imitation, and the boundary between reality and cloning is gone.

The reality is no longer truth; it’s just an operational reproduction. In other words, the reality that we live in can be ‘replications’ rather than ‘reality’ and ‘replications’ can be ‘real’. The further boundary is gone and inside the simulation.

 

3) Application of spatial properties of simulation

Virtual refers to a futile world that, subjectively, seems real, but objectively does not exist. In other words, Virtual is an earlier stage of practical or realistic embodiment and is a very major dimension of reality in a philosophical sense. Virtual reality seems to represent a contradictory or mysterious world, as the reality often considered the opposite of virtual, assumes material embodiment, detectable appearance. In reality, however, virtuality and presentability are two different ways of reality. Thus, virtual and reality exist together without dichotomous separation. Therefore, virtual does not mean that it is not true or that it is not clear, but that it coexists with reality, which implies that the present is a source of existence. In other words, ‘virtual space’ refers to a place that exists with an experienced reality, not a vague and mysterious world.

According to Jean Baudrillard, the visual phenomena of the 20th century are no longer paintings, photographs, fiction, and unrealistic visions under the perspective. In other words, a world of simulation that the boundaries of reality and hyperréel-hyperreal become blurred, and the meaning of reality and hyperréel-hyperreal disappear is created. Real experience and memory of action become real, and space creates virtual space through hands-on experience. So, virtual space gradually becomes the product of a simulation that is more real than real, and the simulation hits the simulacrum and destroys the boundary. Jean Baudrillard argues that this place where we live is none other than a virtual reality within a confusion of Simulacrum.

 

3. User Experience – Emotion

 

1) Definition

The brain that controls the user experience controls all human behaviour, and neurons do information processing activities such as learning, memory, and problem-solving. The structure of the brain can be largely divided into the forebrain, the midbrain, and the hindbrain. The forebrain is a region associated with the cerebral cortex that accounts for most of the brain. The brain is divided into the left brain and the right brain and is also divided into the frontal lobe, the parietal lobe, the temporal lobe and the occipital lobe based on the measuring of human brain electroencephalogram.

 

2) Brainwave classification

The types of brain waves for measuring emotions are divided into alpha wave, beta wave, delta wave, gamma wave, and brain waves vary with the degree of brain activityxviii, as follows:

 

• Alpha wave – Alpha wave is a band of 8-15Hz and occurs during rest and It usually appears relaxed and increases amplitude in a relaxed state. For normal people, increasing alpha waves means meditation, peace, and calm internal conditions.
• Beta Wave – a band of 16-31Hz, brain wave that occurs in a state of awakening, tension, or
• Delta wave – a band of 0.5-4Hz, brain wave that occurs in a deep
• Gamma wave – bands greater than 32Hz, brain wave that occurs when highly cognitive action is A waveform usually expressed when enlightenment or insight occurs.
• Theta wave – 4-7Hz band, brain wave that is related to creativity, meditation and drowsiness. Appears in a shallow sleep state between sleepiness and wakefulness. Usually, theta wave is defined as a state of the boundary between perception and dream.

 

4. Garden and culture

 

1) Overview

The gardens in the world are different from country to country and from region to region and from time to time, and differ according to the profusion and affluence of flowers and leaves that bloom and fall according to the season, but they are all one in all, given that they are gifts with a thousand faces given by nature entering the garden of the house.

These gardens, which are grouped according to different regions and cultures around the world, maintained a consistent appearance that was formatted in a certain pattern, as they had almost consistent basic principles based on the problems of vegetation and irrigation in accordance with the climate and cultural characteristics unique to the culture.

 

2) English garden and Landscape

By the era of Elizabeth I, Britain had formed a strong monarchy since Henry VIII, and a high level of culture, such as the literature of William Shakespeare, had blossomed. Britain, which has not been able to make clear its successor after the death of Queen Elizabeth I, is once again in turmoil, and a major change in the garden occurred as well.

The French garden style, which was abundant until the Tudor dynasty, has become a target of criticism, and calls for breaking the framework of the garden locked in the formal style are beginning to emerge through literature.

The appearance of landscape gardens began in the late 17th century, the biggest change in English gardens. The background of the landscape garden can be discussed with the basis of understanding Alexander Pope, the poet of Neo-classicalism with the Grand Tour. In 1727 Sir. John Clerk (1676-1755) sang the ideal beauty of enjoying the landscape in the countryside through the poem The Country Sheet, and Alexander Pope (1688-1744) claimed that French formal garden shows low beauty and true beauty is the beauty that resembles nature.

The desire for political freedom and the tiredness felt in the stereotyped French garden led to the desire for a new garden, and this desire made an important turning point for the English garden.

Grand tour, which was popular with the young, wealthy aristocrats at the time, made many Britons interested in Italian gardens, wanted to create landscapes in front of their homes, and created the most natural but artificial British landscape gardens.

 

In the early 18th century, there was a growing interest in the Italian landscape expressed in landscape paintings by artists such as Claude Lorrain and Gaspard Puossin and attempts to create a new culture by interpreting these landscapes in a classical way. The tiredness of strictly controlled formal gardens and yearning for nature-like spaces began to make landscape paintings that were no more than flat into three-dimensional ones when they needed the creation of new forms of gardens. This result created the Landscape Garden in England.

William Shenstone (1714-1763), a poet who sang the scenery of the countryside, said the garden resembled a dramatic poem, and created a view point where visitors could see the scenery while expressing the beautiful scenery in his garden work. Experience the scenery through the view has become the main way to appreciate landscape gardens, and various landscape sequences have been created using curved walkways.

The Ha-ha technique also attracted scenic views outside of the territory into the garden. The landscape garden, which was born and started to be created by writers, was established as a genre by William Kent (1684-1784) in the 1730s, and Kent, an ordinary artist, began to make landscape gardens using his painting techniques. This means that the landscape painters began to jump directly into the garden building, and although landscape artists had designed the garden through paintings so far, they later played a role in making garden designers appear in society.

Roman sculptures and garden buildings were the first to be used in Chatsworth House garden, but they became the most popular garden in England, such as Rousham House Garden, Stowe Landscape Garden, and Stourhead Garden. The vast landscape garden was also open to the public, and Richmond Park, designed and created by Charles Bridgeman (1680-1783) was a classic example of the transformation into a public park.

Particularly, Stowe Garden is also known for bringing about an innovative change in English landscape gardens. Landscape gardens have been designed to appreciate gardens in one of the best views, but Stowe Garden is designed to stroll through the gardens or take a tour of connected copper lines to enjoy natural, beautiful, and connected scenery. Since then, many gardens have developed into landscape gardens with sequences along these patterns.

 

Research Methods

 

1. Research materials

The garden is a multi-sensory art that allows experiencing the emotions of its aesthetic object with the movement. Garden art movements include first, actual and explicit movements of natural elements, second, visitor movements made when experiencing the garden directly, third, relatively slow-paced movements made by changes in daytime, seasonal and weather, fourth, allusive and metaphorical movements produced by ancestors and monuments. Especially for the walkers who walk through the forest, a new way of recognizing the landscape is created, such as a tree that looks like moving. And the movements listed above result in a move called “emotion” that touches the mind and heart of the viewer.

From the above perspective, ‘Picturesque English Garden’ was selected as the material of this proposal with a focus on visual perception, emotion and art, and the main reasons for this proposal are as follows:

First, the landscape created by nature changes every moment, and at one point it is described as “beautiful as a picture.” From the “picturesque” landscape that captivates the human eye, a philosophical landscape that attracts the soul is created. It’s to enjoy a time, a unique moment in the garden that never comes again.

Second, in the 18th century, visual media such as panorama, diorama, stereoscope, photography, and Movie were invented, expanding the public’s senses and changing the way views were viewed in the era of Picturesque. In other words, the visual system based on linear perspective has been transformed by the influence of the new optical instrument into a ‘visuality of movement’, which is summarized as ‘dynamic perspective’ and ‘relativity of perspective.’ This visual process has been applied to ‘garden’.

Third, the gardens of the study are as follows.

2. Experimental equipment

The experimental equipment used in this proposal is a high-performance computer, virtual reality hardware, and a brainwave measurement device for virtual reality implementation.

 

1) Computer

A custom PC with high-performance CPU and the graphic card was used, as it was required to receive a high level of real-time 3D graphics rendering and simultaneously implement interactive feedback from the viewer to implement virtual gardens with a high sense of immersion. Details are shown in the following table.

 

2) Virtual Reality Hardware

As virtual reality equipment, ‘Oculus Quest’ that was released in summer 2019 by ‘Oculus’ was used. The higher resolution of 1440×1600, compared with 1080×1200 resolution of the previous model ‘Oculus Rift CV1’, gives a cleaner and more immersive experience. Also, the external body tracking sensor that used to be in the ‘Oculus Lift’ disappeared, and it is also equipped with ‘6DoF inside-out tracking built-in cameras’, a technology that allows to calculate and track user locations on the device itself.

3) Brain wave equipment

The brain wave measurement device is recorded through electrodes attached to the scalp using the electroencephalogram (EEG) .  The brainwave record uses the international standard brain wave arrangement, “10-20 system.”

In this proposal, I used InteraXon’s Muse2 for brainwave measuring devices, which can receive information directly from the computer and can automatically convert the raw data into five types of brain waves.

Implementing Virtual Garden

 

1. Experimental process and fabrication

The main issues of this proposal are in the construction of a virtual reality garden environment using immersion-experienced virtual reality devices, and the main contents are as follows.

 

1) Studio proposal results

The virtual garden implementation test was conducted in about 2 x 3 meters of space in the interaction design fourth grade studio. To implement the virtual garden space, a programme called Unity (2019.2.9f1) was used. The main details of the production status of the virtual garden are as follows. There are two main categories: the implementation of virtual reality gardens and the implementation of the viewer experience using brain waves.

First, a basic level of the virtual garden environment was constructed using the natural environment assets provided by Unity.

 

Second, after experimenting with interactions such as catching objects in a virtual garden and moving them to other places using the hand controller of a virtual reality device, the two- handed controller of a virtual reality device has confirmed that it can interact with objects in a virtual garden normally.

 

 

Third, using ‘OSC – Open Sound Control,’ it succeeded in transmitting and receiving data with brainwave measuring devices and Unity programmes.

 

 

Fourth, I successfully mapped sample objects of the virtual garden environment (Scene) and data values in the unit programme to test the rotation and colour variations of internal objects according to my brain wave changes. They also confirmed that virtual reality devices work even when they are worn.

 

 

2) Details of future work

First, based on the previous content, the following is to be carried out in the studio proposal; additional garden visits and data collection in the United Kingdom, further data on realistic content and simulacrum, and additional study of user experience. I will collect and utilize pictorial and sketch materials of the Picturesque gardens in the East and the West, especially in the U.K., and use 3D tools such as Maya to create and apply more diverse tree and plant garden elements.

Second, after placing the garden elements in the Unity programme virtual garden work environment, the interaction will be designed in conjunction with the viewer’s response and behaviour by utilizing the trigger-type interaction of the Unity programme. For example, a viewer’s hand and arm joints will be tracked to implement interactions such as sounds or effects as they approach a particular garden element.

Third, it will link the viewer’s brainwave data with the garden elements and natural elements in the virtual garden to create interaction.

 

For example, if the analysis of the viewer’s brainwave data shows that the viewer is bored to the virtual garden’s experience, it would reproduce instructions or sounds to move to a specific location within the virtual garden.

If the viewer focuses on a target in a virtual garden, he or she will present a higher level of user experience by analysing brainwave data, including presenting in front of the viewer a choice interface that enables further interaction with the target.

Fourth, I will collect brainwave data values from the audience who visited during the exhibition in June 2020 and organize the possibility of virtual reality and emotional data as a new form of art.

Fifth, the virtual garden experiment path is as follows. If the exhibition space of the virtual garden is not sufficiently secured, a certain boxed space will be created to allow viewers to experience and interact with the virtual garden in a limited space. If there is enough exhibition space, it will be produced so that viewers can appreciate the virtual garden while walking along a certain track for experiencing a real garden. The exhibition will allow the audience to experience a variety of interactions and new visual experiences as they walk the track and interact with the objects in the garden.

 

2. Work process

 

 

3. Pros, Cons, and limitations

(1) Pros

• Introduce as many audiences as possible that one element of personal data plays an important role in the future direction of art.
• Reconsideration of interest in human nature and the importance of nature
• Reflection of the times (in a technical sense, virtual reality, brainwave measuring devices)
• Anyone can easily access and occupy objects that were limited to physical places called gardens.

 

(2) Cons

• Only those who have virtual reality devices can appreciate them.
• Due to limitations in the resolution of graphics and virtual reality devices, it is difficult to provide the same level of garden experience as it is now in 2020.

 

(3) Limitations

• Although it is a realistic content, only visual, auditory, and tactile feeds have been implemented. Taste and smell cannot be implemented.
• Realistic content is maximized in the online multi-user environment, but only one virtual reality device is used in this proposal due to lack of budget and technical implementation skills.
• The sensor (electrode) of the brain wave device is biased only to the frontal and temporal lobe, which measures the brain wave data, and the absolute number of sensors is insufficient, possibly posing a problem with the reliability of the data compared to the specialized experiment.

 

Conclusion

 

To answer these two questions; “Can future immersive content, such as virtual reality, be used as a form of art?”, “How can we use reactions such as the mind and the emotion of human nature to provide a different experience in art?”, realistic contents, simulacrum, user experience data, and gardens in the theoretical background are researched and analysed. As for practical studio work, last semester, I tested a virtual garden prototype, which will continue to be followed by further research in the future by the detailed garden element production and layout and further theoretical background.

This proposal has pros, cons and limitations; however, the new value of this proposal is that the viewer’s creative response (sensory experience) of virtual reality, emotion, and empathy is newly incorporated into the art as a form of ‘data’ so that it can be understood and based on the new art forms and other forms of creativity can be created. And as these interpersonal interests and interactions accumulate, I expect that This approach will contribute to advances in user experience, user interface, and data visualisation in the interactive design in the future. Seeing in the future is not a passive process that is transmitted on a two-dimensional screen, but seeing has again become an active process.

Even after the proposal is completed, my research on ‘human nature, mind, and brain’ and ‘real content’ will continue, and the interpretation of the viewer’s sensory experiences will help researches to understand how the viewer perceives and responds to the work. Also, I will be able to see how this experience transcends the everyday perception of the world around us.

 

 

Y4_DHT_Critical Journal_YONGWON_CHOI