Technical specifics

Technically this mapping process is executed by preprocessing the MIDI file, which is in industry standard binary format. MIDI data consists mainly of Note-On and Note-Off instructions grouped by instrument (MIDI track) and arranged chronologically within each track.

The MIDI file is first converted to ASCII text using a perl script, to facilitate reviewing the raw data for problems and to assist later debugging and verification. (The perl script was written by Stuart Levy of the National Center for Supercomputing Applications at University of Illinois, Champaign-Urbana.)

This text file is then processed with a Microsoft Visual C++ program (written by Absolute Reality's David Britton) into a list in which each line represents one note, with its attendant attributes. This program reads a master list from a text file containing both the RGB color definition and the landscape image to be assigned to each instrument timbre found in the MIDI data.

This list is then processed by a custom Caligari TrueSpace plugin (written with Borland C++ by Richard Rodriguez of Absolute Reality). The plugin can either directly generate the 3D geometry in TrueSpace to use in testing or viewing still images, or it can output a text file describing the geometry for use in the dynamic VR world.

This final geometry text file contains one line for each note consisting of four sets of five numbers and the name of an image file. The four sets each specify a point in 3D space (the x, y, and z coordinates) and the positions (the u-v coordinates) specifying which parts of the image file to use to display on the face of the four sided polygon. It is this text file that is read in when the VR piece is started, and its data is used to create the visual representation of the music synchronized with the music's playing in the VR world.

These several steps could be combined into one, if it were desired to make a piece that would react to live real time MIDI input. Separating the steps was useful in evolving the precise algorithms to use, and preprocessing as much data as possible reduces the real-time load on the display computer.

Interactivity

The viewer is allowed to move through space at will and view/hear from their position in relation to where the music is playing. All of the sound is being programed to be interactive with the position of the viewer at Boston U. by Robert Putnam. . The sound programming should also include acoustical context. For instance, if we should add architectural spaces to the composing space possibilities then the sound programing would have to reflect the changes in venues. Architectural interiors bounce sound differently than desert landscapes.

Absolute Reality Corporation's AbsoluteAuthor-VR is the virtual reality software application that places these models into a virtual world in which the user can move the viewpoint at will, and in which events occur; in this case the music plays while its geometrical visualization appears. David Britton of Absolute Reality programmed this software. Drawing on his experience with developing the software for LASCAUX, a VR tour of the French prehistoric painted cave, Britton selected the World Toolkit (WTK) VR library from the Sense8 product line offered by Engineering Animation, Inc. (EAI). Sense8-WTK provides a powerful library of functions to manage the simulation and rendering loop that computes for each frame or unit of time what the new updated view looks like and paints it onto the computer screen. AbsoluteAuthor-VR is written in ANSI standard C++ for cross platform capability, using the Sense8-WTK library, and is available for both Windows and SGI computers.

The AbsoluteAuthor-VR program allows a non-programmer to interactively define locations and paths in the world, and to create events that can be triggered to occur under specified circumstances. For Q, the program was enhanced to incorporate dynamic creation of the musical geometry shapes synchronized to appear and to grow in time as the music plays.

The Sense8-WTK library also provides integration with various virtual reality immersion devices, such as head mounted displays and motion sensors. This feature of the WTK library is especially useful because Q is intended for a fully immersed installation, such as the CAVE environment and the Immersa-Desk developed at the Electronic Visualization Laboratory (EVL) of University of Illinois, Chicago.

Hardware

The physical computer implementation, i.e. the hardware, for the VCO requires a computer of sufficient power to render a large graphical environment in real time, at speeds approaching 20 frames per second or more. The 1997 version of the CAVE at NCSA used an SGI Onyx with three Infinite Reality graphics pipelines, each split into two channels, according to their web site. The Scientific Visualization Lab at Boston University uses an SGI 4-processor Onyx2 with 2GB memory and InfiniteReality2 Graphics for their ImmersaDesk installation. A Windows NT computer system would need at least a dual, and preferably a quad processor Intel Pentium III CPU and a high end graphics subsystem such as Intergraph's Wildcat 4100 board.

The VCO Project has physical installation specifications for several levels of immersed performance, from a single-user screen display, to a full CAVE environment.

Resolution problems

This is the point where I would like to talk about the resolution as a matter of primary importance. I am an artist who has come from the world of extremely high resolution drawing and painting. My very large works, sometimes reaching a size of 800 sq. feet, have all been created with #1 and #2 tiny sable brushes. They have been produced in a precise way so that they will still hold on to visual meaning even when they are cut and collaged into tiny pieces. The success of my visual work has always depended on being able to see and experience the layering together of complex, detailed systems of image, collage, and color. I discovered quite early in this process that if resolution goes too much the work ceases to carry meaning. You have just been viewing 35mm slide projections of still shots from Q and a fair amount of detail has been seen, although it is nothing compared to the way we could render them into digital continuous tone prints. However, you can see the metal textures on strips which are produced by metal percussion instruments, and you can see the images which are embedded. This is a far better resolution than you can get in the low resolution environment of high performance computing. What kinds of losses does work sustain which depends upon layers of details? I think that the damage is great, but this should not stop our attempts. The solution we came up with mid-process is to aim for a very high resolution and downgrade as much as one needs to accommodate a particular system. Q is being prepared so that it could go into IMAX with its large scale film size and show full detail. This way, as the technology improves, the piece can be reinvented. It serves no useful purpose to design a work for a level which will be surpassed in a short time.