A4 Programmers Manual

Conitec 2000
3D GameStudio A4 Programmer's Manual
1999, 2000 Johann Christian Lotter / Conitec GmbH
This manual is protected under the copyright laws of Germany and the U.S. Any reproduction of the material and artwork printed herein without the written permission of Conitec is prohibited. We undertake no guarantee for the accuracy of the information given. Conitec reserves the right to make alterations or updates without further announcement.

Contents

The A5 DLL interface _________________________________________________________ 4
WDL object structures ____________________________________________________________ 4
The WMP map format ________________________________________________________ 6
Blocks __________________________________________________________________________ 8 Map Properties___________________________________________________________________ 9

The MDL model format

_____________________________________________________ 10
MDL file header _________________________________________________________________ 10 MDL skin format ________________________________________________________________ 11 MDL skin vertices _______________________________________________________________ 11 MDL mesh triangles _____________________________________________________________ 11 MDL frames ____________________________________________________________________ 12 MDL bones _____________________________________________________________________ 13

The A5 DDL interface

DLLs are customer-programmed extensions to the engine and to the WDL language. They can be used with the commercial and professional editions, but can only be created with the SDK (source development kit) that comes with the professional edition. This way, professional edition owners can create arbitrary DLLs for adding new effects, actor AI or WDL instructions, and distribute or sell them to other 3D GameStudio users. The Microsoft Visual C++ development system is required for creating DLL extensions. The DLL SDK contains an interface library that must be linked to any DLL. An example VC++ project with a DLL template is also provided, which makes it pretty easy to create extensions even for not-soexperienced C programmers who have never used DLLs before. DLL extensions work bidirectionally: WDL instructions can access DLL functions, and DLL functions can access essential engine functions and variables. On opening a DLL, the engine transfers the pointer to an internal interface structure to the interface library. The interface contains pointers to engine variables and functions, like the frame buffer, the DirectX interface, the network interface, the DirectInput interface, the level, the WDL functions and so on. Theoretically everything - MP3 or MOD players, a physics engine, another 3D engine or even another scripting language could be added to the engine this way. On accessing system resources like sound, video, joystick and so on, the DLL must take care of possible resource conflicts. The engine shares its resources and expects the same from the code inside the DLL. For instance, code that requires exclusive access to the sound device (like some old MOD players) won't work. Some resources (like the midi player) can't be shared - if midi music is played by the DLL, the engine must not play a midi file at the same time and vice versa. Also care must be taken that for writing something into the frame buffer it must be locked before, and unlocked afterwards. The interface library provides functions for that. WDL object structures Pointers to WDL objects can be transferred to DLL functions, thus allowing object manipulation. The internal engine format of important WDL objects is listed here.
typedef long fixed; typedef struct { long index; // char *name; // long next; // long modelindex; // internal 22.10 fixed point number format

A4 A4 A4 //

internal use only internal use only internal use only A4 internal use only
// the following variables can be externally used by WDL or DLL functions fixed x,y,z; // position of the entity fixed pan,tilt,roll; // angles fixed scale_x,scale_y,scale_z; long flags; fixed frame; // model or sprite frame fixed nextframe; // next frame for inbetweening fixed skin; // skin number fixed ambient; // brightness added to the entity, in percent fixed alpha; // transparency of the entity, in percent fixed lightrange; // range of dynamic light emitted by the entity fixed red,green,blue; // colour of dynamic light emitted by the entity long emask; // event enable flags long eflags; // internal status flags

fixed min[3],max[3]; fixed trigger_range; fixed push; fixed shadow_range; fixed floor_range; long client_id; // fixed skill[48]; // } A4_ENTITY; // bounding box
// collision behaviour // range within a shadow is projected onto to the floor // range within the entity reflects light from the floor client who has created this entity (when in multiplayer mode) entity skills

(to be continued)

The WMP map format
WMP files are plain ASCII files that can be viewed with any text editor. They contain blocks, entities, and objects that are built of them or of further objects. This is a typical WMB file:
// WMP2 // ACKWED V 4.39 // Created 11.09.2000 blocks 1 entities 2 objects 4 wed { palette { 00 7b 7b 7b 0f 0b 3b 1b 0b 0b 0f 4f 4f 5f 07 8f 0b 7f 53 3f ab 8b abb 73 9f 5f 33 3f db c3 bb 6b 6f 83 7b 2f ff f3 1b 7b ff 2f 2f 7f 2b b0f a7 7b 3b 8b }
0f 8b 17 5b 13 5b 07 3f 0f 4f 1b 6b 2f 9f 1b 8b 9f 4b af 53 cb 5f 67 2b ef 6b 0b 2f 3b c3 b7 b3
0f 8b 0f 5b 07 3f 1b 4b 17 4f 13 5f 7f 2f 6b 2b b3 4b 7b 3b df 53 0b 2f 00 4b 9b 00
0f 8b 0b 1f 1b 7f 00 0b 0b 37 8f 37 a7 3b 6f 2f ef 6f 00 1b 37 00
1f 9b 1f 63 1b 63 0b 77 3b af 2b 9b abf 53 5f 23 db 5b 13 2f 4b cf c7 d7
1f 9b 17 4b 1b 63 0b 53 1f 6b 5f 23 a3 3f cb 2f c3 00
1f 9b 0b 1f 27 8b 00 0f 0f 2f 0f 2f 83 2b 9b 00 df 5f 00 2b 37 00
2f ab 27 6b 27 6b 13 4b 1f 5f 2f 83 4b bf 37 a7 8b 3b af 1f cb 4b 1b 2b 5f db e7 ff
2f ab 1b 6b 13 4b 2b 77 2b 7b 67 1f 57 1f 6b 2b b1b 2b 07 7f e3 00
2f ab 0f 1f 00 0b 13 2f 13 5f 7b 23 8b 27 5f 1f 0f 00 cf 4f 00 3b 57 00
3f bb 2f 73 2f 73 1b 8b 57 cf 47 b7 7f 2b 8b 2f a4f 17 bb 3b 6f e3 7f ff
3f bb 2f 73 1b 00 2f 5b 2b 8f 5b 17 4f 2b a7 2b 0f 97 bf f3
3f bb 13 1f 3f a0b 17 2b 1b 6b 6f 1b 6b 1b 7b 1f 0f 00 bf 3f 00 4f ff 93
4b cb 37 7b 37 7b 23 5b 2f 6f 63 df 53 c23 7f 0f ab 2b 2b 1b 7f e7 ab ff
4b cb 2b 5f 37 7b 23 5b 37 5f 2f ab 4b 13 7b 1f 5b 1b 97 1f 2b 1b 17 ab e7 f7
4b cb 4b af 00 0b 00 1b 1f 7b 5f 13 6f 17 4f 13 0b 00 af 2f 07 5f ff c7
5b db 3f 83 3f 83 2b 77 4b aef 63 d3 6b 1b 3f 0b 9b 1b 2f ef d7 ff

5b db 2f 67 3f 83 2b 00 3b cb 3f a3 4b 0b 43 0b 6f 0f 2f 13 1f bf ff ff
5b db 57 bb 07 0b 07 1f 23 1f 2b 8b 57 0b 53 0b 5f 0f 47 0b 9f 1f ff ff
6b eb 4b 8f 47 8b 2f 6b 3f 7f 57 af 7f ff 6f e3 5f 0f 6b 0f 7b 0f 8b 0b 2f 0b a3 f9f
6b eb 37 6f 47 8b 2f 6b 3b fb3 3f 07 3b 0b 4b 0b 2f 0b 27 d5b
6b eb 1b cb 07 0f 2b 1b 4b 07 4b 07 3f 00 8f 0f 0b 8b 00 53
wad { 0 <standard.wad> 1 } bookmark { *waterblue earthtile ; ; ; ; ; } view { xy.1 -0.0 0.0 0.0 1536.0 0.745 3d.327 250.0 -250.0 250.0 1536.0 -0.785398 -0.61548 0.0 xz.1 -0.0 0.0 0.0 1536.0 0.745 yz.603 0.0 0.0 0.0 1536.0 0.745 textures 0 0
object } } block 0 { ( 0.0 0.0 32.0 ) ( 0.0 500.0 32.0 ) ( 32.0 0.0 0.0 ) ( 32.0 0.0 500.0 ) ( 0.0 -32.0 0.0 ) ( 0.0 -32.0 500.0 ( -32.0 0.0 0.0 ) ( -32.0 500.0 0.0 ( 0.0 32.0 0.0 ) ( 500.0 32.0 0.0 ) ( 0.0 0.0 -32.0 ) ( 500.0 0.0 -32.0 { 0000 ndef } } level 0 { <block.$$w> <Palette.raw> 30 0.000 60.000 <test.wdl> 19.60784340 19.60784340 19.60784340 3.92156863 3.92156863 3.92156863 } light 1 { 49.84313583 49.84313583 49.84313583 300.0 } object 0 { (0.0 0.0 0.0) (0.0 0.0 0.0) (1.0 1.0 1.0) object 1 level 0 } object 1 { (0.0 0.0 0.0) (0.0 0.0 0.0) (1.0 1.0 1.0) object 2, 3 } object 2 { (-44.0 4.0 0.0) (0.0 0.0 0.0) (1.0 1.0 1.0) block 0 } object 3 { (80.0 0.0 48.0) (1.570796 0.0 0.0) (1.0 1.0 1.0) light 1 }

( ( ) ) ( )

500.0 0.0 32.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.32.0 500.0 0.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.( 500.0 -32.0 0.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.( -32.0 0.0 500.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.0.0 32.0 500.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.( 0.0 500.0 -32.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.0 0000
All elements of the WMP file are surrounded by winged brackets { }. The WED element has no influence on the level, it just stores the palette of WEDs texture and 3D display (which has only 256 colors) and the WED settings when the level was saved. The important elements are the level properties, the blocks, lights, sounds, entities, and the objects that store the group information.
A4 Programmers Manual Blocks
A4 uses a right-handed XYZ coordinate system with the Z-axis standing upright. If you're not familiar with the reference of "right-handed" to a coordinate system, it basically provides a tactile and visual discription of the mechanics of the system. If you wrap your right hand around the zaxis, with your thumb facing the positive position, and clench your hand, your knuckles will face in the direction of the positive x-axis and your fingertips will face the postitive y-axis. Here's a picture of A4's coordinate system:

^ z | ^ y | / | / |/ --------> x
Blocks are the primary componens of a MAP file. Each block defines a solid or passable convex region. Blocks define this region as the intersection of four or more planes. Each plane is defined by three non-colinear points. These points must go in a clockwise orientation:
1--2-----> | / |/ 3 | V
Each block statement looks like this:
block 0 { ( 0.0 0.0 32.0 ) ( 0.0 500.0 32.0 ) ( 32.0 0.0 0.0 ) ( 32.0 0.0 500.0 ) ( 0.0 -32.0 0.0 ) ( 0.0 -32.0 500.0 ( -32.0 0.0 0.0 ) ( -32.0 500.0 0.0 ( 0.0 32.0 0.0 ) ( 500.0 32.0 0.0 ) ( 0.0 0.0 -32.0 ) ( 500.0 0.0 -32.0 { 0000 ndef } } ( ( ) ) ( ) 500.0 0.0 32.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.32.0 500.0 0.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.( 500.0 -32.0 0.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.( -32.0 0.0 500.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.0.0 32.0 500.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.( 0.0 500.0 -32.0 ) wood 0 1.00 1.00 : 0000 0.0 50.0 ndef ndef 0.0 0000
That's probably just a bit confusing when you first see it. It defines a cube that extends from (-32,32,-32) to (32,32,32). Here's what a single line means:
( 0.0 0.0 32.0 ) ( 0.0 500.0 32.0 ) ( 500.0 0.0 32.0 ) wood 0 1.00 1.00 : 1st point 2nd point 3rd point texture x_offs y_offs angle x_scale y_scale 0000 0.0 50.0 flags ambient albedo ndef ndef 0.future expansion not used yet
Here are more details about those fields: 1st point \ Those three points define a plane, so they must not be colinear. 2nd point > Each plane should only be defined once. 3rd point / Plane normal is oriented towards the cross product of (p1 - p2) and (p3 -p2) Texture - name of the texture. X_offs - texture x-offset in pixels Y_offs - texture y-offset in pixels
Angle - texture rotation angle, in degree X_scale - size of a pixel in x-direction (pixels per quant) Y_scale - size of a pixel in y-direction (pixels per quant) flags - render mode flags for the texture. ambient - brightness value in percent albedo - albedo/fog value in percent Map Properties The following structure contains the global level or map properties:
level 0 { <block.$$w> <Palette.raw> 30 0.000 60.000 <test.wdl> 19.6 19.6 19.6 3.9 3.9 3.9 } interim WAD palette nexus azimuth elevation script fog 1 fog 2 fog 3 fog 4 sun ambient

WAD the name of the interim WAD file that contains all textures used in the level. Palette the name of the palette file. The.RAW format just consists of 768 bytes r,g,b values for the 256 colors. Nexus the nexus value, used for running and publishing the game. Azimuth the horizontal angle of the sun in degrees, counterclockwise beginning with 0 = East. Elevation the vertikal angle of the sun in degrees. Script the WDL script that runs the game Fog 1.4 the four fog colors in RGB percent Sun the strength and colour of the sun light in RGB percent Ambient the strength and colour of the ambient brightness in RGB percent (to be continued) Light and Sound Sources Entities Groups
A wireframe mesh, made of triangles, gives the general shape of a model. 3D vertices define the position of triangles. For each triangle in the wireframe, there will be a corresponding triangle cut from the skin picture. Or, in other words, for each 3D vertex of a triangle that describes a XYZ position, there will be a corresponding 2D vertex positioned that describes a UV position on the skin picture. It is not necessary that the triangle in 3D space and the triangle on the skin have the same shape (in fact, it is not possible for all triangles), but they should have shapes roughly similar, to limit distortion and aliasing. Several animation frames of a model are just several sets of 3D vertex positions. The 2D vertex positions always remain the same. A MDL file contains: - A list of skin textures in 8-bit palettized or 16-bit 5-6-5 format. - A list of skin vertices, that are just the UV position of vertices on the skin texture. - A list of triangles, which describe the general shape of the model. - A list of animation frames. Each frame holds a list of 3D vertices. - A list of bone vertices, which are used for creating the animation frames. MDL file header Once the file header is read, all the other model parts can be found just by calculating their position in the file. Here is the format of the.MDL file header:
typedef float vec3[3]; typedef struct { char version[4]; long final; vec3 scale; vec3 offset; float pad; vec3 eye; long numskins ; long skinwidth; long skinheight; long numverts; long numtris; long numframes; long numskinverts; long flags; long numbones; } mdl_header;

// // // // // // // // // // // // // // //
"MDL3" or "MDL4" not used yet 3D position scale factors. 3D position offset. not used yet. not used yet. number of skin textures width of skin texture; must be a multiple of 2 height of skin texture number of 3d wireframe vertices number of triangles surfaces number of frames number of 2D skin vertice 0 = normal, 1 = terrain model number of bone vertices (MDL4 only, otherwise 0)
The size of this header is 0x54 bytes (84). The "MDL3" format is used by the A4 engine, while the "MDL4" format is used by the A5 engine. After the file header follow the skins, the skin vertices, the triangles, the frames, and finally the bones (future expansion).
A4 Programmers Manual MDL skin format
The model skins are flat pictures that represent the texture that should be applied on the model. There can be more than one skin. You will find the first skin just after the model header, at offset baseskin = 0x54. There are numskins skins to read. Each of these model skins is either in 8-bit palettized (type == 0) or in 16-bit 5-6-5 format (type == 2). The structure is:
typedef byte unsigned char; typedef struct { int skintype; // 0 for 8 bit (bpp == 1), 2 for 16 bit (bpp == 2) byte skin[skinwidth*skinheight*bpp]; // the skin picture } mdl_skin_t;
8 bit skins are a table of bytes, which represent an index in the level palette. If the model is rendered in overlay mode, index 0x00 indicates transparency. 16 bit skins are a table of shorts, which represent a true colour with the upper 5 bits for the red, the middle 6 bits for the green, and the lower 5 bits for the blue component. Green has one bit more because the human eye is more sensitive to green than to other colours. If the model is rendered in overlay mode, colour value 0x0000 indicates transparency. The width of skins should be a multiple of 4, to ensure long word alignement. The skin pictures are usually made of as many pieces as there are independent parts in the model. For instance, for the a player, there may be two pieces that defines the body, and two others that define the gun. MDL skin vertices The list of skin vertices indicates only the position on texture picture, not the 3D position. That's because for a given vertex, the position on skin is constant, while the position in 3D space varies with the animation. The list of skin vertices is made of these structures:

typedef struct { short u; // position, horizontally in range 0.skinwidth-1 short v; // position, vertically in range 0.skinheight-1 } mdl_uvvert_t; mdl_uvvert_t skinverts[numskinverts];
u and v are the pixel position on the skin picture. The skin vertices are stored in a list, that is stored at offset basestverts = baseskin + skinsize. skinsize is the sum of the size of all skin pictures. If they are all 8-bit skins, then skinsize = (4 + skinwidth * skinheight) * numskins. If they are 16-bit skins, then skinsize = (4 + skinwidth * skinheight * 2) * numskins. MDL mesh triangles The model wireframe mesh is made of a set of triangle facets, with vertices at the boundaries. Triangles should all be valid triangles, not degenerates (like points or lines). The triangle face must be pointing to the outside of the model. Only vertex indexes are stored in triangles. Here is the structure of triangles:
typedef struct { short index_xyz[3]; // Index of 3 3D vertices in range 0.numverts short index_uv[3]; // Index of 3 skin vertices in range 0.numskinverts } mdl_triangle_t; mdl_triangle_t triangles[numtris];
At offset basetri = baseverts + numskinverts * sizeof(uvvert_t) in the.MDL file you will find the triangle list. MDL frames A model contains a set of animation frames, which can be used in relation with the behavior of the modeled entity, so as to display it in various postures (walking, attacking, spreading its guts all over the place, etc). Basically the frame contains of vertex positions and normals. Because models can have ten thousands of vertices and hundreds of animation frames, vertex posistion are packed, and vertex normals are indicated by an index in a fixed table, to save disk and memory space. Each frame vertex is defined by a 3D position and a normal for each of the 3D vertices in the model. In the MDL3 format, the vertices are always packed as bytes; in the MDL4 format that is used by the A5 engine they can also be packed as words (unsigned shorts). Therefore the MDL4 format allows more precise animation of huge models, and inbetweening with less distortion.

typedef struct { byte rawposition[3]; // X,Y,Z coordinate, packed on 0.255 byte lightnormalindex; // index of the vertex normal } mdl_trivertxb_t; typedef struct { unsigned short rawposition[3]; // X,Y,Z coordinate, packed on 0.65536 byte lightnormalindex; // index of the vertex normal byte boneindex; // index of the bone this vertex belongs to } mdl_trivertxs_t;
To get the real X coordinate from the packed coordinates, multiply the X coordinate by the X scaling factor, and add the X offset. Both the scaling factor and the offset for all vertices can be found in the mdl_header struct. The formula for calculating the real vertex positions is:
float position[i] = (scale[i] * rawposition[i] ) + offset[i];
The lightnormalindex field is an index to the actual vertex normal vector. This vector is the average of the normal vectors of all the faces that contain this vertex. The normal is necessary to calculate the Gouraud shading of the faces, but actually a crude estimation of the actual vertex normal is sufficient. That's why, to save space and to reduce the number of computations needed, it has been chosen to approximate each vertex normal. The ordinary values of lightnormalindex are comprised between 0 and 161, and directly map into the index of one of the 162 precalculated normal vectors:
float lightnormals[162][3] = { {-0.525725, 0.000000, 0.850650}, {-0.442863, 0.238856, 0.864188}, {-0.295242, 0.000000, 0.955423}, {-0.309017, 0.500000, 0.809017}, {-0.162460, 0.262866, 0.951056}, {0.000000, 0.000000, 1.000000}, {0.000000, 0.850651, 0.525731}, {-0.147621, 0.716567, 0.681718}, {0.147621, 0.716567, 0.681718}, {0.000000, 0.525731, 0.850651}, {0.309017, 0.500000, 0.809017}, {0.525731, 0.000000, 0.850651}, {0.295242, 0.000000, 0.955423}, {0.442863, 0.238856, 0.864188}, {0.162460, 0.262866, 0.951056}, {-0.681718, 0.147621, 0.716567}, {-0.809017, 0.309017, 0.500000}, {-0.587785, 0.425325, 0.688191}, {-0.850651, 0.525731, 0.000000}, {-0.864188, 0.442863, 0.238856}, {-0.716567, 0.681718, 0.147621}, {-0.688191, 0.587785, 0.425325}, {-0.500000, 0.809017, 0.309017}, {-0.238856, 0.864188, 0.442863}, {-0.425325, 0.688191, 0.587785}, {-0.716567, 0.681718, -0.147621}, {-0.500000, 0.809017, -0.309017}, {-0.525731, 0.850651, 0.000000}, {0.000000, 0.850651, -0.525731}, {-0.238856, 0.864188, -0.442863}, {0.000000, 0.955423, -0.295242}, {-0.262866, 0.951056, -0.162460}, {0.000000, 1.000000, 0.000000}, {0.000000, 0.955423, 0.295242}, {-0.262866, 0.951056, 0.162460}, {0.238856, 0.864188, 0.442863}, {0.262866, 0.951056, 0.162460}, {0.500000, 0.809017, 0.309017}, {0.238856, 0.864188, -0.442863}, {0.262866, 0.951056, -0.162460}, {0.500000, 0.809017, -0.309017}, {0.850651, 0.525731, 0.000000}, {0.716567, 0.681718, 0.147621}, {0.716567, 0.681718, -0.147621}, {0.525731, 0.850651, 0.000000}, {0.425325, 0.688191, 0.587785}, {0.864188, 0.442863, 0.238856}, {0.688191, 0.587785, 0.425325},

A whole frame has the following structure:
typedef struct { long type; // 0 for byte-packed positions, and 2 for word-packed positions mdl_trivertx_t bboxmin,bboxmax; // bounding box of the frame char name[16]; // name of frame, used for animation mdl_trivertx_t vertex[numverts]; // array of vertices, either byte or short packed } mdl_frame_t;
The size of each frame is sizeframe = 20 + (numverts+2) * sizeof(mdl_trivertx_t), while mdl_trivertx_t is either mdl_trivertxb_t or mdl_trivertxs_t, depending on whether the type is 0 or 2. In the MDL3 format the type is always 0. The beginning of the frames can be found in the.MDL file at offset baseframes = basetri + numtris * sizeof(mdl_triangle_t). MDL bones This is only for future expansion of the MDL format, and not implemented yet.
Bones are a linked list of 3D vertices that are used for animation in the MDL4 format. Each bone vertex can have a parent, and several childs. If a bone vertex is moved, the childs move with it. If on moving a bone vertex the connection line to his parent rotates, it's childs are rotated likewise about the parent position. If the distance of the bone vertex to its parent changes, the change is added onto the distance between childs and parent. So the movement of the childs is done in a spherical coordinate system, it is a combination of a rotation and a radius change. Each bone vertex has an influence on one or more mesh vertices. The mesh vertices influenced by a bone vertex move the same way as it's childs. If a mesh vertex is influenced by several bone vertices, it is moved by the average of the bone's movement.

2020Classroom Content Development
1. Research Team Prof. Wee Ling Wong, IMSC and Biomedical Engineering Prof. Chris Kyriakakis, Electrical Engineering Eduardo Carriazo Sion Lee
Project Leader: Other Faculty: Research Staff: Graduate Student: 2.
Statement of Project Goals
The goal of the 2020Classroom Content Development for immersive learning environments is to fully explore the delivery of content within applications of immersive technologies to provide teachers and students with experiences that would not otherwise be available. As educators increasingly look to technology to meet the challenges of teaching and learning in a rapidly changing educational environment, the field of interactive visualization illustrating educational concepts through visual, interactive applications, and simulations presents a promising and emerging paradigm for learning and content delivery. Interactive visualization puts students and their teacher into the picture of what they are studying, allowing them to share and manipulate the objects of study. It promises a rich learning experience that engages, excites, and motivates the learning in a dynamic environment. Current technology-enhanced learning programs are based on existing curricula that have been ported to the web or to televised lectures. In most cases students are not challenged to form their own models, nor asked to demonstrate their understanding through meaningful assessments. We view the lack of innovative curriculum that is optimized rather than adapted to a set of technologies, as an opportunity to design content material to be delivered, managed, and accessed in a novel way. 3. Project Role in Support of IMSC Strategic Plan
In the IMSC Strategic Plan, the basic premise behind the IMSC 2020Classroom Immersipresence Project is that immersive technology closely coupled with innovative curriculum design and meaningful assessment will not simply increase efficiency, but it will significantly improve the quality of the learning experience. Our vision is to pioneer a new learning paradigm for collaborative learning among learners in distributed geographic locations. One component to materialize this vision over the next five years is to implement a novel curriculum with content and tools that are specifically designed for this type of environment. The prototype 2020Classroom that we create can be used as a stepping stone for creating an immersive-technology-enabled educational paradigm that shifts the emphasis from a passive dependency on lectures to active learning through peer interaction. BioSIGHT is strategically positioned to develop and test a new paradigm for the application of advanced integrated media technologies to science learning and education. In the course of research on student learning with various representations of scientific phenomena, novel enabling technologies will be 10
developed that will catalyze advances in scientific understanding. As the project evolves, the collection and analysis of human factors data on student learning will provide feedback for the technology research and development. We view this as two complementary processes: research in integrated media systems gives rise to new educational tools and the lessons learned from using these tools provide new directions for the development of more advanced systems. As the BioSIGHT effort makes progress towards the education vision scenario described in Volume One, the use of a game engine platform as a model provides the flexibility to rapidly develop and pilot test prototype learning activities while concurrently presenting an opportunity to the IMSC Research program to identify integration issues that we can transfer to the IMSC software architecture for Immersipresence, as IMSCs components of the software modules matures. One such example is the integration of optimized algorithms for real time animation and rendering of deformable objects that are within the IMSC Software Architecture for Immersipresence being developed by Alex Franois [1-3], as well as real time head related transfer function audio algorithm for movable objects in 3D environments within Immersive Learning Environments [4]. 4. Discussion of Methodology Used

The design methods for immersive learning environments include the activity scenarios (e.g., task(s) that the user has to accomplish); the information scenarios (e.g., background and detailed content to convey and metaphors to use); and the interaction scenarios (e.g., the user interface as well as the way(s) that the user will encounter the task and avenues to resolution and information). This is an iterative process where the, analysis of usability claims and redesign complement HCI theory and guidelines into usability specifications of a prototype. Our approach to 2020Classroom Content Development in immersive learning environments addresses the pedagogy of how learning can be conveyed through games without diminishing content, specifically examining the notion of how students can play with ideas or concepts central to the curriculum. Specifically we will address: How immersive environments can be designed to convey scientific concepts to a novice target audience, What pedagogical issues impact corresponding classroom implementation, and How immersive environments influence the teaching and learning process 5. Short Description of Achievements in Previous Years
April and November 2002: Student usage pilot tests were conducted using tutorials and several learning activities developed in Conitecs A4 game engine framework. We informally observed users in the course of interacting with the learning activities and discovered that despite the familiarity of computer games, our user population was overwhelmed with information, and exhibited a fairly high frustration level of navigating in the context of the game environment, prompting a redesign that was re-evaluated by students. The Exploratory Challenge from BioSIGHTs Immunology module is a computer game-like simulation developed in collaboration with our colleagues at TERC that introduces students to the visual and spatial concepts that must be understood before the learner can successfully 11
visualize the topic [8]. Treated in purely abstract ways, key ideas are presented without explicit reference to the biological subject. We used what has been learned from the realm of educational computer games to present challenges that are self-motivating, and that can easily engage students without requiring lengthy preparations as well as from research literature that identifies certain features of computer games, such as mystery, fantasy, challenge, and reward to name a few, that motivate students. 5a. Detail of Accomplishments During the Past Year

In the activity scenario, consider a study group of students in a Biomedical Engineering course that wish to collaborate to work on class assignments. Their tasks will require that they work together as a team, capture information from their lectures so that they can go back to it at a later time, share high fidelity audio, video, and image content among them, and create content to demonstrate how well they have grasped certain principles. The content we have identified focuses on the electrophysiology of the heart. It involves dynamic phenomena in time and space that are particularly difficult to teach. It offers students an opportunity to discover and explore difficult concepts such as drug design and delivery as well as science and engineering principles as they apply to a complex living system. We have created 3D models, information scenarios, and preliminary interaction scenarios for content assets in cardiovascular physiology using 3D StudioMax. In addition, we have also designed an investigatory module for exploring kinetics and reaction rates in physiological systems. Students will be exploring concepts in basic chemistry, neurophysiology, zero and first order kinetics, integration by approximation, and physiology to prosecute a defendant driving under the influence of alcohol. 6. Other Relevant Work Being Conducted and How this Project is Different
The Electronic Games for Education in Math and Science (E-GEMS) project is one such example of an interdisciplinary team doing research and development on strategies and materials to integrate game-like activities with other forms of classroom learning, as well as the design and use of educational games [9]. Another example is the Logical Journey of the Zoombinis (Broderbund) game is an example of applying insights from informal learning to the application of technology in the class for leaning math [10]. In addition, there are many research groups exploring the use of technologies such as virtual reality as applied to the area of education to create novel educational experiences and environments. Examples of educational virtual reality projects include the Narrative-based Immersive Constructionist/Collaborative Environment (N.I.C.E.), which gives children an authentic activity in building an experimental learning environment; the Round Earth project, which teaches concepts that are counter-intuitive to young learners; the Virtual Explorer, which allow players in a virtual world to explore the human immune system and participate in an immune response on the cellular and molecular level; the Virtual Solar System project and the Virtual Gorilla Project to name a few [11 - 16] 7. Plan for the Next Year
Using focus group and survey methods, we will pilot test the stand-alone cardiovascular materials and investigatory module in the fall 2003 with sophomore Biomedical Engineering students to identify shortcomings to the activity and information scenarios. Findings and 12

recommendations will feed back into our design process to be reflected in subsequent revisions that can be implemented into the 2020Classroom Immersipresence Project. In addition, we will continue to develop information scenarios for pharmacokinetics and the cardiovascular system. In collaboration with Prof. Chris Kyriakakis, the challenge that we present to the IMSC Research Program is how to use auditory cues to augment a 3D immersive experience. The majority of the visual cues that we obtain assist in the determination of depth in a 3D environment. We explore the notion of sound associated with interactive and dynamic elements in a 3D environment to enhance the visual cues in the perception of depth. To provide true spatial auditory cues requires integrating optimized audio algorithms within the 3D simulation engine capable of synthesizing head-related transfer functions with respect to the dynamic objects in the 3D environment in real time in what we call volumetric audio rendering. Novel ways to render HRTF-based sound over multiple loudspeakers will be investigated. 8. 9. Expected Milestones and Deliverables User studies for content appropriateness Specifications for mapping content and interaction into functional software through a game engine for functional interface and navigation in immersive environments Member Company Benefits
Not Applicable 10. References
[1] Franois, A. and G. Medioni. A Modular Middleware Flow Scheduling Framework in Proceedings of ACM Multimedia 2000, pp. 371-374, Los Angeles, CA November 2000. [2] Franois, A. and G. Medioni. A Modular Software Architecture for Real-time Video Processing in Proceedings of International Workshop on Computer Vision Systems, Vancouver BC, Canada, July 2001. [3] Franois, A. Modular Flow Scheduling Middleware. http://mfsm.SourceForge.net [4] Immersive Audio Project Report in IMSC Volume 2 Project Reports, 2002. [5] 3D GameStudio A4 Game Engine, Conitec Corporation, http://www.conitec.net/a4info.htm. [6] DeLeon, V. and R. Berry. Bringing VR to the Desktop: Are you game? IEEE Multimedia April-June, 68 72, 2000. [7] Prototype Immunology project: http://biosight.usc.edu/research/projects/immunology_prototype/index.htm [8] TERC: a non-profit education research organization whose mission is to improve mathematics, science, and technology teaching and learning. http://www.terc.edu [9] Electronic Games for Education in Math and Science (E-GEMS), University of British Columbia, Ontario: http://taz.cs.ubc.ca/egems/home.html [10] Logical Journey of Zoombinis, Broderbund, 1996. [11] N.I.C.E, U. Chicago at Illinois: http://www.ice.eecs.uic.edu/~nice [12] Round Earth project, U. Chicago at Illinois: http://www.evl.uic.edu/roundearth; 13