Projector Camera Applications: Shooting Range Simulation
M.Tech Project 2nd Stage Report January 9th, 2006 Submitted in partial fulllment of the requirements for the degree of M.Tech. by Jhunjhunwala Rohit K Roll No: 04305803 under the guidance of Prof. Sharat Chandran
Department of Computer Science and Engineering Indian Institute of Technology, Bombay Mumbai
Chairman: Prof. M. Sohoni Examiner: Prof. S. N. Merchant Venue: 226, EE (old) Date: 18th January 2006
Acknowledgement January 9,2006
I would like to take this opportunity to thank my guide, Prof. Mr. Sharat Chandran, for his valuable insights and for providing me with directions especially when I found it dicult to move forward.
Jhunjhunwala Rohit K 04305803
Abstract With the tremendous boost in the processing power of computers in recent times coupled with drastic fall in their cost, it has become possible to process large amount of videos in real-time. Imaging devices like projectors and cameras have become a common accessory for personal computer desktops. These devices combined with even low-end modern processors facilitates intelligent video processing, using which, a system can be built for human activity recognition, for learning in robotics, for virtual reality based systems or for self-calibrating camera projector system. A camera projector system is one such domain where the feedback from camera makes the system aware of the current state of its environment. This report captures the use of such a camera projector system for simulation of a Shooting Range. A live virtual scene and laser guns will make the simulation complete. We discuss the general approach to developing such a system and the challenges faced.

Contents

1 Introduction 1.1 Motivation. 1.2 Challenges. 1.2.1 Geometric Issues. 1.2.2 Photometric Issues 1.3 Shooting Range Simulator 1.3.1 Aim of Our Project 1.4 Outline Of The Report. 16 16
2 Projector Camera Systems 2.1 A Simple Projector Camera System. 2.2 Camera Model. 2.2.1 The Basic Pinhole Camera. 2.2.2 Camera Rotation and Translation. 2.3 Camera Calibration. 2.3.1 Camera Knowledge. 2.3.2 Characteristics of projector based system 2.4 Screen-Camera Homography. 2.5 Steps for Determining Homography. 3 Shooting Range Simulation 3.1 Shooting Range. 3.1.1 A Virtual Environment. 3.1.2 Soldier. 3.1.3 Computer Based Simulator. 3.1.4 Display. 3.1.5 Projector and Camera. 3.2 Previous Work. 3.3 Scope of this project. 3.3.1 Problem Statement.
4 System Implementation 17 4.1 Approach to the problem. 17 ii
4.2 Overview of working of the system 4.3 Current System. 4.3.1 Problems and Solutions. 4.3.2 Algorithm. 4.4 Design for next stage of the system 4.4.1 Problems and Solutions. 4.4.2 Algorithm. 4.5 Future Work. 4.6 Conclusion.

List of Figures

1.1 A snapshot of scene from Game: Hidden & dangerous. 5
2.1 Projector-Camera System. 7 2.2 Perspective Projection. 8 2.3 Left: the relationships between the three frames of reference corresponding to the computer display (source image frame), camera (camera image frame) and projection screen (projected image frame). Note that the last can only be indirectly observed by our system, through the camera. T is obtained using the calibration method; C is obtained by locating the screen within the camera image. Finally, the mapping responsible for the key-stoning distortion, is given by P = C1T. Right: the application image can be appropriately distorted (pre-warped), using the mapping W so that it appears rectilinear after projection through a misaligned projector (modeled by the mapping P). 10 3.1 3.2 3.3 3.4 A snapshot from a shooting range used for practice. A snapshot from the game Hidden and Dangerous. A simple block diagram of a shooting range simulation system. Snapshot from Game: Delta force. 15 15

Chapter 1 Introduction

Projection is the most simple and economical way of producing large displays. Today, the cost of portable projectors are going down while the resolution oered by them is soaring high. In recent years, advances in portable projectors have made possible the development of novel projected displays like multi-planar displays [1], large multi-projector walls [5], steerable camera-projector systems [3], immersive environments [11], intelligent presentation systems [14], intelligent oce environments [6]. The eorts involved in manually calibrating multiple projectors to each other and aligning projected displays to physical surfaces is the driving force behind bringing research interest in projector-camera systems, where techniques adopted from multi-view geometry are applied to a collection of projectors and cameras. A system for multi-planar display can be used in oce like environments. An oce desk pushed against the wall can be used to display the elevation and plan of a map simultaneously. Continuous and rapid improvements in CPU performance, storage density and network bandwidth have provided sucient bandwidth and computational resources to support high-resolution displays and natural human computer interactions. With the modern advances in technology, the main bottleneck in an interactive computer system occurs in the link between computer and human, not between computer components within the system.

Motivation

Recently there has been an explosion of interest in systems which combine projection technology with computer vision. A characteristic of these systems is their ability to passively sense an environment in support of real-time control of projected light. Research in this area spans a number of disciplines including computer vision, computer graphics, HCI and display technologies. In particular, the theory and techniques used by researchers in the area are related, sometimes complementary to traditional computer vision techniques Large format display deemployed in stereo-camera and gesture recognition systems. vices, like projectors and at panels, are day by day becoming commodity items. New technologies like Organic Light Emitting Diodes(OLED) will soon be available at inex2
pensive prices [5]. The coupling of an imaging device (a camera) to a projector makes the assembly aware of its environment and as lower price technologies develop, the use of such assemblies will spread to everyday life and any man-made surface will become a potential display screen.

Challenges

Geometric Issues
For an ideal display from the projector, the projector must be perfectly aligned to the display surface such that the axis of projection is perpendicular to surface of the display screen. The geometric problems relate to computation of the spatial correspondences between pixels in the projectors and the projected display on the screen. The projectors should be accurately and automatically calibrated to the screen, to the camera and to each other. The calibration should enable the images in each projector to be pre-warped so as to create a desired projected display that is aligned with the screen. Achieving the accuracy of the order of a pixel or higher is a challenge in Projector Camera Systems. The devices used in the system, like projector(s) and camera(s), are prone to mechanical drifts. Modern portable projectors, which are usually mounted right before use, can drift during the process of display due to reasons like a push to the table. These kind of changes are not expected beforehand and the system has to be robust to such geometric deformations in the environment. The system has to re-initialize its knowledge of the environments geometry and this has to be as transparent to the user as possible. In systems for display on multi-planar surfaces, accurate localization of specic geometric features, such as the edge that denes the intersection of two planar surfaces, are critical for good performance. The increase in number of display planes and projectors leads to increased complexity in geometry of the scene. Also, rendering of scene becomes complicated.

Photometric Issues

Image intensity and color is another major problem to a projector system. The problem of non-uniform image intensity exists for imagery of individual projector. This problem becomes more predominant when two projector images overlap. The overlap from multi-projector results in brighter region in the overlapping area. Sometimes, a part of one display surface reects light onto another part, giving rise to change in the intensity (inter-reection). This problem is important because human are quite sensitive to such variations: we can perceive as little as 2% variation in brightness and a 2nm variation in wavelength.
For projected imagery there are essentially three relevant factors: 1. Nature of the light source: Factors like spatial resolution, visibility of surface, angle of incidence with a surface, distance, and polarization come from the nature of the light source. 2. Surface properties: Surface factors include texture, gain, reectance, persistence, polarization, and orientation. 3. Human perception: Human perception of light varies with wavelength, intensity, viewer distance to the surface, and image resolution. All these factors make the use of Projector Camera System non-trivial. Even in a simple Projector Camera System, the issues related to the technical properties of projector and camera are of big concern. A display pattern sent to projector is not ever same as the one captured by the camera. This is because of light transformations encountered while going from the projector to the screen and from screen to the camera. The system has no way of predicting these transformations. A normal user of the system is not expected to provide the system with all the technical details of the hardware and neither are the details that are provided by manufacturers expected to be accurate in real world. The camera(s) is the only input interface of the system to the world, and is even more instable during dynamic changes in the lighting conditions of its environment. Even slight change in the lighting conditions due to windows, shadows or even wind can make the system see the entire image dierent from what it expects. The system has to adjust to these situations in real-time. In a Projector Camera based presentation system, the photometric issues are the accurate and fast computation of the desired pixel intensities in each projector so as to eliminate shadows and suppress occluder illumination. This involves occlusion detection based on camera input and correctly adapting the projector output to achieve the necessary goals.

Shooting Range Simulator

A shooting range is a controlled environment used for practicing and testing shooting skills of shooter. Shooter is asked to shoot at a pre-specied target at farther range. From dierence between actual target and point hit by shooter, accuracy of shooting is determined. A Projector Camera system can be used for simulating the same, reducing the cost of training, and the toxic wastes produced by shooting with real ammunition.

Aim of Our Project

The aim of our project is to create a virtual environment for simulation of a shooting range, which can test the shooting skills of its user. A virtual environment will be created and rendered as First Person View (FPV). For example, please refer gure 1.1. 4
This rendered scene will be displayed on a display surface, which can even be multi planar. Shooter will shoot target displayed on screen. The camera will detect the hit and the simulator system will convert the point as seen by camera to corresponding point in the image. Thus, the system will nd accuracy of shooting. To support display on multi-planar surface, we take help of homography between small fragments of the display surface and the projector.
Figure 1.1: A snapshot of scene from Game: Hidden & dangerous

Outline Of The Report

The rest of the report is organized as follows. In chapter 2, a simple Projector Camera System is discussed in greater detail. In chapter 3, the problem statement of this project has been presented with general challenges involved. In chapter 4, the implementation of basic laser tracking and challenges involved in the next step of implementation are discussed. The scope of this project has been discussed thereafter.
Chapter 2 Projector Camera Systems
A projector and a camera are both inverse of each other. A projector is used to generate lighting patterns, while a camera is used to capture them. When these devices are combined into one system, they form a loop in the system. The projector displays some image on the screen while the camera works as a feedback agent, telling the system how does the projection look from the viewpoint of the user. This provides the system with a sense of intelligence to improve its display and make the environment and user interface of the system more comfortable to its users. For such a system to work it must have at least one projector and one camera. Use of multiple projectors is required to achieve increase in projection size, increase in brightness(intensity), increase in the sharpness(resolution) of the display or to avoid the situation when some part of the display area is occluded by some moving object(like a speaker giving presentation). Use of multiple cameras is to facilitate the scenario where every part of the surrounding is always visible to at least one camera, to gain geometry of a real world object from multiple viewpoints and to track dynamic changes(or events) in dierent directions when the display is immersive. The power of feedback loop makes a Projector Camera System intelligent. Such a system is being used for a lot of educational and research purposes and is proposed to become a common utility for a normal oce employee or even a household in near future. Some applications where such system is being used include display on multi-planar surfaces, large format displays, annotate real world objects. All of these were discussed in the previous stage [8].

A Simple Projector Camera System
A simple Projector Camera system shown in [Figure 2.1] consists of 3 components: 1. Projector(s) to project the pattern (image) on the screen 2. Camera(s) facing towards the screen to capture the pattern (image) 6
3. A (planar / non-planar) screen i.e. display surface

Z Screen X

Y Z Camera Y Projector(s) X Z

Camera X

Figure 2.1: Projector-Camera System The main challenge in building the assembly of Projector Camera System is calibration of projector, display screen and the camera. For this, one needs to nd the geometric relation between the pairs screen and projector & screen and camera. This gives the geometric relation between projector and camera, which is dicult to acquire by itself. Then we give some simple models to acquire the geometry of the surrounding and calibrate the assembly.

Camera Model

A camera is a mapping between the 3D world (object space) and a 2D image. In this section, we will start with the simplest camera model, which is basic pinhole camera.

The Basic Pinhole Camera

Consider a simple case where the projection center is placed at the origin of the world frame and the image plane is the plane Z = f. The projection process can then be modeled as follows: fY fX y= x= Z Z For a world point (X, Y, Z) and the corresponding image point (x, y). Using the homogeneous representation of the points a linear projection equation is obtained as:

f 0 fx fy = 0 f f 7

X Y Z 1

Optical Axis

Figure 2.2: Perspective Projection This projection is illustrated in Figure 2.2. The optical axis passes through the center of projection C and is orthogonal to the retinal plane R. Its intersection with the retinal plane is dened as the principal point c.
Camera Rotation and Translation
Motion of scene points can be modeled as follows
r11 r12 r13 tx R t r21 r22 r23 ty M = M r31 r32 r33 tZ 1 0
with R as a rotation matrix and t = [tx ty tz ] as a translation vector.

Camera Calibration

In this section we will see calibration technique in brief.

Camera Knowledge

Knowledge about the camera can be used to restrict the ambiguity on the reconstruction from projective to metric or even beyond. Dierent parameters of the camera can be known. Both knowledge about the extrinsic parameters (i.e. position and orientation) as the intrinsic parameters can be used for calibration. 8
Interior Orientation Basic camera calibration is the recovery of the principle distance f and the principle point (x0 , y0 )T in the image plane or, equivalently, recovery of the position of the center of projection (x0 , y0 , f )T in the image coordinate system. This is referred to as interior orientation in photogrammetry. Interior orientation is the relationship between cameracentric coordinates and image coordinates. The camera coordinate system has its origin at the center of projection, its z axis along the optical axis, and its x and y axes parallel to the x and y axes of the image. Camera coordinates and image coordinates are related by the perspective projection equations: xI x 0 xC yI y 0 yC = and = f zC f zC (2.3)

where f is the principle distance (distance from the center of projection to the image plane), and (x0 , y0 ) is the principle point (foot of the perpendicular from the center of projection to the image plane). That is, the center of projection is at (x0 , y0 , f )T , as measured in the image coordinate system. Interior orientation has three degrees of freedom. The problem of interior orientation is the recovery of x0 , y0 , and f. This is the basic task of camera calibration. However, as indicated above, in practice we also need to recover the position and altitude of the calibration target in the camera coordinate system. Exterior Orientation A calibration target can be imaged to provide correspondences between points in the image and points in space. It is, however, generally impractical to position the calibration target accurately with respect to the camera coordinate system using only mechanical means. As a result, the relationship between the target coordinate system and the camera coordinate system typically also needs to be recovered from the correspondences. This is referred to as exterior orientation in photogrammetry.
Characteristics of projector based system
Projector based display systems are so impressive that they are now a days used everywhere. They oer attractive combination of dense pixels over wider area. Some of important characteristics of projector based display system are : 1. Size of projector: The size of the projector can be much smaller than the size of the image it produces. 2. Size of displayed image: The image generated are larger in size than their CRT and LCD counterparts. 3. Multi-projector Display: Overlapping images from multiple projectors can be eectively superimposed on the display surface. 9
4. Blending of Heterogeneous images: Images from projectors with quite dierent specications and form factors can be easily blended together. 5. Display surface: The display surface does not need to be planar or rigid, allowing us to augment many types of surfaces and merge projected images with the real world.

Screen-Camera Homography

The relation between the screen and camera world can be represented by a homography given by a matrix. As only the camera images are available to us, it is important to accurately estimate this screen-to-camera homography if we want to obtain geometrical parameters between the screen and the projector. The screen-camera homography can be easily computed if there are markers (or points of known coordinates on the screen). Authors in [14] use the fact that screen for presentation is usually square. The basic assumptions are that: 1. The positions, orientation and optical parameters of both camera and projector are unknown. 2. Camera and projector optics can be modeled by perspective transforms. 3. The projection screen is at.

Figure 2.3: Left: the relationships between the three frames of reference corresponding to the computer display (source image frame), camera (camera image frame) and projection screen (projected image frame). Note that the last can only be indirectly observed by our system, through the camera. T is obtained using the calibration method; C is obtained by locating the screen within the camera image. Finally, the mapping responsible for the key-stoning distortion, is given by P = C1T. Right: the application image can be appropriately distorted (pre-warped), using the mapping W so that it appears rectilinear after projection through a misaligned projector (modeled by the mapping P). 10
Consider a point (x, y) in the projector slide. This point is projected by perspective transform to some unknown point on the projection screen. The parameters of this perspective projection depend on the unknown position and orientation of the projector relative to the screen, and the unknown focal length of the projector optics. The point on the screen is then observed by a camera at pixel location (X, Y ). The parameters of the second perspective projection depend on the unknown position and orientation of the camera relative to the screen, and the unknown focal length of the camera optics. We need to nd the mapping between (x, y) and (X, Y ) without any additional information about the unknowns; in other words, given that we observe a feature at (X, Y ) in the camera image, we would like to know the point (x, y) in the projector slide that corresponds to this feature. (e.g. Given a bright laser pointer hit in camera image we need to determine whether the laser pointer has hit the specied target in the actual image?). This mapping appears to be impossible to determine in the presence of so many unknowns. But, as all of the observed points in the scene lie on some unknown plane (the at projection screen), we can establish a homography between the camera and projector frames of reference. Thus, we can show that the compounded transforms mapping (x, y) in the projector frame to some unknown point on the projection screen, and then to pixel (X, Y ) in the camera frame, can be expressed by a single projective transform expressed in homogeneous equation as,
X p1 p2 p3 xw yw = p p p Y 4 1 p7 p8 p9 w
with eight degrees of freedom, = (p1. p9 )T constrained by || = 1. can be deterp p p 1 mined from as few as four pixel correspondences. When more than four correspondences are available, the system nds the best estimate in a least-squares sense. Given n feature point matches,{(xi , yi ), (Xi , Yi )}, let A be the following 2n 9 matrix:

X1 Y 0 X2 Y 0. Xn Y n 0 0

X 1 Y0 X 2 Y2. X n Yn

X1 x1 X1 x1 X2 x2 X2 x2.

0 Xn xn 1 Xn xn

Y1 y1 x1 Y1 y1 y1 Y2 y2 x2 Y2 y2 y2 . . . Yn yn xn Yn yn yn
The goal is to nd the unit vector that minimizes |A|, and this is given by the p p T eigenvector corresponding to the smallest eigenvalue of A A.
Four correspondences are sucient provided that no three points are collinear
Steps for Determining Homography
The Projector Camera system oers a signicant advantage in determining these pixel correspondences: 1. The projector displays calibration slides to recover the projector-camera mapping parameters. 2. An initial estimate is obtained by projecting an image with a bright rectangle against a contrasting background. 3. The locations of the corners of the projected rectangle (in the camera image) are determined by standard image processing techniques and used as features to recover the projector-camera homography according to the procedure outlined above. 4. This homography is applied (as a pre-warp) to a calibration slides consisting of bright circles on a dark background. 5. The system estimates the center of each circle by calculating the centroid of the observed bright region in the camera image, and uses this sub-pixel estimate to rene its estimate of the projector-camera homography. Robust feature extraction is made simpler by the calculated homography since the system knows where and when to look for a feature in the camera image. The system can also monitor for any dynamic deformation in the projected image to determine when the mapping is no longer accurate and trigger automatically recalibration as necessary. As we have seen that the homography between the camera and the projection screen has to be calculated and hence the limitation arises. The above method cant be used if: 1. An ordinary wall or oor is used whose surface has no clue. 2. The boundary of projector screen can not be determined due to light conditions. To resolve this by simple means we can put xed markers at the screen corners and use the camera to nd the screen corners. This will give us the location of screen in camera frame and thus the screen to camera homography. It might seem dicult to determine the screen-camera homography. If the projector is calibrated and its projection pattern is known, the screen-camera homography can be determined [1].
Chapter 3 Shooting Range Simulation
A shooting range is used for practicing and testing shooting skills of shooter. A snapshot of shooting range is shown in g 3.1. The shooter is asked to shoot a pre-specied target at farther range with gun. The bullet hit by the shooter is compared with the actual target. From the dierence between actual target and point hit by shooter, accuracy of shooting can be determined. In real shooting range, we need to deal with cost of ammunition and most importantly toxic wastage produced by ammunition. Hence, there is need for a virtual shooting range practice environment. The standard simulation equipment like head mounted display (HMD) are not cost eective. Projector can provide large display in low cost and camera can be used for determining accuracy of shooting. Hence, a projector-camera system can be used for shooting range simulation. A virtual environment can be created and rendered as First Person View (FPV) 3.2.

Scope of this project

The main aim of this project is to develop a basic framework which can be used by simple simulation engines to generate a Shooting Range Simulation. The basic framework will provide the seamless display on single or multi-planar surface given a image and will detect the laser shots and return the coordinates of shot points to the simulation engine in the given image frame. Multiple Game like applications can be developed using this kind of service, The image processing part for these systems will be limited to generation and updating of environment only, in presence of the said framework.

Problem Statement

A basic framework to provide a seamless display and detect all shots red with laser on the scene. The system will take an image as input, transform and render it for pleasant display, detect all laser shots red and return the coordinates of those shots in the frame of input image.
Chapter 4 System Implementation
4.1 Approach to the problem
The laser point on the screen is captured by a video camera, and its location recognized by image processing techniques. Kirstein and Miller have user laser point detection in [9] to simulate presence of mouse in a presentation environment. We need to dierentiate between multiple soldiers shooting the same target at the same time. For this we can 1. Do a polling of the infrared guns in a time-division format to know who shot and when. 2. Put a camera with screen facing shooters and detect shots red and the orientation of their guns using image processing techniques. 3. Have the gun send an interrupt to the system whenever its trigger is pressed and look for shot target at that time. To detect infrared shots, we can use an infrared camera or a normal camera with a infrared lter attached to its lens.
Overview of working of the system
1. Do the pre-processing as described in 2.5 and compute H and H1. 2. Create the Virtual Environment in the computer. 3. Render the scene from a rst person view (FPV). 4. Pre-warp the scene by pre-computed H1 , start projecting scene, which will be almost free of distortion, irrespective of projector position. 5. Find the background intensity from rst frame of each shot. 6. Subtract background intensity model from camera captured frames. In the ideal scenario dierence should be zero. 17
7. Position the soldier (Shooter ) in front of display surface, facing the screen. 8. Ask shooter to shoot the specied target on screen by laser-gun ( Gun with laser pointer attached). The Laser point will help us detect the hit. 9. If laser pointer is displayed on screen, we will get change in the color value in camera image. The dierence between current camera image and reference background intensity model will be maximum where laser dot is visible. 10. Find the region with high value region and mark its centroid as laser pointer position. 11. After detecting the laser pointer position, compute the distance between actual target coordinate and detected hit coordinate for nding accuracy of shooting.

Current System

The current state of the system does laser shot detection for single red light laser dot. The laser is extracted from the captured image based on color identication and blob properties of the laser dot in the image. The system successfully tracks the laser detecting individual shots and logs them. Multiple candidates are detected during the captured image analysis. The detection then is rened by using laser properties like the color at center of dot, the size of laser for given camera-screen position, the intensity of laser, etc.

Problems and Solutions

The problems faced until the current stage of implementation and their solutions are discussed below. The color of laser dot is varying from laser to laser and thus it is dicult to look for a general color. If we try to customize depending on the laser being used, the system will need to be trained for each new laser. Here we use the fact that the laser forms a set of rings on the screen of dierent colors. For a specic color range, the system will always detect some number of these rings in almost all lasers. This(ese) rings are then made more complete by closing the mask with standard structuring element. Theoretically, the system will detect portions of the image with color and size similar to laser as laser dots. With the assumption than the movement of image from projector to screen, brings down the intensity factor of the entire image, we are assured that even if the image has regions of laser color, the laser detection will not even consider them as possible candidates. 18
In case the projector is used at a (very) high intensity causing the image parts with laser color to be captured as bright laser color, the system will detect them as lasers. This kind of confusion will not arise between laser and image parts because the system can distinguish them based on the knowledge of projected image. Here, we assume that the camera view is clear i.e. the confusion is not because of some unknown object between camera and screen. In case of high probability of laser(red) color in the projected image, the blob analysis method will detect many blobs and each of them will need to be processed, even if the laser dot is found in the rst blob. This can be solved by running the system with all possible laser and light conditions to nd the extremes and most probable color/size of laser dot. This information then can be used to discard furthur processing of the image after a match is found. Inter - Reection is the property of light illuminating a surface after being reected from a nearby surface. If on a multi-planar geometry of screen, the laser fall on the corner between two surfaces, the reected light on the nearby surface gives the same color as the ring around the laser dot. This is solved by the fact that for a laser(red) dot, the center of the dot is always very intense and represents white in the captured image because of camera saturation. Whereas, in case of inter-reection, the reection of laser will always be a more uniform laser(red) blob.

Algorithm

The current algorithm for the laser detection is as follows: 1. The image is retrieved from the captured video, frame by frame. 2. Laser Detection procedure is called for each of these frames. 3. The laser detection procedure nds regions in the image falling under specied color range, generating a mask. 4. The mask is then closed to ll empty regions of the blob. 5. The Blob analysis routine is then called to detect possible candidates for laser dot. 6. Each blob is processed independently to nd the best match. Currently the blob with size closest to the laser dot size, specied in the system settings, is identied as the laser dot. The system is designed to discard candidates based on similarity with original image. 19
For a red laser, the fact that the center of the laser dot is intense white, is also considered. 7. Everytime a new blob with size closer to laser dot is found, the center of blob is taken to be laser shot location. 8. The system scans the entire image and then logs the nal laser dot. 9. the current system display a rectangle following the last known location of laser in a separate tracking window.
Design for next stage of the system
The next stage of the system implementation is to integrate the laser detection module with the homography calculation. The system will project images on screen and detect laser shots in the original image cordinate system instead of camera cordinates that it does now.
The problems faced in the design for the next stage and their solutions are discussed below.

1. 2. 3. 4.

The algorithm that will be followed for this implementation is as follows:

Future Work

The system will need to be extended to support multiple lasers and a dynamic scene generation. Support for multiple laser tagging and tracking has to be incorporated followed by use of a sample game engine to render a natural scene for the user to shoot in. 20

Conclusion

We have seen an implementation of laser detection. The system successfully tracks a single laser dot across the camera view. The design for implementation to support multiple lasers in projected video is also discussed. The next stage of implementation will involve merging of the tracking module with the seamless display module(multi-planar using homography) followed by support for multiple lasers and real time game like scene generation. []

Bibliography

[1] Mark Ashdown, Matthew Flagg, Rahul Sukthankar, and James M. Rehg. A exible camera projector system for multi-planar displays. In Computer Vision and Pattern Recognition, 2004. [2] Mark Ashdown and Rahul Sukthankar. Robust calibration of camera-projector system for multi-planar displays. Technical report, University of Cambridge, HP Labs, Carnegie Mellon University, January 2003. [3] Andreas Butz and Christian Schmitz. Annotating real world objects using a steerable projector-camera unit. In Computer Vision and Pattern Recognition, 2005. [4] C. Frueh and A. Zakhor. Capturing 2d depth and texture of time-varying scenes using structured infrared light. In IEEE International Workshop on Projector-Camera Systems, June 2005. [5] Thomas Funkhouser and Kai Li. Guest Editors introduction: Large-format displays. IEEE Computer Graphics and Applications, 20(4):2021, July/August 2000. [6] D. Hall, C. Le Gal, J. Martin, O. Chomat, T. Kapuscinski, and J. Crowley. Magicboard: A contribution to an intelligent oce environment, 1999. [7] Nilesh Heda. Projector-camera based solution for simulation system. Technical report, Department Of Computer Science, Indian Institure of Technology, Bombay, July 2004. [8] Jhunjhunwala Rohit K. Projector camera systems. Technical report, Department Of Computer Science, Indian Institure of Technology, Bombay, July 2005. [9] Carsten Kirstein and Heinrich Mueller. Interaction with a projection screen using a camera-tracked laser pointer. In MMM 98: Proceedings of the 1998 Conference on MultiMedia Modeling, page 191, Washington, DC, USA, 1998. IEEE Computer Society. [10] C. Liu, Y. Zhang, R. Xiong, and Y. Sun. Method of computer-aided measurement in a shooting range. In Proc. SPIE Vol. 4220, p. 226-230, Advanced Photonic Sensors: Technology and Applications, Jinfa Tang; Chao-Nan Xu; Haizhang Li; Eds., pages 226230, October 2000. 22
[11] Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. The oce of the future: A unied approach to image-based modeling and spatially immersive displays. Computer Graphics, 32(Annual Conference Series):179 188, 1998. [12] Peng Song and Tat-Jen Cham. A theory for photometric self-calibration of multiple overlapping projectors and cameras. In IEEE International Workshop on ProjectorCamera Systems, June 2005. [13] R.M. Steele, S. Webb, and C. Jaynes. Monitoring and correction of geometric distortion in projected displays. In Central European Conference on Computer Vision, Visualization, and Graphics, page 429, February 2001. [14] R. Sukthankar, R. Stockton, and M. Mullin. Smarter presentations: Exploiting homography in camera-projector systems. In International Conference on Computer Vision, 2001. [15] Rahul Sukthankar, Tat-Jen Cham, and Gita Sukthankar. Dynamic shadow elimination for multi-projector displays. In Proceedings of IEEE Computer Conference on Computer Vision and Pattern Recognition, 2001. [16] Rahul Sukthankar, Tat-Jen Cham, Gita Sukthankar, and James M. Rehg. Shadow elimination and occluder light suppression for multi-projector displays. In Proceedings of IEEE Computer Conference on Computer Vision and Pattern Recognition, 2001. [17] Tsai and Roger Y. An ecient and accurate camera calibration technique for 3d machine vision, 1986. [18] Tsai and Roger Y. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using o-the-shelf tv cameras and lenses, 1987.

Camera Projector Systems

M.Tech Project 1st Stage Report July 17, 2005 Submitted in partial fulllment of the requirements for the degree of M.Tech. by Jhunjhunwala Rohit K Roll No: 04305803
under the guidance of Prof. Sharat Chandran
Department of Computer Science and Engineering Indian Institute of Technology, Bombay Mumbai
Chairman: Prof. Sharat Chandran Examiner: Prof. Sharat Chandran Room: 238, EE Date: 3rd August 2005
Acknowledgement July 18,2005
I would like to take this opportunity to thank my guide, Prof. Mr. Sharat Chandran, for his valuable insights and for providing me with directions especially when I found it dicult to move forward.
Jhunjhunwala Rohit K 04305803
Abstract With the tremendous boost in the processing power of computers in recent times coupled with drastic fall in their cost, it has become possible to process large amount of videos in real-time. Imaging devices like projectors and cameras have become a common accessory for personal computer desktops. These devices combined with even low-end modern processors facilitates intelligent video processing, using which, a system can be built for human activity recognition, for learning in robotics, for virtual reality based systems or for self-calibrating camera projector system. A camera projector system is one such domain where the feedback from camera makes the system aware of the current state of its environment. This report captures the overall domain of camera projector systems applications and various challenges that are part of it. The general approach is to make multiple projectors and camera work together to understand the environment and improve the projected video in real-time.

Contents

1 Introduction 1.1 Motivation. 1.2 Challenges. 1.2.1 Geometric Issues. 1.2.2 Photometric Issues 1.3 Shooting Range Simulator 1.3.1 Aim of Our Project 1.4 Outline Of The Report. 18 18
2 Projector Camera Systems 2.1 A Simple Projector Camera System. 2.2 Camera Model. 2.2.1 The Basic Pinhole Camera. 2.2.2 Charge-coupled Device(CCD) Camera. 2.2.3 Camera Rotation and Translation. 2.2.4 Projection Matrix. 2.3 Camera Calibration. 2.3.1 Camera Knowledge. 2.3.2 Characteristics of projector based system 2.4 Screen-Camera Homography. 2.5 Steps for Determining Homography.
3 Projector Camera Systems Application 3.1 Multi-Planar Display Surface. 3.2 Large Display Screens. 3.3 Annotation of Real-World Objects. 3.4 Capturing Depth and Texture of Time-Varying Scenes 3.5 Shadow Removal in Presentation Systems.
4 Shooting Range Simulation 20 4.1 Shooting Range. 21 4.1.1 A Virtual Environment. 21 4.1.2 Soldier. 22 ii
4.1.3 Computer Based Simulator. 4.1.4 Display. 4.1.5 Projector and Camera. 4.2 Previous Work. 4.3 Scope of this project. 4.3.1 Problem Statement. 4.4 Approach to the problem. 4.4.1 Working of the system.

List of Figures

1.1 A snapshot of scene from Game: Hidden & dangerous. 2.1 2.2 2.3 2.4 Projector-Camera System. Perspective Projection. From retinal coordinates to image coordinates. Left: the relationships between the three frames of reference corresponding to the computer display (source image frame), camera (camera image frame) and projection screen (projected image frame). Note that the last can only be indirectly observed by our system, through the camera. T is obtained using the calibration method; C is obtained by locating the screen within the camera image. Finally, the mapping responsible for the key-stoning distortion, is given by P = C1T. Right: the application image can be appropriately distorted (pre-warped), using the mapping W so that it appears rectilinear after projection through a misaligned projector (modeled by the mapping P). 8 8
. 12. 16. 17. 18. 19. 22 22
3.1 An instance of what can be the oce of the future [10]. 3.2 A simple Camera Projector System Using structured light to obtain the surface geometry [1]. 3.3 An steerable Camera Projector System and An instance of its working [3] 3.4 Result of shadow elimination using method discussed in ??. 4.1 4.2 4.3 4.4 A snapshot from a shooting range used for practice. A snapshot from the game Hidden and Dangerous. A simple block diagram of a shooting range simulation system. Snapshot from Game: Delta force.

Chapter 1 Introduction

Projection is the most simple and economical way of producing large displays. Today, the cost of portable projectors are going down while the resolution oered by them is soaring high. In recent years, advances in portable projectors have made possible the development of novel projected displays like multi-planar displays [1], large multi-projector walls [5], steerable camera-projector systems [3], immersive environments [10], intelligent presentation systems [13], intelligent oce environments [6]. The eorts involved in manually calibrating multiple projectors to each other and aligning projected displays to physical surfaces is the driving force behind bringing research interest in projector-camera systems, where techniques adopted from multi-view geometry are applied to a collection of projectors and cameras. A system for multi-planar display can be used in oce like environments. An oce desk pushed against the wall can be used to display the elevation and plan of a map simultaneously. Continuous and rapid improvements in CPU performance, storage density and network bandwidth have provided sucient bandwidth and computational resources to support high-resolution displays and natural human computer interactions. With the modern advances in technology, the main bottleneck in an interactive computer system occurs in the link between computer and human, not between computer components within the system.

Motivation

Recently there has been an explosion of interest in systems which combine projection technology with computer vision. A characteristic of these systems is their ability to passively sense an environment in support of real-time control of projected light. Research in this area spans a number of disciplines including computer vision, computer graphics, HCI and display technologies. In particular, the theory and techniques used by researchers in the area are related, sometimes complementary to traditional computer vision techniques Large format display deemployed in stereo-camera and gesture recognition systems. vices, like projectors and at panels, are day by day becoming commodity items. New technologies like Organic Light Emitting Diodes(OLED) will soon be available at inex2
pensive prices [5]. The coupling of an imaging device (a camera) to a projector makes the assembly aware of its environment and as lower price technologies develop, the use of such assemblies will spread to everyday life and any man-made surface will become a potential display screen.

Challenges

Geometric Issues
For an ideal display from the projector, the projector must be perfectly aligned to the display surface such that the axis of projection is perpendicular to surface of the display screen. The geometric problems relate to computation of the spatial correspondences between pixels in the projectors and the projected display on the screen. The projectors should be accurately and automatically calibrated to the screen, to the camera and to each other. The calibration should enable the images in each projector to be pre-warped so as to create a desired projected display that is aligned with the screen. Achieving the accuracy of the order of a pixel or higher is a challenge in Projector Camera Systems. The devices used in the system, like projector(s) and camera(s), are prone to mechanical drifts. Modern portable projectors, which are usually mounted right before use, can drift during the process of display due to reasons like a push to the table. These kind of changes are not expected beforehand and the system has to be robust to such geometric deformations in the environment. The system has to re-initialize its knowledge of the environments geometry and this has to be as transparent to the user as possible. In systems for display on multi-planar surfaces, accurate localization of specic geometric features, such as the edge that denes the intersection of two planar surfaces, are critical for good performance. The increase in number of display planes and projectors leads to increased complexity in geometry of the scene. Also, rendering of scene becomes complicated.

Photometric Issues

Image intensity and color is another major problem to a projector system. The problem of non-uniform image intensity exists for imagery of individual projector. This problem becomes more predominant when two projector images overlap. The overlap from multi-projector results in brighter region in the overlapping area. Sometimes, a part of one display surface reects light onto another part, giving rise to change in the intensity (inter-reection). This problem is important because human are quite sensitive to such variations: we can perceive as little as 2% variation in brightnes and a 2nm variation in wavelength.

1.1. This rendered scene will be displayed on a display surface, which can even be multi planar. Shooter will shoot target displayed on screen. The camera will detect the hit and the simulator system will convert the point as seen by camera to corresponding point in the image. Thus, the system will nd accuracy of shooting. To support display on multi-planar surface, we take help of homography between small fragments of the display surface and the projector.
Figure 1.1: A snapshot of scene from Game: Hidden & dangerous

Outline Of The Report

The rest of the report is organized as follows. In chapter 2, a simple Projector Camera is discussed in greater detail. Chapter 3 has systems that have been developed or proposed by use of a Projector Camera System. This includes a brief introduction of cited systems and some pictures to explain their working. In chapter 4, the problem statement of this project has been presented with the challenges that are interpreted till now. The scope of this project has been discussed thereafter.
Chapter 2 Projector Camera Systems
A projector and a camera are both inverse of each other. A projector is used to generate lighting patterns, while a camera is used to capture them. When these devices are combined into one system, they complete a loop in the system. The projector displays some image on the screen while the camera works as a feedback agent, telling the system how does the projection look from the viewpoint of the user. This provides the system with a sense of intelligence to improve its display and make the environment and user interface of the system more comfortable to its users. A Projector Camera system has at least one projector and one camera. Use of multiple projectors is required to achieve increase in projection size, increase in brightness(intensity), increase in the sharpness(resolution) of the display or to avoid the situation when some part of the display area is occluded by some moving object(like a speaker giving presentation). Use of multiple cameras is to facilitate the scenario where every part of the surrounding is always visible to at least one camera, to gain geometry of a real world object from multiple viewpoints and to track dynamic changes(or events) in dierent directions when the display is immersive. The power of feedback loop makes a Projector Camera System intelligent. Such a system is being used for a lot of educational and research purposes and is proposed to become a daily life utility for a normal oce employee or even a household in near future. Some applications where such system is being used include display on multi-planar surfaces, large format displays, annotate real world objects. Some of these are discussed in the next chapter.

Camera Rotation and Translation
Motion of scene points can be modeled as follows
r11 r12 r13 tx r21 r22 r23 ty R t M = r31 r32 r33 tZ 0 1
with R as a rotation matrix and t = [tx ty tz ] as a translation vector.

Projection Matrix

Combining equations 2.3 and 2.5 the following expression is obtained for a camera with some specic intrinsic calibration and with a specic position and orientation:
x fx s c x R t f y cy y =

Camera Calibration

In this section we will see calibration technique in brief.

Camera Knowledge

Knowledge about the camera can be used to restrict the ambiguity on the reconstruction from projective to metric or even beyond. Dierent parameters of the camera can be known. Both knowledge about the extrinsic parameters (i.e. position and orientation) as the intrinsic parameters can be used for calibration. Interior Orientation Basic camera calibration is the recovery of the principle distance f and the principle point (x0 , y0 )T in the image plane or, equivalently, recovery of the position of the center of projection (x0 , y0 , f )T in the image coordinate system. This is referred to as interior orientation in photogrammetry. Interior orientation is the relationship between cameracentric coordinates and image coordinates. The camera coordinate system has its origin at the center of projection, its z axis along the optical axis, and its x and y axes parallel to the x and y axes of the image. Camera coordinates and image coordinates are related by the perspective projection equations: xI x 0 xC yI y 0 yC = and = (2.7) f zC f zC where f is the principle distance (distance from the center of projection to the image plane), and (x0 , y0 ) is the principle point (foot of the perpendicular from the center of projection to the image plane). That is, the center of projection is at (x0 , y0 , f )T , as measured in the image coordinate system. Interior orientation has three degrees of freedom. The problem of interior orientation is the recovery of x0 , y0 , and f. This is the basic task of camera calibration. However, as indicated above, in practice we also need to recover the position and attitude of the calibration target in the camera coordinate system. Exterior Orientation A calibration target can be imaged to provide correspondences between points in the image and points in space. It is, however, generally impractical to position the calibration target accurately with respect to the camera coordinate system using only mechanical means. As a result, the relationship between the target coordinate system and the camera coordinate system typically also needs to be recovered from the correspondences. This is referred to as exterior orientation in photogrammetry.

Characteristics of projector based system
Projector based display systems are so impressive that they are used everywhere now a days. They oer attractive combination of dense pixels over wider area. Some of 10
important characteristics of projector based display system are : 1. Size of projector: The size of the projector can be much smaller than the size of the image it produces. 2. Size of displayed image: The image generated are larger in size than their CRT and LCD counterparts. 3. Multi-projector Display: Overlapping images from multiple projectors can be eectively superimposed on the display surface. 4. Blending of Heterogeneous images: Images from projectors with quite dierent specications and form factors can be easily blended together. 5. Display surface: The display surface does not need to be planar or rigid, allowing us to augment many types of surfaces and merge projected images with the real world.

Screen-Camera Homography

The relation between the screen and camera world can be represented by a homography given by a matrix. As only the camera images are available to us, it is important to accurately estimate this screen-to-camera homography if we want to obtain geometrical parameters between the screen and the projector. The screen-camera homography can be easily computed if there are markers (or points of known coordinates on the screen). Authors in [13] use the fact that screen for presentation is usually square. The basic assumptions are that: 1. The positions, orientation and optical parameters of both camera and projector are unknown. 2. Camera and projector optics can be modeled by perspective transforms. 3. The projection screen is at. Consider a point (x, y) in the projector slide. This point is projected by perspective transform to some unknown point on the projection screen. The parameters of this perspective projection depend on the unknown position and orientation of the projector relative to the screen, and the unknown focal length of the projector optics. The point on the screen is then observed by a camera at pixel location (X, Y ). The parameters of the second perspective projection depend on the unknown position and orientation of the camera relative to the screen, and the unknown focal length of the camera optics. We need to nd the mapping between (x, y) and (X, Y ) without any additional information about the unknowns; in other words, given that we observe a feature at (X, Y ) in the camera image, we would like to know the point (x, y) in the projector slide that corresponds to 11
Figure 2.4: Left: the relationships between the three frames of reference corresponding to the computer display (source image frame), camera (camera image frame) and projection screen (projected image frame). Note that the last can only be indirectly observed by our system, through the camera. T is obtained using the calibration method; C is obtained by locating the screen within the camera image. Finally, the mapping responsible for the key-stoning distortion, is given by P = C1T. Right: the application image can be appropriately distorted (pre-warped), using the mapping W so that it appears rectilinear after projection through a misaligned projector (modeled by the mapping P).

this feature. (e.g. Given a bright laser pointer hit in camera image we need to determine whether the laser pointer has hit the specied target in the actual image?). This mapping appears to be impossible to determine in the presence of so many unknowns. But, as all of the observed points in the scene lie on some unknown plane (the at projection screen), we can establish a homography between the camera and projector frames of reference. Thus, we can show that the compounded transforms mapping (x, y) in the projector frame to some unknown point on the projection screen, and then to pixel (X, Y ) in the camera frame, can be expressed by a single projective transform expressed in homogeneous equation as,
xw p1 p2 p3 X yw = p p p Y 4 w p7 p8 p9 1
with eight degrees of freedom, = (p1. p9 )T constrained by || = 1. can be deterp p p 1 mined from as few as four pixel correspondences. When more than four correspondences are available, the system nds the best estimate in a least-squares sense. Given n feature
Four correspondences are sucient provided that no three points are collinear
point matches,{(xi , yi ), (Xi , Yi )}, let A be the following 2n 9 matrix:

X1 Y 0 X2 Y 0. Xn Y n 0 0

X 1 Y0 X 2 Y2. X n Yn

X1 x1 X1 x1 X2 x2 X2 x2.

0 Xn xn 1 Xn xn

Y1 y1 x1 Y1 y1 y1 Y2 y2 x2 Y2 y2 y2 . . . Yn yn xn Yn yn yn
The goal is to nd the unit vector that minimizes |A|, and this is given by the p p T eigenvector corresponding to the smallest eigenvalue of A A.
Steps for Determining Homography
The Projector Camera system oers a signicant advantage in determining these pixel correspondences: 1. The projector displays calibration slides to recover the projector-camera mapping parameters. 2. An initial estimate is obtained by projecting an image with a bright rectangle against a contrasting background. 3. The locations of the corners of the projected rectangle (in the camera image) are determined by standard image processing techniques and used as features to recover the projector-camera homography according to the procedure outlined above. 4. This homography is applied (as a pre-warp) to a calibration slides consisting of bright circles on a dark background. 5. The system estimates the center of each circle by calculating the centroid of the observed bright region in the camera image, and uses this sub-pixel estimate to rene its estimate of the projector-camera homography. Robust feature extraction is made simpler by the calculated homography since the system knows where and when to look for a feature in the camera image. The system can also monitor for any dynamic deformation in the projected image to determine when the mapping is no longer accurate and trigger automatically recalibration as necessary. As we have seen that the homography between the camera and the projection screen has to be calculated and hence the limitation arises. The above method cant be used if: 1. An ordinary wall or oor is used whose surface has no clue. 2. The boundary of projector screen can not be determined due to light conditions. 13

To resolve this by simple means we can put xed markers at the screen corners and use the camera to nd the screen corners. This will give us the location of screen in camera frame and thus the screen to camera homography. It might seem dicult to determine the screen-camera homography. If the projector is calibrated and its projection pattern is known, the screen-camera homography can be determined [?].
Chapter 3 Projector Camera Systems Application
The primary objective of the feedback loop formed by the camera in the Projector Camera System is to tell the system about the appearance of projected image from users viewpoint. This enables the system to do monitoring and correction of geometric and photometric distortions in the display. Many systems have been proposed and developed in theory and in reality to do the same. Some of the methods to achieve distortion free displays on planar and multi-planar surfaces can also be found in [12, 11]. Apart from this, there is no limitation on what other tasks a camera in such a system can be used to accomplish. The camera can be used to detect motion of the user in the environment so that appropriate transformation can be applied to the images to make the projection most suitable for viewing from users current location. The camera can also track the users body parts in order to identify signals which can be interpreted as commands by the underlying system. For all these purposes the assembly of Projector Camera System may have an array of projectors and cameras.
Multi-Planar Display Surface
The space requirement for display area is the main hindrance in generating large displays in small oce like environment. To use a small room for large display, we need to project on a structure of multiple planar surfaces. For a multi-planar projection surface [1, 10], the complete 3D geometry of the projection area can constructed by projecting structured light from the projector and viewing the deformations through a camera. This facilitates the system to generate a homography among the camera(s), the projector(s) and the plane(s) of display surface. Once the system has acquired the homography between the projector and the display, the image sent to projector are pre-warped by a transformation reverse to the homography. This nullies the eects of non-uniform display surface on the nal display from viewers viewpoint. The camera is used to capture the image and the system updates the homography in real-time whenever needed due to mechanical drift 15
Figure 3.1: An instance of what can be the oce of the future [10] in projector, camera or the display surface. Such assembly is also used to nullify other ill-eects of projector system such as photometric deformations by using the camera as a feedback agent. This kind of corrections are done in steps by means of a factor to prevent jitter in the projected image. The Oce of the Future as presented in [10] is a proposal which seem to become more and more realistic everyday as we see new emerging technologies in the eld of computer vision and projection. The ceiling mounted projectors are congured to use the entire surrounding even the windows and oors for projecting images. A person sitting in such an oce, when wants tele-conferencing, can actually get the feeling of working with ones colleagues. Though the idea seems very futuristic and is called the oce of the future, but with current resources in terms of research and technology, it will not be a bad idea to build such a system. Though it is possible to develop such a system, the hindrances come as cost of the imaging devices and their cost/resolution ratio.

Large Display Screens

When we have the resources to use a wall or a portable/xed screen as a large planar uniform display area, we are limited by the capability of the available projector to display 16
Figure 3.2: A simple Camera Projector System Using structured light to obtain the surface geometry [1] such a large image. For a given projector, the resolution of the device is xed and thus taking the projector farther away from the screen makes the projection blur. To tackle such a scenario we can use an array of cheap low resolution projectors instead of investing in a high-end costly device. Such a system is discussed in [11]. Multiple projectors are aligned with a small overlapping region between each neighboring projector to together make a large display. Now, for such a system, manual alignment of projectors is a troublesome task and also due to inevitable mechanical drifts the requirement for alignment will be frequent. So, the need for a self-calibrating system arises which can align the display of the projector assembly and also coordinate the input end of projectors. A small story presented in [5] by Thomas goes like this. Im walking down the hall towards my oce when Im reminded that Im late for a meeting with Mary to discuss the design of the students center being built on campus. Unsure of the location of her oce, I tap on the wall next to me and a large oor plan appears. After following a path displayed on the oor to guide me, I arrive at her oce, and we begin to work. Mary and I view an immersive walk-through of the design on her smart wall, and I draw modications with a virtual marker, whose strokes are recognized and used to manipulate the computer model.
Annotation of Real-World Objects
For a system to annotate the real world objects, a combined assembly of steerable camera projector system is used to display tool-tips (a common synonym used in desktop interfaces for brief introduction) in [3]. A steerable camera projector system turns the whole surrounding into a large continuous display surface. The idea of making the environment interactive is proposed to be achieved by incorporating interactive elements in the display. The system shown in 3.3 is discussed in [3] where it is shown to make the instrumented environment explorable and self explaining by automatically labeling physical objects in it. Each object detected in the physical environment is assigned an object ID and an annotation text. These are combined after formatting the text. A 17
Figure 3.3: An steerable Camera Projector System and An instance of its working [3] display surface is selected for the annotation and the annotation is rendered as bitmap. The choice of display area is made by ranking of free grid cells on selected display plane.

Capturing Depth and Texture of Time-Varying Scenes
An approach has been presented by Frueh and Zakhor in [4] to capture 2D depth and texture of time-varying scenes using structured infrared light. This is a system where we see use of infra-red camera(s) to view the environment. The infrared camera is used to simultaneously capture the visual appearance as well as the depth of a time-varying scene. The structured light is composed of a static vertical IR stripe pattern and a horizontal IR laser line sweeping up and down the scene. Vertical lines in the IR frames are identied using the horizontal line, intra-frame tracking, and inter-frame tracking; depth along these lines is reconstructed via triangulation. Interpolating these sparse depth lines within the foreground silhouette of the recorded video sequence, gives a dense depth map for every frame in the video sequence.
Shadow Removal in Presentation Systems
A projector is used most for presentation purposes, where a speaker explains the text or image displayed by the projector. Two problems which occur in such systems when a user obscures the projector are (i) undesirable shadow cast on the display by the users, and (ii) projected light falling on and distracting the user. The simplest solution is to mount the projector at an extreme solution and warp the image in order to remove key-stoning distortions. A better computational framework to solve these problems is discussed in [15]. The removal of shadow from the screen is accomplished by use of multiple projectors with 18
Figure 3.4: Result of shadow elimination using method discussed in ?? overlapping display regions and the removal of light from the users face is achieved by not displaying specic pixels from specic projector(s) which caused the shadow. Every frame to be displayed during presentation is captured by the camera in advance to form a set of reference images, which are then used to detect shadow during presentation. The system does not work in case when all the projectors are occluded by the user. The system can be improved by using multiple cameras to cover the case when the eld of view for the camera is also partially occluded. A simple result using this method presented in [14] is shown in 3.4. To prevent jerky motion from removing shadow at once when detected, the method moves the observed image closer to the reference image in steps using a controlling factor. It is shown in the result that the shadow is eliminated in three iterations.

Chapter 4 Shooting Range Simulation
A shooting range is used for practicing and testing shooting skills of shooter. A snapshot of shooting range is shown in g 4.1. The shooter is asked to shoot a pre-specied target at farther range with gun. The bullet hit by the shooter is compared with the actual target. From the dierence between actual target and point hit by shooter, accuracy of shooting can be determined. In real shooting range, we need to deal with cost of ammunition and most importantly toxic wastage produced by ammunition. Hence, there is need for a virtual shooting range practice environment. The standard simulation equipment like head mounted display (HMD) are not cost eective. Projector can provide large display in low cost and camera can be used for determining accuracy of shooting. Hence, a projector-camera system can be used for shooting range simulation. A virtual environment can be created and rendered as First Person View (FPV) 4.2.
Figure 4.1: A snapshot from a shooting range used for practice. Projector can display rendered scene on the display surface, which can be single planar or even multi planar. Soldier will shoot target displayed on screen using infra-red or normal laser gun. A infra-red or normal camera (depending on type of laser gun used) will detect the hit and nd accuracy of shooting. The environment can support multiple 20
users sharing a common screen or even users in dierent environments (like in dierent rooms of the facility) coordinated by a common server rendering the simulation of shooting range. A shooting range is used for practicing and testing shooting skills of shooter.
Figure 4.2: A snapshot from the game Hidden and Dangerous. The shooters are positioned with guns and have a pre-specic target. The accuracy of the shooting depends on distance between target and actual hit. The important issues with system are Accurate detection Real-time response

Shooting Range

A simple sketch of shooting range is shown in gure 4.1. A simple block diagram of the system is shown in gure 4.3. The major components of the system are as follows:

A Virtual Environment

A virtual world is simulated in the computer. It is a image based simulation of the real world model. A sample of such an environment is shown in gure 4.4. The target can be an enemy soldier or an enemy tank or any other means of attack used by the enemy. The virtual world is rendered as First Person View (FPV) scene and projected on a display surface. Assuming the display surface is single planar, the system sketch shown in gure 4.3. 21

a(x,y) a(x,y)

Screen

Soldier

Projector
Figure 4.3: A simple block diagram of a shooting range simulation system.
Figure 4.4: Snapshot from Game: Delta force
The soldier is the user of the system. Soldier is a person who wants to practice or whose skills are to be tested. The soldier is positioned in front of screen. He/She is given a gun with a laser pointer on it. The laser dot hits the target whenever the guns trigger gets pressed. Hence, we get the eect of a bullet.

Computer Based Simulator

The Projector Camera System plays the important part. The projector displays the FPV scene on the display surface. Because, the system will mostly be used with a front projection, it has to be assumed that soldier doesnt shadow the display surface. The camera is used as visual feedback for adjusting the projection as well as detection of laser pointer hitting on the display surface.

Display

The display screen is the surface on which the FPV scenes from the projector will be shown. All the images are pre-warped before projection such that projector can display scene on the display screen without any distortion. Need for the system to handle multiplanar display surface or only single-planar displays depends on the resources at hand. 22
The system will be able to handle multi-planar surfaces transparent to the user by the method discussed in [7].

Projector and Camera

Another part of the system is the projector and the camera. The projector projects the virtual worlds image rendered as a FPV scene on the display. The camera is used to calibrate the projector for a seamless display on (possibly multi-planar) display screen and also to detect the shootings. The soldier is provided with a laser gun (a normal red-light laser dot gun or an infra-red gun) to shoot the targets displayed on screen. The camera captures the laser dot appearing on screen and the system coordinates the shot location and time with its virtual world. The distance between the target and the actual point of shot, as discussed earlier, determines the accuracy of shooting.

Bibliography

[1] Mark Ashdown, Matthew Flagg, Rahul Sukthankar, and James M. Rehg. A exible camera projector system for multi-planar displays. In Computer Vision and Pattern Recognition, 2004. [2] Mark Ashdown and Rahul Sukthankar. Robust calibration of camera-projector system for multi-planar displays. Technical report, University of Cambridge, HP Labs, Carnegie Mellon University, January 2003. [3] Andreas Butz and Christian Schmitz. Annotating real world objects using a steerable projector-camera unit. In Computer Vision and Pattern Recognition, 2005. [4] C. Frueh and A. Zakhor. Capturing 2d depth and texture of time-varying scenes using structured infrared light. In IEEE International Workshop on Projector-Camera Systems, June 2005. [5] Thomas Funkhouser and Kai Li. Guest Editors introduction: Large-format displays. IEEE Computer Graphics and Applications, 20(4):2021, July/August 2000. [6] D. Hall, C. Le Gal, J. Martin, O. Chomat, T. Kapuscinski, and J. Crowley. Magicboard: A contribution to an intelligent oce environment, 1999. [7] Nilesh Heda. Projector-camera based solution for simulation system. Technical report, Department Of Computer Science, Indian Institure of Technology, Bombay, July 2004. [8] Carsten Kirstein and Heinrich Mueller. Interaction with a projection screen using a camera-tracked laser pointer. In MMM 98: Proceedings of the 1998 Conference on MultiMedia Modeling, page 191, Washington, DC, USA, 1998. IEEE Computer Society. [9] C. Liu, Y. Zhang, R. Xiong, and Y. Sun. Method of computer-aided measurement in a shooting range. In Proc. SPIE Vol. 4220, p. 226-230, Advanced Photonic Sensors: Technology and Applications, Jinfa Tang; Chao-Nan Xu; Haizhang Li; Eds., pages 226230, October 2000. 25
[10] Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. The oce of the future: A unied approach to image-based modeling and spatially immersive displays. Computer Graphics, 32(Annual Conference Series):179 188, 1998. [11] Peng Song and Tat-Jen Cham. A theory for photometric self-calibration of multiple overlapping projectors and cameras. In IEEE International Workshop on ProjectorCamera Systems, June 2005. [12] R.M. Steele, S. Webb, and C. Jaynes. Monitoring and correction of geometric distortion in projected displays. In Central European Conference on Computer Vision, Visualization, and Graphics, page 429, February 2001. [13] R. Sukthankar, R. Stockton, and M. Mullin. Smarter presentations: Exploiting homography in camera-projector systems. In International Conference on Computer Vision, 2001. [14] Rahul Sukthankar, Tat-Jen Cham, and Gita Sukthankar. Dynamic shadow elimination for multi-projector displays. In Proceedings of IEEE Computer Conference on Computer Vision and Pattern Recognition, 2001. [15] Rahul Sukthankar, Tat-Jen Cham, Gita Sukthankar, and James M. Rehg. Shadow elimination and occluder light suppression for multi-projector displays. In Proceedings of IEEE Computer Conference on Computer Vision and Pattern Recognition, 2001. [16] Tsai and Roger Y. An ecient and accurate camera calibration technique for 3d machine vision, 1986. [17] Tsai and Roger Y. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using o-the-shelf tv cameras and lenses, 1987.