Exposing Digital Forgeries Through Specular Highlights on the Eye
Micah K. Johnson and Hany Farid
Department of Computer Science Dartmouth College Hanover, NH 03755 {kimo,farid}@cs.dartmouth.edu www.cs.dartmouth.edu/{kimo,farid}
Abstract. When creating a digital composite of two people, it is dicult to exactly match the lighting conditions under which each individual was originally photographed. In many situations, the light source in a scene gives rise to a specular highlight on the eyes. We show how the direction to a light source can be estimated from this highlight. Inconsistencies in lighting across an image are then used to reveal traces of digital tampering. Key words: Digital Tampering, Digital Forensics

Introduction

The photograph in Fig. 1 of the host and judges for the popular television show American Idol was scheduled for publication when it caught the attention of a photo-editor. Coming on the heels of several scandals that rocked major news organizations, the photo-editor was concerned that the image had been doctored. There was good reason to worry the image was a composite of several photographs. Shown in Fig. 1 are magnications of the hosts and judges eyes. The inconsistencies in the shape of the specular highlight on the eyes suggest that the people were originally photographed under dierent lighting conditions. In this work, we show how the location of a specular highlight can be used to determine the direction to the light source. Inconsistencies in the estimates from dierent eyes, as well as dierences in the shape and color of the highlights, can be used to reveal traces of digital tampering. In related work, the authors of [5] showed how to estimate the light source direction in 2-D. While this approach has the benet of being applicable to arbitrary objects, it has the drawback that it can only determine the direction to the light source within one degree of ambiguity. In contrast, we estimate the full 3-D light source direction by leveraging a 3-D model of the human eye. Although not specically developed for a forensic setting, the authors of [7] described a technique for computing an environment map from eyes that embodies the illumination in the scene. While the environment map provides a rich source
Fig. 1. This photograph of the American Idol host and judges is a digital composite of multiple photographs. The inconsistencies in the shape of the specular highlight on the eyes suggest that these people were originally photographed under dierent lighting conditions. Photo courtesy of Fox News and the Associated Press.
of information about the lighting, it has the drawback of requiring a relatively high-resolution image of the eye. We describe how to estimate the 3-D direction to a light source from specular highlights on the eyes. We show the ecacy of this approach on synthetic and real images and visually plausible forgeries.

Methods

The position of a specular highlight is determined by the relative positions of the light source, the reective surface and the viewer (or camera). In Fig. 2, for example, is a diagram showing the creation of a specular highlight on an eye. In this diagram, the three vectors L, N and R correspond to the direction to the light, the surface normal at the point at which the highlight is formed, and the direction in which the highlight will be seen. For a perfect reector, the highlight is seen only when the view direction V = R. For an imperfect reector, a specular highlight can be seen for viewing directions V near R, with the strongest highlight seen when V = R. We will rst derive an algebraic relationship between the vectors L, N , and V. We then show how the 3-D vectors N and V can be estimated from a single image, from which the direction to the light source L is determined. The law of reection states that a light ray reects o of a surface at an angle of reection r equal to the angle of incidence i , where these angles are

Camera Eye

Fig. 2. The formation of a specular highlight on an eye (small white dot on the iris). The position of the highlight is determined by the surface normal N and the relative directions to the light source L and viewer V.
measured with respect to the surface normal N , Fig. 2. Assuming unit-length vectors, the direction of the reected ray R can be described in terms of the light direction L and the surface normal N : R = L + 2(cos(i )N L) = 2 cos(i )N L. By assuming a perfect reector (V = R), the above constraint yields: L = 2 cos(i )N V = 2 V TN N V. (2) (1)
The light direction L can therefore be estimated from the surface normal N and view direction V at a specular highlight. In the following sections, we describe how to estimate these two 3-D vectors from a single image. Note that the light direction is specied with respect to the eye, and not the camera. In practice, all of these vectors will be placed in a common coordinate system, allowing us to compare light directions across the image. 2.1 Camera Calibration
In order to estimate the surface normal N and view direction V in a common coordinate system, we rst need to estimate the projective transform that describes the transformation from world to image coordinates. With only a single image, this calibration is generally an under-constrained problem. In our case, however, the known geometry of the eye can be exploited to estimate this required transform. Throughout, upper-case symbols will denote world coordinates and lower-case will denote camera/image coordinates. The limbus, the boundary between the sclera (white part of the eye) and the iris (colored part of the eye), can be well modeled as a circle [7]. The image of the limbus, however, will be an ellipse except when the eye is directly facing the
camera. Intuitively, the distortion of the ellipse away from a circle will be related to the pose and position of the eye relative to the camera. We therefore seek the transform that aligns the image of the limbus to a circle. In general, a projective transform that maps 3-D world coordinates to 2-D image coordinates can be represented, in homogeneous coordinates, as a matrix. We assume that points on a limbus are coplanar, and dene the world coordinate system such that the limbus lies in the Z = 0 plane. With this assumption, the projective transformation reduces to a planar projective transform [2], where the world points X and image points x are represented by 2-D homogeneous vectors. Points on the limbus in our world coordinate system satisfy the following implicit equation of a circle: f (X; ) = (X1 C1 )2 + (X2 C2 )2 r2 = 0,

where = ( C1 C2 r ) denotes the circle center and radius. Consider a collection of points, Xi , i = 1,. , m, each of which satisfy Equation (3). Under an ideal pinhole camera model, the world point Xi maps to the image point xi as follows: xi = HXi , (4)
where H is a projective transform matrix. The estimation of H can be formulated in an orthogonal distance tting framework. Let E() be an error function on the parameter vector and the unknown projective transform H:

E(, H) =

min xi H X
where X is on the circle parametrized by. The error embodies the sum of the squared errors between the data, x, and the closest point on the model, X. This error function is minimized using non-linear least squares via the LevenbergMarquardt iteration [9] (see Appendix A for details). Once estimated, the projective transform H can be decomposed in terms of intrinsic and extrinsic camera parameters [2]. The intrinsic parameters consist of the camera focal length, camera center, skew and aspect ratio. For simplicity, we will assume that the camera center is the image center, that the skew is 0 and the aspect ratio is 1, leaving only the focal length f. The extrinsic parameters consist of a rotation matrix R and translation vector t that dene the transformation between the world and camera coordinate systems. Since the world points lie on a single plane, the projective transform can be decomposed in terms of the intrinsic and extrinsic parameters as: H = K ( r1 r2 t), (6)
where the intrinsic matrix K is: f K =0 0

0 0, 1

is a scale factor, the column vectors r1 and r2 are the rst two columns of the rotation matrix R, and t is the translation vector. With a known focal length f , and hence a known matrix K, the world to camera coordinate transform H can be estimated directly: K H = ( r1 H = ( r1 r2 r2 t) t), (8)
where the scale factor is chosen so that r1 and r2 are unit vectors. The complete rotation matrix is given by: R = ( r1 r2 r1 r2 ) , (9)
where denotes cross product. If the focal length is unknown, it can be directly estimated as described in Appendix B. 2.2 View Direction
Recall that the minimization of Equation (5) yields both the transform H and the circle parameters for the limbus. The unit vector from the center of the limbus to the origin of the camera coordinate system is the view direction, v. Let Xc = ( C1 C) denote the estimated center of a limbus in world coordinates. In the camera coordinate system, this point is given by: xc = HXc. (10)
The view direction, as a unit vector, in the camera coordinate system is then given by: v= xc , xc (11)
where the negative sign reverses the vector so that it points from the eye to the camera. 2.3 Surface Normal
The 3-D surface normal N at a specular highlight is estimated from a 3-D model of the human eye [6]. The model consists of a pair of spheres as illustrated in Fig. 3(a). The larger sphere, with radius r1 = 11.5 mm, represents the sclera

Limbus

r1 r2 d q

Cornea Sclera

(a) (b) Fig. 3. (a) A side view of a 3-D model of the human eye. The larger sphere represents the sclera and the smaller sphere represents the cornea. The limbus is dened by the intersection of the two spheres. (b) The surface normal at a point S in the plane of the limbus depends on the view direction V.
and the smaller sphere, with radius r2 = 7.8 mm, represents the cornea. The centers of the spheres are displaced by a distance d = 4.7 mm. The limbus, a circle with radius p = 5.8 mm, is dened by the intersection of the two spheres. The distance between the center of the smaller sphere and the plane containing the limbus is q = 5.25 mm. These measurements vary slightly among adults, and the radii of the spheres are approximately 0.1 mm smaller for female eyes [3, 6]. Consider a specular highlight in world coordinates at location S = ( Sx Sy ), measured with respect to the center of the limbus. The surface normal at S depends on the view direction V. In Fig. 3(b) is a schematic showing this relationship for two dierent positions of the camera. The surface normal N is determined by intersecting the ray leaving S, along the direction V , with the edge of the sphere. This intersection can be computed by solving a quadratic system for k, the distance between S and the edge of the sphere,
2 (Sx + kVx )2 + (Sy + kVy )2 + (q + kVz )2 = rk 2 + 2(Sx Vx + Sy Vy + qVz )k + (Sx + Sy + q 2 r2 ) = 0,
where q and r2 are specied by the 3-D model of the eye. The view direction V = ( Vx Vy Vz ) in the world coordinate system is given by: V = R1 v, (13)
where v is the view direction in camera coordinates, Section 2.2, and R is the estimated rotation between the world and camera coordinate systems, Section 2.1. The surface normal N in the world coordinate system is then given by: Sx + kVx N = Sy + kVy , (14) q + kVz and in camera coordinates: n = RN.

Light Direction

Consider a specular highlight xs specied in image coordinates and the estimated projective transform H from world to image coordinates. The inverse transform H 1 maps the coordinates of the specular highlight into world coordinates: Xs = H 1 xs (15)
The center C and radius r of the limbus in the world coordinate system determine the coordinates of the specular highlight, S, with respect to the model: S= p (Xs C) , r (16)
where p is specied by the 3-D model of the eye. The position of the specular highlight S is then used to determine the surface normal N , as described in the previous section. Combined with the estimate of the view direction V , Section 2.2, the light source direction L can be estimated from Equation (2). In order to compare light source estimates in the image, the light source estimate is converted to camera coordinates: l = RL

Results

We tested our technique for estimating the 3-D light source direction on both synthetically generated and real images. In all of these results the direction to the light source was estimated from specular highlights in both eyes. This required a slight modication to the minimization in Equation (5) which is described in Appendix A. The view direction, surface normal and light direction were then estimated separately for each eye. 3.1 Synthetic Images
Synthetic images of eyes were rendered using the pbrt environment [8]. The shape of the eyes conformed to the 3-D model described in Section 2.3 and the eyes were placed in one of 12 dierent locations. For each location, the eyes were rotated by a unique amount relative to the camera. The eyes were illuminated with two light sources: a xed light directly in line with the camera, and a second light placed in one of four dierent positions. The twelve locations and four light directions gave rise to 48 images, Fig. 4. Each image was rendered at a resolution of pixels, with the cornea occupying less than 0.1% of the entire image. Shown in Fig. 4 are several examples of the rendered eyes, along with a schematic of the imaging geometry. The limbus and position of the specular highlight(s) were automatically extracted from the rendered image. For each highlight, the projective transform H, the view direction v and surface normal n were estimated, from which the
Fig. 4. Synthetically generated eyes. Each of the upper panels corresponds to dierent positions and orientations of the eyes and locations of the light sources. The ellipse t to each limbus is shown in dashed green, and the red dots denote the positions of the specular highlights. Shown below is a schematic of the imaging geometry: the position of the lights, camera and a subset of the eye positions.
Fig. 5. A subject at dierent locations and orientations relative to the camera and two light sources. Shown to the right are magnied views of the eyes. The ellipse t to each limbus is shown in dashed green and the red dots denote the positions of the specular highlights. See also Table 1.
direction to the light source l was determined. The angular error between the estimated l and actual l0 light directions is computed as: = cos1 lT l0. (17)

where the vectors are normalized to be unit length. With a known focal length, the average angular error in estimating the light source direction was 2.8 with a standard deviation of 1.3 and a maximum error of 6.8. With an unknown focal length, the average error was 2.8 with a standard deviation of 1.3 and a maximum error of 6.3. 3.2 Real Images
To further test the ecacy of our technique, we photographed a subject under controlled lighting. A camera and two lights were arranged along a wall, and the subject was positioned 250 cm in front of the camera and at the same elevation. The rst light L1 was positioned 130 cm to the left of and 60 cm above the camera. The second light L2 was positioned 260 cm to the right and 80 cm above the camera. The subject was placed in ve dierent locations and orientations relative to the camera and lights, Fig. 5. A six mega-pixel Nikon D100 camera with a 35 mm lens was set to capture in the highest quality JPEG format. For each image, an ellipse was manually t to the limbus of each eye. In these images, the limbus did not form a sharp boundary the boundary spanned

image 5

left eye L1 L2 5.8 7.6 8.7 9.3 12.5 16.4 14.0
right eye L1 L2 3.8 1.6 0.8 11.0 7.5 7.3 13.8
left eye L1 L2 5.8 7.7 10.4 17.6 10.4 13.6 17.4
right eye L1 L2 3.9 1.7 18.1 10.1 7.4 5.6 16.5
Table 1. Angular errors (degrees) in estimating the light direction for the images shown in Fig. 5. On the left are the errors for a known focal length, and on the right are the errors for an unknown focal length. A indicates that the specular highlight for that light was not visible on the cornea.
roughly 3 pixels. As such, we t the ellipses to the better dened inner outline [4], Fig. 5. The radius of each limbus was approximately 9 pixels, and the cornea occupied 0.004% of the entire image. Each specular highlight was localized by specifying a bounding rectangular area around each highlight and computing the centroid of the selection. The weighting function for the centroid computation was chosen to be the squared (normalized) pixel intensity. The location to the light source(s) was estimated for each pair of eyes assuming a known and unknown focal length. The angular errors, Equation (17), for each image are given in Table 1. Note that in some cases an estimate for one of the light sources was not possible when the highlight was not visible on the cornea. With a known focal length, the average angular error was 8.6 , and with an unknown focal length, the average angular error was 10.5. There are several reasons for the increase in error over the synthetic images. First, the average size of the cornea in our real images is much smaller than the size of the cornea in the synthetic images, 256 pixels2 versus over 1000 pixels2. Second, the limbus in an adult human eye is slightly elliptical, being 1 mm wider than it is tall [3], while our model assumes a circular limbus. Shown in Fig. 1 is a photograph of the host and judges of the television show American Idol, and shown in Fig. 6 are the results of estimating the direction to the light source for each person. These estimates are rendered as Gaussian blobs ( = 15 ) on a hemisphere. The nal estimate is depicted as a sum of Gaussians, one for each specular highlight. Note that the estimates in the two right-most plots are visually consistent with one another, but are signicantly dierent from the two left-most estimates. Shown in Fig. 7 is a composite where the fathers face has been replaced with a dierent face. Two specular highlights are visible on each of the childrens eyes. The light direction was estimated from each specularity and for each eye. Across the childrens eyes, the average pair-wise dierence in orientation for the rst specularity was 8.5 with a maximum dierence of 11.6. The average dierence for the second specularity was 9.4 with a maximum dierence of 13.9. By comparison, the average dierence in orientation between the fathers specularities to those of the children was 40.3. We did not estimate the light

Fig. 6. The estimated light source direction for each person in Fig. 1 is depicted as Gaussian blobs on a hemisphere, each centered about the estimated 3-D direction. Superimposed on each hemisphere is an image of one of the eyes from which the estimates were made. Note that the inconsistencies in the light source direction suggest that the photograph is a composite of at least three photographs.
direction for the woman because we have found that glasses distort the shape and location of the specularity on the eye.

Discussion

When creating a composite of two or more people it is often dicult to match the lighting conditions under which each person was originally photographed. Specular highlights that appear on the eye are a powerful cue as to the shape, color and location of the light source(s). Inconsistencies in these properties of the light can be used as evidence of tampering. We have described how to measure the 3-D direction to a light source from the position of the highlight on the eye. While we have not specically focused on it, the shape and color of a highlight are relatively easy to quantify and measure and should also prove helpful in exposing digital forgeries. Since specular highlights tend to be relatively small on the eye, it is possible to manipulate them to conceal traces of tampering. To do so, the shape, color and location of the highlight would have to be constructed so as to be globally consistent with the lighting in other parts of the image. Inconsistencies in this lighting may be detectable using the technique described in [5]. Also working in our favor is that even small artifacts on the eyes are visually salient. Nevertheless, as with all forensic tools, it is still possible to circumvent this technique. We expect this technique, in conjunction with a growing body of forensic tools, to be eective in exposing digital forgeries.

Acknowledgments

We are grateful to Fabio Pellacini for many helpful conversations and suggestions. This work was supported by a Guggenheim Fellowship, a gift from Adobe Systems, Inc., a gift from Microsoft, Inc., a grant from the United States Air Force (FA8750-06-C-0011), and under a grant (2005-DD-BX-1091) awarded by the Bureau of Justice Assistance (points of view or opinions in this document
Fig. 7. A composite where rock star Gene Simmons face has been inserted into a family portrait.
are those of the author and do not represent the ocial position or policies of the United States Department of Justice).

References

1. Sung Joon Ahn. Least Squares Orthogonal Distance Fitting of Curves and Surfaces in Space, volume 3151 of Lecture Notes in Computer Science. Springer, 2004. 2. Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2004. 3. Michael J. Hogan, Jorge A. Alvarado, and Joan Esperson Weddell. Histology of the Human Eye. W.B Saunders Company, 1971. 4. D. Robert Iskander. A parametric approach to measuring limbus corneae from digital images. IEEE Transactions on Biomedical Engineering, 53(6):11341140, June 2006. 5. Micah K. Johnson and Hany Farid. Exposing digital forgeries by detecting inconsistencies in lighting. In ACM Multimedia and Security Workshop, New York, NY, 2005. 6. Aaron Lefohn, Richard Caruso, Erik Reinhard, Brian Budge, and Peter Shirley. An ocularists approach to human iris synthesis. IEEE Computer Graphics and Applications, 23(6):7075, 2003. 7. Ko Nishino and Shree K. Nayar. Eyes for relighting. ACM Transactions on Graphics, 23(3):704711, 2004. 8. Matt Pharr and Greg Humphreys. Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann, 2004. 9. Andrzej Ruszczyski. Nonlinear Optimization. Princeton University Press, 2006. n

Appendix A

In this appendix we describe the minimization of the error function:
which yields the perspective transform H and circle parameters. For notational convenience, we express this error function as E(u) where u = ( h ), and where the vector h contains the nine elements of the matrix H. This error function is minimized in nested iterations as described in [1]. The inner iteration computes the closest point X on the model for each image point xi , where the model is specied by the current state of u. The outer iteration then updates the parameter vector u according to the results of the inner model tting. This process is repeated and terminates when the norm of the update to u is below a specied threshold.
Closest point: For a given point xi in image coordinates, we seek the closest point X on the model. The point X that satises this condition must, of course, be on the model, f (X; ) = 0. Recall that f () is the equation of a circle, Equation (3). In order to contend with the scale ambiguity inherent to homogeneous coordinates, this model takes on a slightly dierent form: f (X; ) = (X1 /X3 C1 )2 + (X2 /X3 C2 )2 r2 (19)
For X to be the closest point, it must satisfy two additional criteria. First, the vector between the image point xi and the model point H X (expressed in image coordinates) must be parallel to the gradient of the model in image coordinates, H T f , yielding the following constraint: z T ((xi H X) H T f ) = 0, (20)
where z T = ( 1 ) restricts this constraint to the image plane. Second, the model point H X must lie in the image plane (recall that the homogeneous points xi lie in the plane z = 1): z T (xi H X) = 0. (21)
These three constraints form a system of non-linear equations that can be solved using the Gauss-Newton method, where the vector-valued function to be minimized is: f (X, ) g(u, xi , X) = z T ((xi HX) H T f ) . (22) z T (xi HX)
In practice, the image point xi is expressed in terms of world coordinates xi = HXi. This error function is given by: f (X, ) g(u, Xi , X) = z T (H(Xi X) H T f ) . (23) z T H(Xi X) This inner iteration is initialized with H equal to the identity matrix, and , the circle parameters, equal to a bounding circle t to the image data. Parameter update: Once the inner iteration completes and the closest points X have been computed for each image point xi , the parameter vector u can be updated. The outer iteration uses a Levenberg-Marquardt minimization, which requires the derivative of xi HX with respect to u, evaluated at the closest point X: (xi HX) u =

where [X] is a block-diagonal matrix with X on the diagonal. The derivative H/u is computed by simply dierentiating the matrix H with respect to each of its components hi. The derivative X/u is computed by implicitly dierentiating g() with respect to u: g g X g Xi + + = 0, u X u Xi u and solving for X/u: X = u g X

g g Xi + u Xi u

The individual derivatives in this expression are determined by straight-forward dierentiation of each function with respect to its unknowns. The derivatives for all m image points, x1 to xm , are then stacked into a 3m 12 Jacobian matrix, where 12 corresponds to the total number of unknowns (9 elements of H and 3 circle parameters ). This Jacobian matrix is used by the Levenberg-Marquardt minimization to compute the update to the parameter vector u. Constraints: The minimization described above can be extended to handle two circles by creating a block-diagonal Jacobian matrix from the Jacobian matrices of the individual eyes. In addition, constraint equations can be added to the error function E(u), Equation (18), to ensure that the transform H for both eyes is the same and that the radii of the circles are equal to 5.8 mm. The error function for both eyes with constraints is then given by: E(u1 , u2 ) = E(u1 ) + E(u2 ) +w h1 h2
+ (det(H1 ) 1)2 + (det(H2 ) 1)2 (27)
+ (r1 5.8)2 + (r2 5.8)2 ,
where w is a scalar weighting factor. The Jacobian of this system is: J1 (u1 ) J(u1 , u2 ) = J2 (u2 ) , J1 (u1 ) J2 (u2 )
where J1 and J2 are the Jacobian matrices from the individual eyes, and J1 and J2 are the Jacobians of the constraint equations with respect to u1 and u2. The transforms H1 and H2 are initially set to the identity matrix, and the circle parameters were chosen to enclose the limbus of each eye.

Appendix B

In this appendix we describe how to decompose the projective transform H in Equation (6) in the case when the focal length f is unknown. The transform H has eight unknowns: the focal length f , the scale factor , the three rotation angles x , y and z for the rotation matrix R, and the three coordinates of the translation vector t. By multiplying the matrices on the righthand side of Equation (6), H can be expressed in terms of these unknowns: f cy cz f cy sz f tx H = f (sx sy cz cx sz ) f (sx sy sz + cx cz ) f ty , (29) cx sy cz + sx sz cx sy sz sx cz tz where cx = cos(x ), sx = sin(x ), etc, and where the rotation matrix follows the x-y-z convention. Consider the upper-left sub-matrix of H rewritten in terms of the four unknowns x , y , z , and f = f. These unknowns are estimated by minimizing the following error function using non-linear least-squares: E(x , y , z , f ) = (f cy cz h1 )2 + (f cy sz h2 )2 + (f (sx sy cz cx sz ) h4 )+ (f (sx sy sz + cx cz ) h5 ) , (30) where hi corresponds to the ith entry of H. A Gauss-Newton iterative approach is employed to minimize E(). In practice, we have found that z = tan1 (h2 /h1 ), f = 1 and random values for x and y provide good starting conditions for this minimization. These estimated parameters then yield two possible estimates of the focal length: f1 = f (cx sy cz + sx sz ) h7 and f2 = f (cx sy sz sx cz ). h8 (31)

These two estimates are combined using the following weighted average: f= h2 f1 + h2 f7. h2 + h8 (32)
Note that the focal length f is undened for h7 = h8 = 0. In addition, this estimation is vulnerable to numeric instabilities for values of h7 and h8 near zero. As such, the weighting was chosen to favor larger values of h7 and h8.

ARTICLE REPRINT

Design Management Journal
Digital ethnography: The next wave in understanding the consumer experience
Davis Masten, Principal, Cheskin Tim M.P. Plowman, Design Anthropologist, Cheskin
Reprint #03142MAS75 This article was first published in Design Management Journal Vol. 14, No. 2 Fusing Design, Strategy and Technology
Copyright Spring 2003 by the Design Management Institute. All rights reserved. No part of this publication may be reproduced in any form without written permission. To place an order or receive photocopy permission, contact DMI via phone at (617) 338-6380, Fax (617) 338-6570, or E-mail: dmistaff@dmi.org. The Design Management Institute, DMI, and the design mark are service marks of the Design Management Institute.

www.dmi.org

SUPPORT
by Davis L. Masten and Tim M.P. Plowman n the search for market insights, Tim Plowman and Davis Masten maintain that the pathways to information should include PCs, cell phones, Webcams, global positioning equipment, digital cameras, and a growing number of other technologies. Structured creatively for self-reporting, passive observation, and participant observation, these media can yield facts businesses can analyze to shape individual and strategic design decisions.
Davis Masten, Principal, Cheskin
Tim M.P. Plowman, Design Anthropologist, Cheskin
The increasingly rapid migration of technology across geographic and socioeconomic boundaries is a fundamental constituent of the times in which we live. It is a process that takes subtle and numerous forms. Parents can check in on their kids at daycare over the Internet by using X10 technology. Russian teenagers organize roving raves through globally oriented blogs. American teens use Pringles potato chip cans to enhance the range of their wi-fi-enabled PCs and warchalk the location (that is, mark on walls and sidewalks to indicate wireless access areas). Students everywhere are learning to surreptitiously text-message each other in class using their cell phones. With the ever-lower prices of chips, disks, and memory, the continuing
broadband revolution, and the development of new protocols, a new domain for the elaboration of self and culture has emerged, and it is worth studying. There has been considerable research on the social aspects of digital communication, online consumption, and the Web as a social phenomenon. Social scientists, marketing professionals, and product designers, however, have paid less attention to the opportunities presented by digital technology for understanding the lives of users and consumers. We propose using the digital and wireless communication revolutions as platforms for rethinking ethnographic principles, methodologies, and analysis. Our goal is to produce new, deep, continuing, and rapid insights into peoples
Design Management Journal Spring 2003
Fusing Design, Strategy, and Technology
lives and needs. We call this convergence and updating of traditional methods with digital technology Digital Ethno. The tools on the customer side are as ubiquitous as cell phones, PDAs, email, Webcams, SMS, GPS, and digital cameras. For anthropologists and, specifically, for ethnographers, all these tools can fall into the class of remote sensing devices. Remote sensing According to Paul Saffo, the director of the Institute of the Future, in Menlo Park, California, this is the decade of remote sensing. Computers and sensors are being embedded in many durable goods as a matter of course. Appliance manufacturers are embedding computers into refrigerators and ovens and, with the imminent adoption of new Internet protocols such as IPv6, many of these will be Web-enabled and connected. As more and more of these tools are produced and used, the price inevitably plummets. Only a few years ago, the basic chip set for a hand-held GPS (global positioning system) receiver cost $3,000 or more. Now, its just a fraction of that price. With manufacturing costs this low, GPS systems are being built into many devices that we consider everyday tools, such as wrist watches and cell phones. This cost/volume relationship holds true for a range of personal technology and sensing devices. Ubiquity and affordability make these technologies more realistic as research tools. The Internet connection Many of these devices have been designed for stand-alone and single-task purposesmeaning that when a motion-sensing plastic frog, designed in the US and built in Taiwan, calls out to you as you near the backyard pool, it is not connected to anything. However, many nations are moving into the deeply connected world of the global, networked economy. According to John Chambers, CEO of Cisco, the wired

We propose using the

digital and wireless communication revolutions as platforms for rethinking ethnographic principles, methodologies, and analysis
countries will be the ones with the fastest-growing and most-productive economies. What this means is that many of the products and devices that we think of as stand-alone will achieve new functionality and utility by being connected to a network. We are only just starting to see examples of this networked world. These days, you can email the pictures you take to anyone with an email address. In Hong Kong, your cell phone will notify you when you are within range of a Starbucks and offer you a discount on a cup of coffee. In this case, your phone is tracked by the cellular ground station antennas, which triangulate on your location. In a striking and recent example, T-Mobile Sidekick users are sharing their daily experiences via their hip-top devices and the Web site, Hiptop Nation (www.hiptop.bedope.com). As the Web and broadband capabilities become increasingly like the water and power utilities of today, remote devices and similar technologies will be built and connected into more and more commodities. And, as the world becomes increasingly wired, it will become ever easier to conduct the type of research we are proposing. There are already at least 35 million Japanese using cell phones that are Web-enabled.1 Imagine if just one percent of them were participating in a sponsored contest to uncover the next big thing in street fashion, and as a result were engaged in collective trend-watching. Or imagine if another one percent emailed pictures from their camera phones to the local government and local media, visually and powerfully illustrating a safety complaint plaguing their neighborhood while they were currently describing it over the phone. The paring away of institutional and social distance and abstraction might have very positive effects in a variety of contexts. Business discovers ethnography During the past five years or so, ethnography has been widely embraced (and to a degree uncritically co-opted) by the business world, and various attempts have been made to reconfigure its techniques to suit business purposes. But because these reconfigurations are still based upon the ethnographic methods of anthropologists like Franz Boas, Bronislaw Malinowski,
1. Source: www.nttdocomo.com.
Alfred Kroeber, and A.R. Radcliffe-Brown practitioners whose work is nearly a century oldinnovation within commercial ethnography is limited to its application in novel contexts. Moreover, commercial ethnography as it is traditionally practiced means large-scale, complex projects, usually involving a multidisciplinary team made up of ethnographers, technologists, psychologists, and the like. These projects are typically done for short periods of time, given that it is very costly to establish behaviors and accompanying analyses over periods of much longer duration. Although Cheskin and a few other firms are fortunate enough to be involved in large-scale, global ethnographies, these studies, often done simultaneously in numerous countries, are frequently impractical for the industry at large. After developing a thorough inventory of ethnographic techniques appropriate to business-based ethnography, Cheskin divided them into three categories of data-gathering: selfreporting, passive observation, and participant observation. We then developed digital equivalents to these traditional methods, as well as entirely new methods of data capture. Introduction to Digital Ethnography In essence, Digital Ethno is the modern, digital equivalent of traditional, Malinowskian ethnographic forms. The critical distinction is that while traditional ethnographers physically immerse themselves in distinct places and their cultures, digital ethnographers capitalize on wired and wireless technologies to extend classic ethnographic methods, like participant

observation, beyond geographic, as well as temporal, boundaries. This method is ideally suited to documenting the fluidity and flexibility already distinguishing contemporary cultures and communities. Participants communicate their experience via the Internet and other digital technologies. Digital ethnographers gather these details, whether theyre in the form of words, images, or audio files, and determine their significance as they are played out in the context of participants lives. Despite the fact that there is now a growing academic literature and practice of what has been called hypermedia ethnography or cybersociology, we have largely had to forge our own way in developing Digital Ethno.2 Much of this previous work concerns online ethnography using data-gathering methods such as site perusal and online interviewing. These are generally text-based techniques transplanted on to the Internet. They are not inherently digital. Digital Ethno concentrates more on how ethnographic data gathering can be extended to the Internet and wireless communication devices in new and creative ways, especially in light of recent software, hardware, and protocol adoption. An extranet and WLAN can be key components in the task of data gathering and analysis. An extranet, for example, can serve as a place to download and upload data quickly and easily and provide a virtual locale and repository
2. Notable exceptions do existfor example, the Digital Ethnography Workshop at the University of California at San Diego, run by Edwin Hutchins (see http://hci.ucsd.edu/dew/html/index.htm).

Privacy

Many people with access to the Internet are already engaged in elaborating their identity through new media: putting their lives on display in both textual and visual terms. This combination of cultural and identity politics and new media has produced some interesting new cultural forms. Extreme examples include performance-art Webcasts of surgeries, births, and other intimate moments of peoples lives. A more mundane example might be pictures of a wedding. Digital photos from family vacations are published on the Web every day. On a more dubious note, the spread of reality-based entertainment, such as Survivor and American Idol, as well as the continuing proliferation of confessional and sensationalistic talk shows, coincides with the advent of technology adept at documenting and disseminating peoples most intimate life details. Amid the confessional and identity-constructing activities occurring through digital media, the question of privacy forcefully emerges. There are still many issues to be sorted out with these new technologies, not the least of which is the question of the protection of our privacy. In using and advancing Digital Ethno, it is clear that the respondents have the rights to data outside our very limited uses. Their lives are their own; we are only visiting with their permission. However, we are just at the beginning stages of determining what is appropriate around the world, and issues concerning respondent privacy are being formally worked out, with reference to guidelines set forth by privacy organizations.

for the project at hand. One complaint frequently heard about the potential of Digital Ethno is that the bandwidth is too narrow and thus, researchers miss the critical aural, gestural, and kinesthetic cues of faceto-face interaction. As the price of remote sensing devices has fallen, so too will the barriers to getting at content-rich data. And this will happen sooner in commercial ethnography than in academic ethnography. The innovations that are occurring on the data-gathering end are equally present in terms of deliverables:
The potential for integrating visual and written media within the same technological environment carries significant implications. It allows ethnographers to make the step from thinking of the visual merely as illustrative of argumentation spelled out through the printed word, to seeing it as itself constitutive of meaning. This is an observation that visual ethnographers have been trying to press home for years. In fact, we need to consider seriously what hypermedia can do that a well-illustrated book or a wellproduced film cannot. There are potential gains to be derived from exploring how ethnographic representation can simultaneously be a verbal and a pictorial, a visual and an aural activity.3
A basket full of socks, given to me by my girlfriend: a practical, if last minute, gift. I think she must have been wrapped up in some Valentines Day survey and forgot about Valentines Day.
While the above paragraph focuses on academic ethnography as product, there is no reason why similar innovations cannot take place with regard to data gathering for commercial purposes. The market is awash with software and shareware that lends itself to the process (MacroMedia, NVivo, Shockwave, PhotoShop, Media Maker, Director, iMovie, CoolEdit, and so forth). Valentines Day: A case study In February 2000, Cheskin piloted the first nonproprietary Digital Ethno project that was done for public consumption. The project focused on Valentines Daya common cultural event, one shared by North Americans throughout the United States and yet often hotly contested in its cultural meanings and personal significance. In order to test the method, we selected eight people from the San Francisco Bay Areasix involved in relationships and two unattachedin order to digitally observe their Valentines Day preparations and practices. In particular, we focused on participants attitudes toward and behaviors associated with contemporary Valentines Day icons:

hearts, candy, flowers, and kisses. The team included two ethnographers. We used a wide variety of techniques including email, cell phones, digital cameras, chatrooms, online questionnaires, and digitized audio diaries, among others to gather the data. While such a study would have been highly appropriate for Tokyo or Rio de Janeiro, where distance and time differences are major components, we decided to keep the complexity to a minimum and scaled the project accordingly. A week before Valentines Day, we asked the respondents to fill out an online questionnaire gauging their reactions to the four icons. We then sent them prompts via email over the course of the next week that asked them to engage in numerous activities documenting their Valentines Day experience. Ethnography participants were required to have access to email, a desktop computer, and a cell phone. At the outset of the project, each was given a digital camera to help gather critical visual data. All these tools were used by Cheskin to relay project-related tasks to the participants and for the participants to submit their findings, visu3. Bruce Mason and Bella Dicks, The Production of Hypermedia Ethnography. Retrieved from http://www.wordcircuits.com/htww/dicks1.htm.
I found a cupcake and a box of heart-shaped chocolates on my desk when I arrived at work. The cupcake was from a co-worker who bakes something for every holiday; the chocolate came from my manager.
I chose this heart picture because of the temporal nature of the hearts presented not because of the larger temporal nature of romantic love, but because if I found myself celebrating Valentines Day, it would be with just such a gesture, and not by buying things.
al and textual. At no point did the researchers meet face-to-face with participants. All interactions were conducted via email, telephone, or cell phone. The study proceeded as follows: Day 1 Online Web survey of attitudes regarding common Valentines Day symbols and icons Email prompt for participants to locate and digitally photograph (or download from the Web) their ideal versions of the Valentines Day icons and email them back, with brief descriptions, to Cheskin Day 2 Impromptu telephone interview via cell phone asking participants to share memories of Valentines Days past and the roles of the Valentines Day icons Day 3 Email prompt to photograph the Valentines Day icons in their common contexts and email these back with descriptions to Cheskin Day 6Valentines Day Email to prompt participants to phone in a verbal description of their Valentines Day experience and any Valentines Day icons they used or observed Day 7 Repeat of the original online survey to tally attitude change Online chatroom to discuss findings and insights on Valentines Day practices and icons After Day 7 Email survey asking each participant to reflect on his or her experiences of the study

Summarized results One noteworthy aspect of this study was the extent to which the respondents engaged in their own analyses of Valentines Day. Participating in the study forced them to think perhaps more deeply than usual about the significance of the holiday. Accordingly, the study not only captured respondents ideas about Valentines Day, it also influenced their observations of the holiday. Moreover, the use of Digital Ethno techniques made the respondents true partners in the data collection processmore consultants than respondents. While respondent partnership was a clear goal in this particular project, digital ethnographers will need to manage for this type of interaction in the future. It is easy to imagine people using their digital camera/cell phone at the behest of a permission-based, randomized digital prompt, without knowing why they are taking the pictures, thereby avoiding the respondent bias. Despite the small number of respondents and the experimental nature of the methodology, Digital Ethno was able to provide rich insights into peoples experience, allowing the team to form notable conclusions. In brief, Valentines Day is a paradox. It is seen as offering a means by which we may demonstrate affection and true sentiment, but the very symbolic tools with which we are provided generally undercut any meaning to our sentiments by virtue of their clichd and commercial character. Thus Valentines may be regarded as simultaneously trivializing love and enabling it. Those who celebrated the holiday generally found creative ways to circumvent the holidays
the heart-shaped cookie [Richs] wife made. He gave us all these adorable cookies with pink frosting. He said he tried to help out but he botched them up and she asked him to leave the kitchen. Its probably a way for his wife to stay in touch with his career/workplace while also doing something out of the ordinary. They looked cute, but I didnt think they tasted really good.
As it stands, I work in a very creative environment. This picture refers to the first Valentine card I received from one of my co-workers. She made some cards by hand and gave them out. I wont throw this one out.
over-commercialized sensibility. They made their own objects or gave unusual ones. They essentially reinvented the holiday for their own purposes and to some extent ended up taking ownership of the holiday. Thus, those who rejected the celebration of the holiday did so on two levels: they rejected the commercial version, and they also refused to reinvent the holiday. Those who were inventive in their celebrations (for example, I hide candy in my boyfriends sock drawer) were regarded by others as providing tips on how to take control of the holiday and make it their own. The chatroom discussion allowed rejectors to see how the holiday might be made more meaningful through small acts not usually associated with Valentines Day. Benefits of Digital Ethno By putting the power of participant observation in the participants hands, Digital Ethno enables participants to convey the real-time richness of their own lives and environments. In the Valentines Day study, we saw how a broken bowl of oatmeal soured one respondents subsequent Valentines Day experience, which he recounted the next day in a long, tragic digitized voicemail. To capture the meaning of a kiss, another respondent photographed herself kissing her cat, Buster, which, she explained, is more relevant than any other nonfamilial kiss as he has outlasted many relationships. Two days later, we learned she had just parted with her boyfriend. Digital Ethno also realizes the possibility of remote and simultaneous research. Researchers can conduct the projects from a centralized location while the participants fan out into their environments to observe their own practices, as well as the practices of those surrounding them. Likewise, researchers can remotely observe all participants at essentially the same time, in the case of our project, during the week prior to Valentines Day. In this way, Digital Ethno is

scaleable, even to transnational proportions. Time and place, not research constraints, identify the opportunities. The opportunity for scaleability should be underscored here. While immediacy and contextually fueled reporting is a clear result of the technology and methodology, it can also provide ongoing deep behavioral observation that was not efficient or even possible before due to the physical constraints of the research observers. For example, highly-focused, longitudinal studies might be designed around regular product innovation cycles for a relatively low investment. In other words, the digital nature of the data collected can allow for deeper and richer analysis. Companies can develop Digital Ethno databases for their consumers, which can provide wonderful guidance for innovation over the long term. In addition, Digital Ethno has the capacity to inform strategy and design decisions at a more fundamental level. Companies routinely find ethnography useful because it provides contextsensitive insights regarding existing processes, products, or services. One advantage of Digital Ethno is that it further enhances the benefits of those insights by allowing a company to invest in comprehensive research further back in the product development process. Beyond an initial project investment and equipment costs, Digital Ethno allows for radical expansion of scope with prototypes and stimuli at varying levels of fidelity. Typically, ethnography is expensive and laborintensive. Digital Ethno enables a broader understanding of factors such as culture, geography, and life-stage differences because of reduced field costs. A company can use Digital Ethno to
Interpretation: Theres something about flowers in a steel washtub that appeals to my sense of the absurd. Maybe people grow flowers in steel washtubs. I dont know. I like hokey combinations like this. What does it say about human beings/love/Valentines Day? Perhaps this arrangement appeals to people seeking a down home look for Valentines Day, something thats not too pretentious. Homely, like a stray dog? A little bit of class, a little bit of Hee-Haw?
I found it difficult to capture kisses. They are very intimate in general. This is a picture of me kissing my cat Buster. I must kiss him at least three times a day, and weve been together 10 years. This is a whole lot of kissing! My kissing him is more relevant than any other nonfamilial kiss, as [he has] outlasted many relationships. And I love him a bunch, so its definitely appropriate.
robustly refine segmentations or even develop a powerful segmentation from scratch around a low-fi prototype in order to inform design strategy at a point well prior to commercialization. For example, embedded sensors and other unobtrusive data-gathering tools could be used for a very large group of life-stage-differentiated alpha testers to see how people are interacting with a product on a continuous basis. This means deeper insights sooner rather than later. Finally, Digital Ethno brings the participant back into the research process. Rather than simply acting as sources of data, participants actively share their findings and their insights on the topic at hand. They become invested in the outcome and, as a result, become more-active contributors to the project. There are some drawbacks to Digital Ethno. Until consumer digital technology products like cellular phones, faxes, and digital cameras become common household items, we will tackle a steeper learning and logistics curve bringing the participants into the research process. Likewise, by putting the tools into the hands of the participants, critical privacy issues must be tackled as they arise in each distinct context (see sidebar). In an era in which mass production is giving way to mass customization and personalization, the benefits of Digital Ethno are evident. Increased consumer input into the design, product development, branding, and marketing

processes will lead to greater production efficiency, more frequent innovation, competitive advantage, and perhaps even more responsible consumer, as well as corporate, behavior. The sooner the inevitable merging of citizenship, politics, and consumption is recognized, the better off we will all be in our personal, as well as our professional, lives. Digital Ethno will only enhance the process whereby people vote with their dollars for designs and products they like. Those not listening to these new, more-relevant polls will be left behind in the marketplace. Reprint # 03142MAS75 Find related articles on www.dmi.org with these keywords: Digital technologies, innovation, marketing research, research methods, technology and innovation
Suggested readings Correll, S. The Ethnography of an Electronic Bar: The Lesbian Cafe. Journal of Contemporary Ethnography, vol. 24, no. 3 (1995), pp. 270-298. Marcus, G.E. Ethnography in/of the World System: The Emergence of Multi-Sited Ethnography. Annual Review of Anthropology, vol. 24 (1995), pp. 95-117. Nunes, M. Baudrillard in Cyberspace: Internet, Virtuality, and Postmodernity. In Style, vol. 29 (1995), pp. 314-327. Paccagnella, L. Getting the Seats of Your Pants Dirty: Strategies for Ethnographic Research on Virtual Communities. Journal of ComputerMediated Communication [On-line], vol. 3 (1997), no. 1. Available at: www.ascusc.org/jcmc /vol3/issue1/paccagnella.html#rcroft. Turkle, S. Virtuality and its discontents: Searching for community in cyberspace [40 paragraphs]. The American Prospect [Online serial], vol. 24 (1996), no. 4. Available at: http://epn.org/ prospect/24/24turk.html.