[IEEE 2013 IEEE International Conference on Consumer Electronics (ICCE) - Las Vegas, NV (2013.1.11-2013.1.14)] 2013 IEEE International Conference on Consumer Electronics (ICCE) - AAM-based face reorientation using a single camera

Abstract-- In a video conference, eye-contact between conferees is an important issue because it makes them feel immersive and friendly. An eye-contact feeling does not mean only parallel gaze angle between conferees. For a real eye-contact feeling, it is necessary that the conferee’s face is also perpendicular to the camera so that users feel a face-to-face conversation to the conferee. In this paper, we present a new method to reorient a face for the real eye-contact using a generic 3D face model based on a single camera. I. INTRODUCTION Nowadays, telecommunication is an essential part in our life. Human network is linked complicatedly and enhanced via telecommunication. Development of high-speed internet and wireless communication makes it possible to transfer visual face, as well as voice. People can talk to their friends, family or loved one at the same time seeing their expressions or gestures. In addition, video conferencing has recently activated due to its convenience and cost reduction. However, there is an obstacle for visual telecommunication. The problem is a discomfort feeling causing that the face of a conversation partner in a display does not look natural during visual communication. The discomfort feeling disturbs an immersive conversation and makes a bad effect to their relationship. In the field of computer vision, we call it eye contact issue [1]. It is caused by differences in location of the video conferencing devices that are a camera and a display. In literature, a number of methods have been proposed in order to solve it. These are mainly classified into two categories: mechanical approach and image processing approach. The mechanical approach [2] provides a good performance but it needs extra installation cost, so that the device of video conferencing makes bigger and more expensive. In contrast, the image processing approach [3] doesn’t require the extra cost. However, it hasn’t shown a satisfactory result. Some researchers are just correcting one’s gaze so that they tried to make an eye contact feeling. Recently, others have proposed a hybrid type method [4] using additional cameras. The hybrid method presents a moderate performance but it also causes an additional cost according to the number of the cameras. For video conferencing, fundamental devices are a camera and a display. Conventionally, a commercial webcam is setup to the top of the display. In this paper, we propose an image processing approach to handle the single top camera issue using a generic 3D face model. Our method can be also expanded to the other camera position. We use a piece-wise linear warping method to generate an eye contacted virtual image. Fig.1. Top-attached video conferencing system and eye-contact failure reason II. SYSTEM DESCRIPTION The proposed system is composed of four parts. The first part is a face detection part to track a user’s face and fit a generic 3D face model using the user’s face features. The second part is the determination warping function to generate a reoriented face. The third part is the texture mapping part so that it procedures an inverse mapping method. The fourth part is the distortion protector part for eliminating artifacts, such as eyeglasses distortion and face distortion, of the image. Our system is implemented at desktop computer aided video conferencing. Fig.2. System block diagram III. ALGORITHM A frame captured from webcam goes though the face detection part, where the position and orientation of the face is calculated by the active appearance model (AAM) [5] and fitting 3D generic face model to one’s face as shown in Fig.3. The desired position and orientation is calculated by the rotation angles with respect to 3D coordinate. Vertices of the fitted 3D generic model are very important for the proposed algorithm, and they are tracked every frame. But, 3D face mesh model is too coarse to describe the face in detail. So, we mix 2D face features and 3D mesh features and make a new mesh as shown in Fig.4. Then, they are handled by control  AAM-based Face Reorientation Using a Single Camera Dowan Kim 1 , Sungjin Kim 1 , Ying Huang 2 , Jianfa Zou 2 , JunJun Xiong 2 and Jongsul Min 1 1 Samsung Electronics, Suwon-si, Gyeonggi-do, Korea 2 Samsung Electronics, Chaoyang, Beijing, China 978-1-4673-1363-6/13/$31.00 ©2013 IEEE 2013 IEEE International Conference on Consumer Electronics (ICCE) 258 points of the face texture in the correction step. Using three vertices of every mesh triangle, we can extract the affine transformation matrix from the current frame to desired one. Next, texture of the current frame is mapped onto the generated frame at the virtual view based on the affine transformation. The reoriented face at the virtual view is generated by accumulating all affine transformation, which is often called a piece-wise linear transformation [6]. It is widely used to apply the sophisticated transformation with a low computational burden. Thereafter, a gaze correction technique is applied to emphasize eye contact feeling. It is done by generating an eye- oriented mesh model. The gaze correction consists of two steps. First one is the size-up of middle area of the eye, and second one is the shift-down of tail area of the eye. After warping process, artifacts may be occurred some part of virtually-generated face, such as chin fattening and eyeglasses distortion, as shown in Fig.4. In order to solve these distortions, we propose a distortion protection algorithm. Chin fattening distortion can be reduced by modification mesh points and the degree of modification is determined by face ratio that is invariant factor to the range. The proposed method maintains pixels near eyeglasses to reduce eyeglasses distortion as shown in Fig.4. Fig.5. shows the main algorithm flow of the proposed face reorientation. (a) (b) (c) (d) Fig.3. AAM Face Models (a) 2D features, (b) 3D Mesh Model, (c) Mixed features, (d) Delaunay triangulation Fig.4. Distortion protection(Left: AAM image, Mid: Chin & eyeglasses distortion, Right: distortion adjusting) Fig.5. Main algorithm flow chart IV. EXPERIMENT We used Logitech Web cam C-910 for video conferencing and OpenCV library for image processing. The proposed method was tested at window 7 OS. In VGA Format, frame rate was 25 fps on Intel core I5 processor with CPU Clock 2.66 GHz. Fig 6. shows the result for face reorientation. Fig.6. Up: Eye only reorientation, Down: Face reorientation result (Left: before, Right: after) In order to verify our face reorientation performance, we measured an eye position ratio improvement indicator (EPRII) and head dimension ratio improvement indicator (HDRII)[7]. Our system shows that average EPRII is 0.91 and average HDRII is 0.90 by measurements using 100 frames. V. CONCLUSION In this paper, we have implemented a face reorientation algorithm for video conferencing. The proposed method shows good performance despite light computation and without extra cost. Even eyeglasses wearer person can enjoy our application. We expect that our research can be adapted to other devices or situation, for example, mobile, TV, side-mounted webcam, etc. REFERENCES [1] Steve McNelly. Immersive Group Telepresence and the Perception of Eye Contact. Nature 407, 477–483 (2000) [2] CISCO. Cisco TelePresence 3000 . http://www.cisco.com/ [3] Ben Yip, Jesse S. Jin. An Effective eye gaze correction operation for video conference using anti-rotation formulas. Proceedings of the 2003 Joint Conference of the Fourth International Conference. (2003) [4] Ruigang Yamg. Zjengyou Zhamg. Eye Gaze Correction with Stereovision for Video-Teleconferencing. Pattern Analysis and Machine Intelligence, IEEE Transactions on. Issue 7. 956 - 960 (2004) [5] T. F. Cootes, G. J. Edwards, and C. J. Taylor. Active appearance models. IEEE TPAMI, 23(6):681–685. (2001) [6] Seiichi Uchida, Hiroki Sakoe. Piecewise Linear Two-dimensional Warping. Pattern Recognition 2000 Proceedings. 15th International Conference on, Page(s): 534 - 537 vol.3. (2000) [7] Ben Yip. Eye Contact Rectification In Video Conference With Monocular Camera. PhD Thesis at Sidney University(2007) 259