In order to promote the widespread use of personal mobility (PM) in society, it is necessary to develop a driver support system that can accurately recognize the environment and improve the sense of security, safety, and comfort. This paper proposes a method for supporting the driver by recognizing obstacles and estimating the width of traffic from 3D point cloud obtained by the 3D lidar attached to the PM. To verify the effectiveness of the proposed method, we created a driving course with obstacles and conducted a driving test with 20 subjects. The results of this test show that the proposed method is effective in improving safety and comfort indices.

Abstract

Laser thermal therapy is one of the treatments for malignant tumors. We developed a thermal endoscope using an ultra-compact thermo-sensor and established a new laparoscopic laser thermal therapy system to heat cancer tissue at an appropriate temperature, focusing on the fact that thermographic cameras are capable of two-dimensional temperature mapping. Hepatocellular carcinoma (N1S1) cells were implanted into the livers of Sprague–Dawley rats (n = 13) to create orthotopic hepatocellular carcinoma. Six of the rats underwent laparoscopic laser thermotherapy (70 °C, 5 min) using the newly developed system, and the others underwent laparoscopic insertion only. Lesion volume measurement and histological evaluation were performed in all of the rats. The laparoscopic laser thermal therapy system provided stable temperature control. When a temperature of 70 °C was used for the set temperature, the temperature of the target cancer was maintained within the range of 68–72 °C for 93.2% of the irradiation time (5 min). The median volume of the tumors that were thermally treated was significantly smaller than that of the untreated tumors. The newly developed laparoscopic laser thermal therapy system was capable of maintaining the temperature of the tumor surface at any desired temperature and was proven to be effective in treatment of the rat hepatocellular carcinoma model.

Vision plays an important role in motion planning for mobile robots which coexist with humans. Because a method predicting a pedestrian path with a camera has a trade-off relationship between the calculation speed and accuracy, such a path prediction method is not good at instantaneously detecting multiple people at a distance. In this study, we thus present a method with visual recognition and prediction of transition of human action states to assess the risk of collision for selecting the avoidance target. The proposed system calculates the risk assessment score based on recognition of human body direction, human walking patterns with an object, and face orientation as well as prediction of transition of human action states. First, we investigated the validation of each recognition model, and we confirmed that the proposed system can recognize and predict human actions with high accuracy ahead of 3 m. Then, we compared the risk assessment score with video interviews to ask a human whom a mobile robot should pay attention to, and we found that the proposed system could capture the features of human states that people pay attention to when avoiding collision with other people from vision.

Under the Synecoculture environment, in which various plants are raised in mixed and dense vegetation, automatic maintenance of the field is difficult because of difficulties in separating each harvest. In this project, the situation in which one plant dominates the other plants is called “dominant situation”, and such dominant plants are to be cut. So, in this paper, we propose a method for detecting dominant plants from RGB images using deep learning. First, we partition the original image into small blocks. We perform VGG16 for each small block to predict the number of plants. If the number of the small blocks in each of which the number of plants is less than two exceeds the threshold, the original image is judged as a candidate of “dominant situation”. If the original image is judged as the candidate, similarity between dominant small blocks is computed using AKAZE, and if the similarity is high, the small blocks are judged to be in dominant situation. Experimental results show that high accuracies for estimating dominant situations are achieved.

Purpose: Augmented reality (AR) technology improves the learning process in interventional radiology. This study hypothesized that using AR to train for central venous access is superior to using ultrasound alone. Methods: This study used an AR central venous catheterization phantom with an internal jugular vein (IJV) and subclavian vein (SCV) made of resin body and soft tubing. Ten radiologists attempted to punctuate, using needle placement simulation, under three conditions (ultrasound-, augmented reality-, and ultrasound and AR-guided methods
US-only, AR-only, and US+AR, respectively) using a smart-glass device (HoloLens, Microsoft, Redmond, WA, USA). Subjective (anatomical understanding and self-confidence for procedure) and objective evaluations (optimized needle position and time) were recorded for each condition. Results: The subjective IJV evaluation showed no difference among the guiding methods (p = 0.26 and p = 0.07 for anatomical understanding and self-confidence for procedure, respectively). Conversely, there were significant improvements in subjective and objective evaluations for SCV using the AR-only and US+AR methods (p &lt
 0.05) and US+AR method (p &lt
 0.05), respectively. The AR-only method reduced the time required to fix the needle position to puncture the SCV (p &lt
 0.05), but its objective evaluation did not improve compared with the US-only method (p = 0.20). Conclusion: Adding the AR-guided method to the US-guided method improved subjective and objective evaluations in the SVC procedure. The AR technology-assisted training may be more beneficial for use in difficult procedures. Though the AR-only method saved time, no time saving is expected with AR+US method.

This paper proposes a method for quantitatively evaluating sprinting motions using the videos of runners. Specifically, this paper explores the coordination between physical motions, which has been recognized as very important in sprinting. After detecting and normalizing the joint coordinates from sprinting videos, the cross-correlations of two windowed time-series data are calculated using the windowing cross-correlation function, and the coordination between the motions of the two joints is quantified. Experiments that use 20 subjects are conducted. As a result of classifying the cross-correlation obtained from the subjects' data into two clusters using k-means clustering, conditions in which the obtained cluster includes a high percentage of inexperienced sprinters are found. To verify whether the motions corresponding to these conditions are valid as the evaluation criterion of sprinting, Spearman's rank correlation coefficients between cross-correlations and 30-m time records are calculated. The results show a weak correlation with respect to the coordination between the elbow and knee motions. Therefore, it can be said that the cross-correlation corresponding to the coordination can be used as a quantitative criterion in sprinting.

This paper proposes a method that can classify foot strike types using a deep learning model and can extract unknown factors, which enables to evaluate running motions without being influenced by biases of sports experts, using the contribution degree of input values (CDIV). Accelerometers are attached to the runner's body, and when the runner runs, a fixed camera observes the runner and acquires a video sequence synchronously with the accelerometers. To train a deep learning model for classifying foot strikes, we annotate foot strike acceleration data for RFS (Rearfoot strike) or non-RFS objectively by watching the video. To interpret the unknown factors extracted from the learned model, we calculate two CDIVs: the contributions of the resampling time and the accelerometer value to the output (foot strike type). Experiments on classifying unknown runners' foot strikes were conducted. As a common result to sport science, it is confirmed that the CDIVs contribute highly at the time of the right foot strike, and the sensor values corresponding to the right and left tibias contribute highly to classifying the foot strikes. Experimental results show the right tibia is important for classifying foot strikes. This is because many of the training data represent difference between the two foot strikes in the right tibia. As a conclusion, our proposed method could extract unknown factors from the classifier and could interpret the factors that contain similar knowledge to the prior knowledge of experts, as well as new findings that are not included in conventional knowledge.

As a novel therapy for peritoneal dissemination, it is desired to actualize an endoscopic photothermal therapy, which is minimally invasive and is highly therapeutically effective. However, since the endoscopic tumor temperature control has not been actualized, conventional therapies could damage healthy tissues by overhearing. In this paper, we develop a thermal endoscope system that controls the tumor temperature so that the heated tumor gets necrotic. In fact, our thermal endoscope contains a thermal image sensor, a visible light endoscope and a laser fiber. Concerning the thermal image sensor, the conventional thermal endoscope has the problem that the diameter is too large, because the conventional endoscope loads a large thermal image sensor with high-resolution. Therefore, this paper uses a small thermal image sensor with low resolution, because the diameter of the thermal endoscope needs to be smaller than 15mm in order to be inserted into the trocar. However, this thermal image sensor is contaminated by much noise. Thus, we develop a tumor temperature control system using a feedback control and tumor temperature estimation based on Gaussian function, so that the noisy, small thermal image sensor can be used. As experimental results of the proposed endoscopic photothermal therapy for the hepatophyma carcinoma model of rats, it turns out that the tumor temperature by which the heated tumor gets necrotic can be kept stable. It can be said that our endoscopic photothermal therapy achieves a certain degree of therapy effect.

To identify the difference in performers' motions, this paper investigates the relationship between running motions and the result of evaluating motions during jump training. To clarify the relationship, two experiments were performed using 17 subjects as follows: i) obtaining sequences of human joints during running to evaluate running motions, and ii) obtaining motions during jump training which could skill up the running motions. According to the result of those experiments, we confirmed that whether a running motion is good or not relies greatly on the number of acquired skills.

To improve running performances, each runner's skill, such as characteristics and habits, needs to be known, and feedback on the performance should be outputted according to the runner's skill level. In this paper, we propose a new coaching system for detecting the skill of a runner and a method of giving feedback using a sprint motion dataset. Our proposed method calculates an extracted feature to detect the skill using an autoencoder whose middle layer is an LSTM layer; we analyse the feature using hierarchical clustering, and we analyse the human joints that affect the skill. As a result of experiments, five clusters are obtained using hierarchical clustering. This paper clarifies how to detect the skill and to output feedback to achieve a level of performance one step higher than the current level.

© 2018, Society for Imaging Science and Technology. This paper proposes a guitar fingering assessing system based on CNN (Convolutional Neural Network) hand pose estimation and SVR (Support Vector Regression) evaluation. To spur our progress, first, a CNN architecture is proposed to estimate temporal 3D position of 16 joints of hand; then, based on a DCT (Discrete Cosine Transform) feature and SVR, fingering of guitarist is scored to interpret how well guitarist played. We also release a new dataset for professional guitar playing analysis with significant advantage in total number of video, professional judgement by expert of guitarist, accurate annotation for hand pose and score of guitar performance. Experiments using videos containing multiple persons' guitar plays under different conditions demonstrate that the proposed method outperforms the current state-of-art with (1) low mean error (Euclid distance of 6,1 mm) and high computation efficiency for hand pose estimation; (2) high rank correlation (0.68) for assessing the fingering (C major scale and symmetrical excise) of guitarists.

© 2018, Society for Imaging Science and Technology. Towards the actualization of an air bag system for the UAV's crash, this paper proposes a method for estimating the UAV's crash site from the video sequence acquired by the camera attached to the UAV. The crash point can be considered to correspond to the divergence point of the optical flows. In the accumulator, the cells at which the optical flows (straight lines) pass through are incremented by one. After performing this process for all the optical flows, the cell with the largest vote is obtained as the crash point (divergence point) in the image plane. Experiments using a hand held camera show that the accuracy of estimating the crash site is increased as the camera approaches the target plane. Overall, the experimental results are promising.

© 2018 Society for Imaging Science and Technology. Towards the actualization of an automatic guitar teaching system that can supervise guitar players, this paper proposes an algorithm for accurately and robustly tracking the 3D position of the fretboard from the video of guitar plays. First, we detect the SIFT features within the guitar fretboard and then match the detected points using KD-tree searching based matching algorithm frame by frame to track the whole fretboard. However, during the guitar plays, due to movements of the guitar neck or occlusions caused by guitar players' fingers, the feature points on the fretboard cannot always be matched accurately even though applying traditional RANSAC homography. Therefore, by using our modified RANSAC algorithm to filter out the matching error of the feature points, perspective transformation matrix is obtained between the correctly matched feature points detected at the first and other frames. Consequently, the guitar neck is tracked correctly based on the perspective transformation matrix. Experiments show promising results such as high accuracy: the total mean tracking error of only 4.17 mm and variance of 1.5 for the four tracked corners of the fretboard. This indicates the proposed method outperforms related tracking works including state-of-art Fully-convolutional Network.

© 2018, Society for Imaging Science and Technology. Towards the establishment of the preventive medical care for the cerebral aneurysm, this paper proposes an SVM based method for building a discrimination function that classifies the presence or absence of the cerebral aneurysm using the cerebral blood vessel's shape features obtained from medical images such as MR images. Using the discrimination function, this paper explores how much each feature affects the onset of the cerebral aneurysm. This paper deals with the internal carotid artery (ICA). The blood vessel (ICA)'s shape features are extracted from medical images of 18 persons without cerebral aneurysm and 13 patients with a cerebral aneurysm. From the medical image, the cross sections and centerline of the ICA are obtained. The cross sections are divided into nine sections along the centerline. Shape features such as the cross sectional area, its circularity, curvature, torsion, length of the centerline and branch angles are obtained in each section; as a total, 113 features including the mean and variance of some features in each section are used for building the SVM. As a result of conducting the experiments, the accuracy for discriminating the presence/absence of the aneurysm by the SVM is 90.3%. In the obtained discrimination function, the coefficient values of the function can be considered how much the features affect the onset of the aneurysm. The features that could significantly cause the onset of the cerebral aneurysm are clarified, and the reasons why these features are significant are discussed.

© 2018, Society for Imaging Science and Technology. To contribute to the actualization of the care worker assistance robot, this paper proposes a method for detecting whether the care receiver is chewing from the video sequence acquired by the camera that observes that receiver. The proposed method detects the receiver's face and areas for both cheeks and chin. After applying some normalization to the areas, chewing detection that uses a variable-intensity template is performed, where the template consists of shape models, interest points and intensity distribution model. A likelihood based on the variableintensity template is computed so that the receiver is judged whether the receiver is chewing. Experiments using seven subjects are conducted. As a result, the accuracy of chewing detection by the proposed method is 83%, which is quite promising.

To alleviate the recent shortage problem of nurses, the actualization of RSN (Robotic Scrub Nurse) that can autonomously judge the current step of the surgery and pass the surgical instruments needed for the next step to surgeons is desired. The authors developed a computer vision based algorithm that can early-recognize only two steps of suture surgery. Based on the past work, this paper explores the effectiveness of utilizing temporal order of the six steps in suture surgery for the early-recognition. Our early-recognition algorithm consists of two modules: start point detection and hand action early-recognition. Segments of the test video that start from each quasi-start point are compared with the training data, and their probabilities are calculated. According to the calculated probabilities, hand actions could be early-recognized. To improve the early-recognition accuracy, temporal order information could be useful. This paper checks confusions of three steps' early recognition results, and if necessary, early-recognizes again after eliminating the wrong result, while for the other three steps, temporal order information is not utilized. Experimental results show our early-recognition method that utilizes the temporal order information achieves better performances.

This paper explores the effectiveness of applying a deep learning based method to segment the amniotic fluid and fetal tissues in fetal ultrasound (US) images. The deeply learned model firstly encodes the input image into down scaled feature maps by convolution and pooling structures, then up-scale the feature maps to confidence maps by corresponded un-pooling and convolution layers. Additional convolution layers with 1×1 sized kernels are adopted to enhance the feature representations, which could be used to further improve the discriminative learning of our model. We effectively update the weights of the network by fine-tuning on part of the layers from a pre-trained model. By conducting experiments using clinical data, the feasibility of our proposed approach is compared and discussed. The result proves that this work achieves satisfied results for segmentation of specific anatomical structures from US images.

This paper proposes a temporal Grouping and pattern analysis-based algorithm that could track the fingertips of guitarists during their guitar playing towards the actualization of the automatic guitar fingering recognition system. First a machine learning-based Bayesian Pixel Classifier is used to segment the hand area on the test data. Then, the probability map of fingertip is generated on the segmentation results by counting the voting numbers of the Template Matching and Reversed Hough Transform. Furthermore, a temporal Grouping algorithm, which is a geometry analysis for consecutive frames, is applied to removal noise and group the same fingertips (index finger, middle finger, ring finger, little finger). Then, a data association algorithm is utilized to associate 4 tracked fingers (index finger, middle finger, ring finger, little finger) with their correspondent tracked results frame by frame. Finally, particles are distributed only between the associated fingertip candidates to track the fingertips of guitarist effectively. The experimental result demonstrates that this fingertip tracking algorithm is robust enough for tracking fingertips (1) without any constrains such us color marker
(2) under the complex contexts, such us complicated background, different illumination conditions, (3) with the high tracking accuracy (mean error 3.36 pixels for four fingertips).

This paper proposes a temporal Grouping and pattern analysis-based algorithm that could track the fingertips of guitarists during their guitar playing towards the actualization of the automatic guitar fingering recognition system. First a machine learning-based Bayesian Pixel Classifier is used to segment the hand area on the test data. Then, the probability map of fingertip is generated on the segmentation results by counting the voting numbers of the Template Matching and Reversed Hough Transform. Furthermore, a temporal Grouping algorithm, which is a geometry analysis for consecutive frames, is applied to removal noise and group the same fingertips (index finger, middle finger, ring finger, little finger). Then, a data association algorithm is utilized to associate 4 tracked fingers (index finger, middle finger, ring finger, little finger) with their correspondent tracked results frame by frame. Finally, particles are distributed only between the associated fingertip candidates to track the fingertips of guitarist effectively. The experimental result demonstrates that this fingertip tracking algorithm is robust enough for tracking fingertips (1) without any constrains such us color marker
(2) under the complex contexts, such us complicated background, different illumination conditions, (3) with the high tracking accuracy (mean error 3.36 pixels for four fingertips).

Robotic Scrub Nurse (RSN) is a developing research and a challenging issue in the field of medical robotics. Audio processing, wearing-sensor and computer vision are applied to control the robot that could support surgeons during surgeries as an alternative of human nurse. On the other hand, surgeons wish that RSN should be able to decide the correct action by recognizing surgical situations autonomously just like skilled human nurses. In general, surgical situations can be discriminated by surgeons hand actions during surgeries. This article introduces and explains the authors computer vision based method for recognizing surgical hand actions of surgeons from video sequences.

Towards the actualization of an autonomous guitar teaching system, this paper proposes the following two video analysis based methods: (1) pressed chord recognition and (2) fingertip tracking. For (1), an algorithm that can extract finger contours and chord changes is proposed so that the chords pressed by the guitar player are recognized. For (2), an algorithm that can track the fingertips by continuously monitoring the appearance and disappearance of the regions of fingertip candidates is proposed. Experimental results demonstrate that the proposed two modules are robust enough under complex contexts such as complicated background and different illumination conditions. Promising results were obtained for accurate tracking of fingertips and for accurate recognition of pressed chords.

Towards the actualization of an autonomous guitar teaching-aid system, this paper proposes the following two video analysis based methods: (1) guitar neck detecting and (2) guitar neck tracking. For (1), an algorithm that can extract the rectangles cut by guitar strings and fret is proposed so that we can detect guitar neck area on the first frame of the input video. For (2), a feature point based guitar neck tracking algorithm is proposed, which can cope with occlusion issue effectively by analyzing geometric relationship between each two feature points in every frame. Experimental results demonstrate that the proposed two modules are robust enough under complex contexts such as complicated background and different illumination conditions.

To enable clean-up robots to vacuum cluttered floors, the robots should be able to return objects on the floors to the proper positions by pushing the objects. This paper proposes a computer vision based method that can detect the objects (clutter) and evaluate the result of the mechanical operation (pushing the objects). First, we construct a dense 3D map of a tidy situation of the room (after-clean-map). Next, we construct another dense 3D map of the cluttered situation of the same room (before-clean-map). Then, we construct the 3D point cloud of the cluttered situation by comparing the after-clean-map and the before-clean-map. Finally, we let a clean-up robot push the cluttered object and evaluate whether the object is successfully moved. Experimental results using real RGBD image sequences demonstrate promising results of our proposed method.

The purpose of abstract painters is to let viewers get the various images and abstract images. However, viewers who do not have enough knowledge of art, cannot easily get abstract images. The authors have proposed a music generation system that utilizes viewers’ gazes. It can be expected that the authors’ music generation system can prompt the viewer of abstract paintings to imagine abstract images, which the painter intended to express. This paper explores whether the authors’ music generation system can enhance abstract imaginations of persons who see abstract paintings, by subjective tests. Experiments using 19 subjects and eight abstract paintings were conducted for the two cases in which the subjects see the abstract paintings without hearing any music and while hearing the viewers’ gaze based music generated by the authors’ system. Experimental results imply that “hearing gaze based music” could enhance the viewers’ abstract imagination.

The purpose of abstract painters is to let viewers get the various images and abstract images. However, viewers who do not have enough knowledge of art, cannot easily get abstract images. The authors have proposed a music generation system that utilizes viewers’ gazes. It can be expected that the authors’ music generation system can prompt the viewer of abstract paintings to imagine abstract images, which the painter intended to express. This paper explores whether the authors’ music generation system can enhance abstract imaginations of persons who see abstract paintings, by subjective tests. Experiments using 19 subjects and eight abstract paintings were conducted for the two cases in which the subjects see the abstract paintings without hearing any music and while hearing the viewers’ gaze based music generated by the authors’ system. Experimental results imply that “hearing gaze based music” could enhance the viewers’ abstract imagination.

This paper proposes a method that combines Temporal Modified-RANSAC(TMR) with a fixation-based segmentation algorithm for reconstructing the 3D structure of moving and still objects that are tracked in video and depth image sequences acquired by moving Kinect© and/or range finders First, we compute 3D optical flow of feature points. Second, TMR classifies all flows into consistent 3D flow sets for the background and each moving object
simultaneously, the rotation matrix and the translation vector for each 3D flow set are computed. Then, an improved fixation-based segmentation method segments each object’s area. Finally, dense 3D models for the background and each moving object are constructed along with each object’s rotation matrix and translation vector in each frame. Experiments using multiple moving objects in color and depth image sequences acquired by Kinect(c) demonstrate the effectiveness of our proposed method.

Toward the actualization of an automatic navigation system for fetoscopic tracheal occlusion (FETO) surgery, this paper proposes a 3D ultrasound (US) calibration-based approach that can locate the fetal facial surface, oral cavity, and airways by a registration between a 3D fetal model and 3D US images. The proposed approach consists of an offline process and online process. The offline process first reconstructs the 3D fetal model with the anatomies of the oral cavity and airways. Then, a point-based 3D US calibration system based on real-time 3D US images, an electromagnetic (EM) tracking device, and a novel cones' phantom, computes the matrix that transforms the 3D US image space into the world coordinate system. In the online process, by scanning the mother's body with a 3D US probe, 3D US images containing the fetus are obtained. The fetal facial surface extracted from the 3D US images is registered to the 3D fetal model using an ICP-based (iterative closest point) algorithm and the calibration matrices, so that the fetal facial surface as well as the oral cavity and airways are located. The results indicate that the 3D US calibration system achieves an FRE (fiducial registration error) of 1.49 +/- 0.44 mm and a TRE (target registration error) of 1.81 +/- 0.56 mm by using 24 fiducial points from two US volumes. A mean TRE of 1.55 +/- 0.46 mm is also achieved for measuring location accuracy of the 3D fetal facial surface extracted from 3D US images by 14 target markers, and mean location errors of 2.51 +/- 0.47 mm and 3.04 +/- 0.59 mm are achieved for indirectly measuring location accuracy of the pharynx and the entrance of the trachea, respectively, which satisfy the requirement of the FETO surgery.

Because of the shortage of nurses in the world, the realization of a robotic nurse that can support surgeries autonomously is very important. More specifically, the robotic nurse should be able to autonomously recognize different situations of surgeries so that the robotic nurse can pass necessary surgical tools to the medical doctors in a timely manner. This paper proposes and explores methods that can classify suture and tying actions during suture operations from the video sequence that observes the surgery scene that includes the surgeon's hands.
First, the proposed method uses skin pixel detection and foreground extraction to detect the hand area. Then, interest points are randomly chosen from the hand area so that their 3D SIFT descriptors are computed. A word vocabulary is built by applying hierarchical K-means to these descriptors, and the words' frequency histogram, which corresponds to the feature space, is computed Finally, to classify the actions, either SVM (Support Vector Machine), Nearest Neighbor rule (NN) for the feature space or a method that combines "sliding window" with NN is performed.
We collect 53 suture videos and 53 tying videos to build the training set and to test the proposed method experimentally It turns out that the NN gives higher than 90% accuracies, which are better recognition than SVM. Negative actions, which are different from either suture or tying action, are recognized with quite good accuracies, while "Sliding window" did not show significant improvements for suture and tying and cannot recognize negative actions.

While the Optical Music Recognition (OMR) of printed and handwritten music scores in modern standard notation has been broadly studied, this is not the case for early music manuscripts. This is mainly due to the high variability in the sources introduced by their severe physical degradation, the lack of notation standards and, in the case of the scanned versions, by non-homogenous image-acquisition protocols. The volume of early musical manuscripts available is considerable, and therefore we believe that computational methods can be extremely useful in helping to preserve, share and analyse this information. This paper presents an approach to recognizing handwritten square musical notation in degraded western plainchant manuscripts from the XIVth to XVIth centuries. We propose the use of image processing techniques that behave robustly under high data variability and which do not require strong hypotheses regarding the condition of the sources. The main differences from traditional OMR approaches are our avoidance of the staff line removal stage and the use of grey-level images to perform primitive segmentation and feature extraction. We used 136 images from the Digital Scriptorium repository (DS, 2007), from which we were able to extract over 90% of the staves and over 88% of all symbols present. For symbol classification, we used gradient-based features and SVM classifiers, obtaining over 90% precision and recall over eight basic symbol classes.

Because of the shortage of nurses in the world, the realization of a robotic nurse that can support surgeries autonomously is very important. More specifically, the robotic nurse should be able to autonomously recognize different situations of surgeries so that the robotic nurse can pass necessary surgical tools to the medical doctors in a timely manner. This paper proposes and explores methods that can classify suture and tying actions during suture operations from the video sequence that observes the surgery scene that includes the surgeon's hands. First, the proposed method uses skin pixel detection and foreground extraction to detect the hand area. Then, interest points are randomly chosen from the hand area so that their 3D SIFT descriptors are computed. A word vocabulary is built by applying hierarchical K-means to these descriptors, and the words frequency histogram, which corresponds to the feature space, is computed. Finally, to classify the actions, either SVM (Support Vector Machine), Nearest Neighbor rule (NN) for the feature space or a method that combines sliding window with NN is performed. We collect 53 suture videos and 53 tying videos to build the training set and to test the proposed method experimentally. It turns out that the NN gives higher than 90% accuracies, which are better recognition than SVM. Negative actions, which are different from either suture or tying action, are recognized with quite good accuracies, while Sliding window did not show significant improvements for suture and tying and cannot recognize negative actions. © 2014 SPIE.

This paper explores the effectiveness of prompting abstract paintings' viewers' inspiration and imagination by the authors' gaze based music generation system. The authors' music generation system detects the viewer's gaze by a gaze detection equipment. At each of the gaze staying positions in the painting, the color of that point is converted to the sound so that as the gaze moves, music that consists of the converted time series sounds is generated. Experiments using six subjects and six abstract paintings were conducted for the three cases in which the subjects see the abstract paintings without hearing any music, while hearing pre-selected music and while hearing the viewers' gaze based music generated by the authors' system. The experimental results imply that "hearing gaze based music" could stimulate the viewers' inspiration and imagination best, "hearing pre-selected music" second best, and "without music" third best. © Springer International Publishing Switzerland 2014.

Toward the actualization of an automatic navigation system for fetoscopic tracheal occlusion (FETO) surgery, this paper proposes a 3D ultrasound (US) calibration-based approach that can locate the fetal facial surface, oral cavity, and airways by a registration between a 3D fetal model and 3D US images. The proposed approach consists of an offline process and online process. The offline process first reconstructs the 3D fetal model with the anatomies of the oral cavity and airways. Then, a point-based 3D US calibration system based on real-time 3D US images, an electromagnetic (EM) tracking device, and a novel cones' phantom, computes the matrix that transforms the 3D US image space into the world coordinate system. In the online process, by scanning the mother's body with a 3D US probe, 3D US images containing the fetus are obtained. The fetal facial surface extracted from the 3D US images is registered to the 3D fetal model using an ICP-based (iterative closest point) algorithm and the calibration matrices, so that the fetal facial surface as well as the oral cavity and airways are located. The results indicate that the 3D US calibration system achieves an FRE (fiducial registration error) of 1.49±0.44mm and a TRE (target registration error) of 1.81±0.56mm by using 24 fiducial points from two US volumes. A mean TRE of 1.55±0.46 mm is also achieved for measuring location accuracy of the 3D fetal facial surface extracted from 3D US images by 14 target markers, and mean location errors of 2.51±0.47 mm and 3.04±0.59 mm are achieved for indirectly measuring location accuracy of the pharynx and the entrance of the trachea, respectively, which satisfy the requirement of the FETO surgery. © 2014 The Institute of Electronics, Information and Communication Engineers.

Since long time ago, speech recognition has been researched, though it does not work well in noisy places such as in the car or in the train. In addition, people with hearing-impaired or difficulties in hearing cannot receive benefits from speech recognition. To recognize the speech automatically, visual information is also important. People understand speeches from not only audio information, but also visual information such as temporal changes in the lip shape. A vision based speech recognition method could work well in noisy places, and could be useful also for people with hearing disabilities.
In this paper, we propose an automatic lip-reading method for recognizing the speech by using multimodal visual information without using any audio information such as speech recognition. First, the ASM (Active Shape Model) is used to track and detect the face and lip in a video sequence. Second, the shape, optical flow and spatial frequencies of the lip features are extracted from the lip detected by ASM. Next, the extracted multimodal features are ordered chronologically so that Support Vector Machine is performed in order to learn and classify the spoken words. Experiments for classifying several words show promising results of this proposed method.

Recently, a minimally invasive surgery (MIS) called fetoscopic tracheal occlusion (FETO) was developed to treat severe congenital diaphragmatic hernia (CDH) via fetoscopy, by which a detachable balloon is placed into the fetal trachea for preventing pulmonary hypoplasia through increasing the pressure of the chest cavity. This surgery is so dangerous that a supporting system for navigating surgeries is deemed necessary. In this paper, to guide a surgical tool to be inserted into the fetal trachea, an automatic approach is proposed to detect and track the fetal face and mouth via fetoscopic video sequencing. More specifically, the AdaBoost algorithm is utilized as a classifier to detect the fetal face based on Haar-like features, which calculate the difference between the sums of the pixel intensities in each adjacent region at a specific location in a detection window. Then, the CamShift algorithm based on an iterative search in a color histogram is applied to track the fetal face, and the fetal mouth is fitted by an ellipse detected via an improved iterative randomized Hough transform approach. The experimental results demonstrate that the proposed automatic approach can accurately detect and track the fetal face and mouth in real-time in a fetoscopic video sequence, as well as provide an effective and timely feedback to the robot control system of the surgical tool for FETO surgeries.

A new approach is proposed for estimating illuminant colors from color images under an unknown scene illuminant. The approach is based on a combination of a gray-world-assumption-based illuminant color estimation method and a method using color gamuts. The former method, which is one we had previously proposed, improved on the original method that hypothesizes that the average of all the object colors in a scene is achromatic. Since the original method estimates scene illuminant colors by calculating the average of all the image pixel values, its estimations are incorrect when certain image colors are dominant. Our previous method improves on it by choosing several colors on the basis of an opponent-color property, which is that the average color of opponent colors is achromatic, instead of using all colors. However, it cannot estimate illuminant colors when there are only a few image colors or when the image colors are unevenly distributed in local areas in the color space. The approach we propose in this paper combines our previous method and one using high chroma and low chroma gamuts, which makes it possible to find colors that satisfy the gray world assumption. High chroma gamuts are used for adding appropriate colors to the original image and low chroma gamuts are used for narrowing down illuminant color possibilities. Experimental results obtained using actual images show that even if the image colors are localized in a certain area in the color space, the illuminant colors are accurately estimated, with smaller estimation error average than that generated in the conventional method. © 2013 SPIE.

In fetoscopic tracheal occlusion (FETO) surgery, the localization of a flexible tool has an important role in easing high demands for surgeons. In this paper, a point-based 3D ultrasound (US) calibration system is proposed for localizing the surgical tool, based on real-time 3D US images, an electromagnetic (EM) tracking device, and a novel cones' phantom. A 3D US probe attached with an EM sensor is used to acquire the 3D US images of the surgical tool
meanwhile, an EM transmitter is fixed at a certain point in the environment as the origin of the world coordinate system. From these 3D US images, the skeleton of the surgical tool is extracted, and 15 labeled points along the surgical tool are then distinguished, whose coordinates in the world coordinate system are calculated by the proposed 3D US calibration system. The results demonstrate that our proposed approach achieves high localization accuracy, i.e. the average TRE of 1.99 ± 0.49 mm and the maximum TRE of 2.61 ± 0.49 mm, which satisfy the requirement of the FETO surgery: less than 3.85 mm. © Springer-Verlag 2013.

With the wide clinical application of 3D ultrasound (US) imaging, automatic location of fetal facial features from US volumes for navigating fetoscopic tracheal occlusion (FETO) surgeries becomes possible, which plays an important role in reducing surgical risk. In this paper, we propose a feature-based method to automatically detect 3D fetal face and accurately locate key facial features without any priori knowledge or training data. The candidates of the key facial features, such as the nose, eyes, nose upper bridge and upper lip are detected by analyzing the mean and Gaussian curvatures of the facial surface. Each feature is gradually identified from the candidates by a boosting traversal scheme based on the spatial relations between each feature. In experiments, all key feature points are detected for each case, and thus a detection success rate of 100% is achieved by using 72 3D US images from a test database of 6 fetal faces in the frontal view and any pose within 15 degrees from the frontal view, and the location error 3.18 +/- 0.91 inn of the detected upper lip for all test data is obtained, which can be tolerated by the FETO surgery. Moreover, this system has a high efficiency and can detect all key facial features in about 625 ms on a quadcore 2.60 GHz computer.

An investigation into encouraging higher user self-disclosure during text-based Computer Mediated Communication (CMC) system known as the Prompter Communication system (PCS), proposed by Yonemura et al. in previous studies, in which 1 to 3 black circular prompter images ("•") are embedded onto the background of a text-field canvas as stimuli and as an embryonic basis for conversation, have been investigated and evaluated. The effects on supporting the projection or reflection of generated mental representations onto messages with the "•" prompter image as part of the representations within the message contents have also been examined. During the investigation, the concepts and features of the Geneplore cognitive creative thinking process model, together with self-projections, were integrated into the proposed system. Based on these concepts and features, the PCS aims to encourage users to express their mental representations by projecting or reflecting them onto the messages in combination with the provided "•" prompter image as part of the message contents. Communication experiments using the prototype PCS system have been conducted, where a Normal Communication System, NCS which uses only a plain text-field canvas without the "•" prompter image, is compared with the PCS. The experimental results indicate that communication conducted using the PCS results in an increased number of messages with the "•" prompter image(s) involved as part of the message contents than as background images. Further analysis into the self-disclosure level of the message contents and the relationships between the degree of self-disclosure and the representations of the "•" prompter image(s) within the messages have indicated that the degrees of self-disclosure increases when the "•" prompter image(s) are used to represent the participant him/herself, or the participant's thoughts and emotions, and decreases when the "•" prompter image(s) are used to represent animals or objects, or when the "•" prompter image(s) presented within the messages have been ignored. Investigations into the differences in self-disclosure level between message contents from both the PCS and the NCS have shown that using the PCS results in a generally higher degree in the average frequency of the appearance of high self-disclosure information than when using the NCS. During the conversation, these messages contain information with a high level of self-disclosure, which, consequently, may further lead to the development of intimate and deep interpersonal relationships.

In fetoscopic tracheal occlusion (FETO) surgery, the localization of a flexible tool has an important role in easing high demands for surgeons. In this paper, a point-based 3D ultrasound (US) calibration system is proposed for localizing the surgical tool, based on real-time 3D US images, an electromagnetic (EM) tracking device, and a novel cones' phantom. A 3D US probe attached with an EM sensor is used to acquire the 3D US images of the surgical tool
meanwhile, an EM transmitter is fixed at a certain point in the environment as the origin of the world coordinate system. From these 3D US images, the skeleton of the surgical tool is extracted, and 15 labeled points along the surgical tool are then distinguished, whose coordinates in the world coordinate system are calculated by the proposed 3D US calibration system. The results demonstrate that our proposed approach achieves high localization accuracy, i.e. the average TRE of 1.99 ± 0.49 mm and the maximum TRE of 2.61 ± 0.49 mm, which satisfy the requirement of the FETO surgery: less than 3.85 mm. © Springer-Verlag 2013.

A new approach is proposed for estimating illuminant colors from color images under an unknown scene illuminant. The approach is based on a combination of a gray-world-assumption-based illuminant color estimation method and a method using color gamuts. The former method, which is one we had previously proposed, improved on the original method that hypothesizes that the average of all the object colors in a scene is achromatic. Since the original method estimates scene illuminant colors by calculating the average of all the image pixel values, its estimations are incorrect when certain image colors are dominant. Our previous method improves on it by choosing several colors on the basis of an opponent-color property, which is that the average color of opponent colors is achromatic, instead of using all colors. However, it cannot estimate illuminant colors when there are only a few image colors or when the image colors are unevenly distributed in local areas in the color space. The approach we propose in this paper combines our previous method and one using high chroma and low chroma gamuts, which makes it possible to find colors that satisfy the gray world assumption. High chroma gamuts are used for adding appropriate colors to the original image and low chroma gamuts are used for narrowing down illuminant color possibilities. Experimental results obtained using actual images show that even if the image colors are localized in a certain area in the color space, the illuminant colors are accurately estimated, with smaller estimation error average than that generated in the conventional method. © 2013 SPIE.

Recently, a minimally invasive surgery (MIS) called fetoscopic tracheal occlusion (FETO) was developed to treat severe congenital diaphragmatic hernia (CDH) via fetoscopy, by which a detachable balloon is placed into the fetal trachea for preventing pulmonary hypoplasia through increasing the pressure of the chest cavity. This surgery is so dangerous that a supporting system for navigating surgeries is deemed necessary. In this paper, to guide a surgical tool to be inserted into the fetal trachea, an automatic approach is proposed to detect and track the fetal face and mouth via fetoscopic video sequencing. More specifically, the AdaBoost algorithm is utilized as a classifier to detect the fetal face based on Haar-like features, which calculate the difference between the sums of the pixel intensities in each adjacent region at a specific location in a detection window. Then, the CamShift algorithm based on an iterative search in a color histogram is applied to track the fetal face, and the fetal mouth is fitted by an ellipse detected via an improved iterative randomized Hough transform approach. The experimental results demonstrate that the proposed automatic approach can accurately detect and track the fetal face and mouth in real-time in a fetoscopic video sequence, as well as provide an effective and timely feedback to the robot control system of the surgical tool for FETO surgeries.

With the wide clinical application of 3D ultrasound (US) imaging, automatic location of fetal facial features from US volumes for navigating fetoscopic tracheal occlusion (FETO) surgeries becomes possible, which plays an important role in reducing surgical risk. In this paper, we propose a feature-based method to automatically detect 3D fetal face and accurately locate key facial features without any priori knowledge or training data. The candidates of the key facial features, such as the nose, eyes, nose upper bridge and upper lip are detected by analyzing the mean and Gaussian curvatures of the facial surface. Each feature is gradually identified from the candidates by a boosting traversal scheme based on the spatial relations between each feature. In experiments, all key feature points are detected for each case, and thus a detection success rate of 100% is achieved by using 72 3D US images from a test database of 6 fetal faces in the frontal view and any pose within 15 degrees from the frontal view, and the location error 3.18 +/- 0.91 inn of the detected upper lip for all test data is obtained, which can be tolerated by the FETO surgery. Moreover, this system has a high efficiency and can detect all key facial features in about 625 ms on a quadcore 2.60 GHz computer.

This paper proposes a method for reconstructing 3D detailed structures of internal organs such as gastric wall from endoscopic video sequences. The proposed method consists of the four major steps: Feature-point-based 3D reconstruction, 3D point cloud stitching, dense point cloud creation and Poisson surface reconstruction. Before the first step, we partition one video sequence into groups, where each group consists of two successive frames (image pairs), and each pair in each group contains one overlapping part, which is used as a stitching region. Fist, the 3D point cloud of each group is reconstructed by utilizing structure from motion (SFM). Secondly, a scheme based on SIFT features registers and stitches the obtained 3D point clouds, by estimating the transformation matrix of the overlapping part between different groups with high accuracy and efficiency. Thirdly, we select the most robust SIFT feature points as the seed points, and then obtain the dense point cloud from sparse point cloud via a depth testing method presented by Furukawa. Finally, by utilizing Poisson surface reconstruction, polygonal patches for the internal organs are obtained. Experimental results demonstrate that the proposed method achieves a high accuracy and efficiency for 3D reconstruction of gastric surface from an endoscopic video sequence.

This paper proposes a method for tracking and recognizing the white line marked in the surface of the road from the video sequence acquired by the camera attached to a walking human, towards the actualization of an automatic navigation system for the visually handicapped. Our proposed method consists of two main modules: (1) Particle Filter based module for tracking the white line, and (2) CLAFIC Method based module for classifying whether the tracked object is the white line. In (1), each particle is a rectangle, and is described by its centroid's coordinates and its orientation. The likelihood of a particle is computed based on the number of white pixels in the rectangle. In (2), in order to obtain the ranges (to be used for the recognition) for the white line's length and width, Principal Component Analysis is applied to the covariance matrix obtained from valid sample particles. At each frame, PCA is applied to the covariance matrix constructed from particles with high likelihood, and if the obtained length and width are within the above-mentioned ranges, it is recognized as the white line. Experimental results using real video sequences show the validity of the proposed method.

This paper improves the authors' conventional method for reconstructing the 3D structure of moving and still objects that are tracked in the video and/or depth image sequences acquired by moving cameras and/or range finder. The authors proposed a Temporal Modified-RANSAC based method [1] that (1) can discriminate each moving object from the still background in color image and depth image sequences acquired by moving stereo cameras or moving range finder, (2) can compute the stereo cameras' egomotion, (3) can compute the motion of each moving object, and (4) can reconstruct the 3D structure of each moving object and the background. However, the TMR based method has the following two problems concerning the 3D reconstruction: lack of accuracy of segmenting into each object's region and sparse 3D reconstructed points in each object's region. To solve these problems of our conventional method, this paper proposes a new 3D segmentation method that utilizes Graph-cut, which is frequently used for segmentation tasks. First, the proposed method tracks feature points in the color and depth image sequences so that 3D optical flows of the feature points in every N frames are obtained. Then, TMR classifies all the obtained 3D optical flows into regions (3D flow set) for the background and each moving object; simultaneously, the rotation matrix and the translation vector for each 3D flow set are computed. Next, Graph-Cut using the energy function that consists of color probability, structure probability and a-priori probability is performed so that pixels in each frame are segmented into object regions and the background region. Finally, 3D point clouds are obtained from the segmentation result image and depth image, and then the point clouds are merged using the rotation and translation from the N-th frame prior to the current frame so that 3D models for the background and each moving object are constructed with dense 3D point data.

Human gesture as a natural interface plays an utmost important role for achieving intelligent Human Computer Interaction (HCI). Human gestures include different components of visual actions such as motion of hands, facial expression, and torso, to convey meaning. So far, in the field of gesture recognition, most previous works have focused on the manual component of gestures. In this paper, we present an appearance-based multimodal gesture recognition framework, which combines the different groups of features such as facial expression features and hand motion features which are extracted from image frames captured by a single web camera. We refer 12 classes of human gestures with facial expression including neutral, negative and positive meanings from American Sign Languages (ASL). We combine the features in two levels by employing two fusion strategies. At the feature level, an early feature combination can be performed by concatenating and weighting different feature groups, and PLS is used to choose the most discriminative elements by projecting the feature on a discriminative expression space. The second strategy is applied on decision level. Weighted decisions from single modalities are fused in a later stage. A condensation-based algorithm is adopted for classification. We collected a data set with three to seven recording sessions and conducted experiments with the combination techniques. Experimental results showed that facial analysis improve hand gesture recognition, decision level fusion performs better than feature level fusion.

Human gesture as a natural interface plays an utmost important role for achieving intelligent Human Computer Interaction (HCI). Human gestures include different components of visual actions such as motion of hands, facial expression, and torso, to convey meaning. So far, in the field of gesture recognition, most previous works have focused on the manual component of gestures. In this paper, we present an appearance-based multimodal gesture recognition framework, which combines the different groups of features such as facial expression features and hand motion features which are extracted from image frames captured by a single web camera. We refer 12 classes of human gestures with facial expression including neutral, negative and positive meanings from American Sign Languages (ASL). We combine the features in two levels by employing two fusion strategies. At the feature level, an early feature combination can be performed by concatenating and weighting different feature groups, and PLS is used to choose the most discriminative elements by projecting the feature on a discriminative expression space. The second strategy is applied on decision level. Weighted decisions from single modalities are fused in a later stage. A condensation-based algorithm is adopted for classification. We collected a data set with three to seven recording sessions and conducted experiments with the combination techniques. Experimental results showed that facial analysis improve hand gesture recognition, decision level fusion performs better than feature level fusion.

The characteristics of line drawing video communication are described in this paper. In the experiment, an original video and a line drawing video were compared. Subjects performed the same task using both video systems and their performances were assessed in terms of the number of utterances in the conversation and the subject's switching pause. The subject's conversation strategy was extracted based on gaze measurement. In the case of the subujects who often gaze at a partner's eyes during conversation, the switching pause for the line drawing video was significantly longer than for the original video. In both video formats, subjects who does not gaze at a partner's eyes had more switching pauses and utterances than the subjects who do.

This paper proposes a gray world assumption based method for estimating an illuminant color from an image by hue categorization. The gray world assumption hypothesizes that the average color of all the objects in a scene is gray. However, it is difficult to estimate an illuminant color correctly if the colors of the objects in a scene are dominated by certain colors. To solve this problem, our method uses the opponent color properties that the average of a pair of opponent colors is gray. Thus our method roughly categorizes the colors derived from the image based on hue and selects them one by one from the hue categories until selected colors satisfy the gray world assumption. In our experiments, we used three kinds of illuminants (i.e., CIE standard illuminants A and D-65, and a fluorescent light) and two kinds of data sets. One data set satisfies the gray world assumption, and the other does not. Experiment results show that estimated illuminants are closer to the correct ones than those obtained with the conventional method and the estimation error for both using CIE standard illuminants A and D-65 by our method are within the barely noticeable difference in human color perception.

While Optical Music Recognition (OMR) of modern printed and handwritten documents is considered a solved problem, with many commercial systems available today, the OMR of ancient musical manuscripts still remains an open problem. In this paper we present a system for the OMR of degraded western plainchant manuscripts in square notation from the XIV to XVI centuries. The system has two main blocks, the first one deals with symbol extraction and recognition, while the second one acts as an error detection stage for the first block outputs. For symbol extraction we use widely known image-processing techniques, such as Sobel filtering and Hough Transform, and SVM for classification. The error detection stage is implemented with a hidden Markov model (HMM), which takes advantage of a priori knowledge for this specific kind of music.

The use of gesture as a natural interface plays an utmost important role for achieving intelligent Human Computer Interaction (HCI). Human gestures include different components of visual actions such as motion of hands, facial expression, and torso, to convey meaning. So far, in the field of gesture recognition, most previous works have focused on the manual component of gestures. In this paper, we present an appearance-based multimodal gesture recognition framework, which combines the different groups of features such as facial expression features and hand motion features which are extracted from image frames captured by a single web camera. We refer 12 classes of human gestures with facial expression including neutral, negative and positive meanings from American Sign Languages (ASL). We combine the features in two levels by employing two fusion strategies. At the feature level, an early feature combination can be performed by concatenating and weighting different feature groups, and LDA is used to choose the most discriminative elements by projecting the feature on a discriminative expression space. The second strategy is applied on decision level. Weighted decisions from single modalities are fused in a later stage. A condensation-based algorithm is adopted for classification. We collected a data set with three to seven recording sessions and conducted experiments with the combination techniques. Experimental results showed that facial analysis improve hand gesture recognition, decision level fusion performs better than feature level fusion.

We propose a Tensor Decomposition based algorithm that recognizes the observed action performed by an unknown person and unknown viewpoint not included in the database. Our previous research aimed motion recognition from one single viewpoint. In this paper, we extend our approach for human motion recognition from an arbitrary viewpoint. To achieve this issue, we set tensor database which are multi-dimensional vectors with dimensions corresponding to human models, viewpoint angles, and action classes. The value of a tensor for a given combination of human silhouette model, viewpoint angle, and action class is the series of mesh feature vectors calculated each frame sequence. To recognize human motion, the actions of one of the persons in the tensor are replaced by the synthesized actions. Then, the core tensor for the replaced tensor is computed. This process is repeated for each combination of action, person, and viewpoint. For each iteration, the difference between the replaced and original core tensors is computed. The assumption that gives the minimal difference is the action recognition result. The recognition results show the validity of our proposed method, the method is experimentally compared with Nearest Neighbor rule. Our proposed method is very stable as each action was recognized with over 75% accuracy.

This paper proposes a Temporal Modified-RANSAC based method that can discriminate each moving object from the still background in the stereo video sequences acquired by moving stereo cameras, can compute the stereo cameras' egomotion, and can reconstruct the 3D structure of each moving object and the background. We compute 3D optical flows from the depth map and results of tracking feature points. We define "3D flow region" as a set of connected pixels whose 3D optical flows have a common rotation matrix and translation vector. Our Temporal Modified-RANSAC segments the detected 3D optical flows into 3D flow regions and computes the rotation matrix and translation vector for each 3D flow region. As opposed to the conventional Modified-RANSAC for only two frames, The Temporal Modified-RANSAC can handle temporal images with arbitrary length by performing the Modified-RANSAC to the set of a 3D flow region that classified in the latest frame and new 3D optical flows detected in the current frame iteratively. Finally, the 3D points computed from the depth map in all the frames are registered using each 3D flow region's matrix to the initial positions in the initial frame so that the 3D structures of the moving objects and still background are reconstructed. Experiments using multiple moving objects and real stereo sequences demonstrate promising results of our proposed method.

This paper proposes a GPU based method that can visualize voxelized surface data with fine and complicated features has high rendering quality at interactive frame rates and provides low memory consumption The surface data is compressed using run length encoding (RLE) for each level of detail (LOD) Then the loop for the rendering process is performed on the GPU for the position of the viewpoint at each time instant The scene is raycasted in planes where each plane is perpendicular to the horizontal plane in the world coordinate system and passes through the viewpoint For each plane one ray is cast to rasterize all RLE elements intersecting this plane starting from the viewpoint and ranging up to the maximum view distance This rasteritization process projects each RLE element passing the occlusion test onto the screen at a LOD that decreases with the distance of the RLE element from the viewpoint Finally the smoothing of voxels in screen space and full screen anti aliasing is performed To provide lighting calculations without storing the normal vector inside the RLE data structure our algorithm recovers the normal vectors from the rendered scene s depth butter After the viewpoint changes the same process is re executed for the new viewpoint Experiments using different scenes have shown that the proposed algorithm is faster than the equivalent CPU implementation and other related methods Our experiments further prove that this method is memory efficient and achieves high quality results.

This paper proposes a method for detecting ellipses from an image despite (1) multiple colors within the ellipses, (2) partially occluded ellipses' boundaries, (3) noisy. locally deformed boundaries of ellipses, (4) presence of multiple objects other than the ellipses in the image, and (5) combinations of (1) through (4). After boundary curves are obtained by edge detection, by utilizing the first-order difference curves of the edge orientation of each pixel in the boundary curves, a segment-reconnect method obtains boundary clusters. Then, a modified RANSAC detects ellipses by choosing five pixels randomly from the boundary clusters, where overlapped ellipses are merged. Experimental results using synthesized images and real images demonstrate the effectiveness of the proposed method together with comparison with the Randomized Hough Transform, a well-known conventional method.

Recognizing hand gestures from the video sequence acquired by a dynamic camera could be a useful interface between humans and mobile robots. We develop a state based approach to extract and recognize hand gestures from moving camera images. We improved Human-Following Local Coordinate (HFLC) System, a very simple and stable method for extracting hand motion trajectories, which is obtained from the located human face, body part and hand blob changing factor. Condensation algorithm and PCA-based algorithm was performed to recognize extracted hand trajectories. In last research, this Condensation Algorithm based method only applied for one person's hand gestures. In this paper, we propose a principal component analysis (PCA) based approach to improve the recognition accuracy. For further improvement, temporal changes in the observed hand area changing factor are utilized as new image features to be stored in the database after being analyzed by PCA. Every hand gesture trajectory in the database is classified into either one hand gesture categories, two hand gesture categories, or temporal changes in hand blob changes. We demonstrate the effectiveness of the proposed method by conducting experiments on 45 kinds of sign language based Japanese and American Sign Language gestures obtained from 5 people. Our experimental recognition results show better performance is obtained by PCA based approach than the Condensation algorithm based method.

In this paper, the end-of-life products (ELPs) collecting and recycling system is studied under the application of extended producer responsibility (EPR) programs. Economic performance is evaluated based on the decomposition and analysis of major cost and revenues components during the recycling process. We identified two problematic scenarios where EPR programs cannot run smoothly. The according prosperous incentive management strategies are discussed as the corresponding solution.

This paper proposes a method that identifies and tracks a walking human across discontinuous fields of views of multiple cameras for the purpose of video surveillance. A typical video surveillance system has multiple cameras, but there are several spaces within the surveillance area that are not within any of the camera's field of view. Also, there are discontinuities between the fields of views of adjacent cameras. In such a system, humans need to be tracked across discontinuous fields of views of multiple cameras. Our proposed model addresses this issue using the concepts of gait pattern, gait model, and motion signature. Each human's gait pattern is constructed and stored in a database. This gait pattern spans a tensor space that consists of three dimensions: person, image feature, and spatio-temporal data. A human's gait model can be constructed from the gait pattern using the "tensor decomposition based approach" described in this paper. When human(s) appears in one of the camera's field of a view (which is often discontinuous from the other camera's field of views), the human's motion signature is calculated and compared to each person in the database's gait model. The person with the gait model that is most similar to the motion signature is identified as same person. After the person is identified, the person is tracked within the field of view of the camera using the mean-shift algorithm based on color parameters. We conducted two experiments; the first experiment was identifying and tracking humans in a single video sequence, and experiments, the percentage of subjects that were correctly identified and tracked was better than that of two currently widely-used methods, PCA and nearest-neighbor. In the second experiment was the same as the first experiment but consisted of multiple-cameras with discontinuous views. The second experiment (human tracking across discontinuous images), shows the potential validity of the proposed method in a typical surveillance system.

This paper proposes a modified RANSAC based method that can detect multiple independently moving objects from the video sequences acquired by active stereo cameras, and can estimate the egomotion of the stereo cameras. We first compute 3D optical flows from consecutive frames, where dense 3D optical flows, which are needed for the subsequent egomotion estimation based on the modified RANSAC, are obtained by the process proposed in this paper. From the dense flows, three 3D optical flows are randomly selected for computing an egomotion (model). The validity of that egomotion model is checked using the weight obtained by calculating a consistency between that model and each of the dense flows. If it turns out that the estimated egomotion model is valid, the label that identifies this model is assigned to the inliers, which are consistent with this model, and then the labeled inliers are deleted. The random choice of three flows and its subsequent processes are repeated till one of the termination con itions is met. Using the inliers present in the static background, the real egomotion is computed. Experiments using synthesized and real stereo sequences demonstrate the effectiveness of the proposed method. © 2010, The Institute of Image Electronics Engineers of Japan. All rights reserved.

In this paper, we propose an improved Kernel-based Fuzzy C-means Algorithm (iKFCM) with spatial information to reduce the effect of noise for brain MR image segmentation. We use k-nearest neighbour model and a neighbourhood controlling factor by estimating image contextual constraints to optimize the objective function of conventional KFCM method. Conventional KFCM algorithms classify each pixel in image only by its own gray value, but the proposed method classifies by the gray values of its neighbourhood system. For this reason, the proposed iKFCM has a strong robustness for image noise in image segmentation. In experiments, some synthetic grayscale images and simulated brain MR images are used to assess the performance of iKFCM in comparison with other fuzzy clustering methods. The experimental results show that the proposed iKFCM method achieves a better segmentation performance than other fuzzy clustering methods. © 2010 IEEE.

This paper explores the feasibility of using machined vision based cash register system in a university cafeteria and automatically determine price of the food items that are contained on a cafeteria food tray. The idea is to take an image of the food tray, identify all the food items on the tray, and display the total food cost. A fully automated version of the system is envisioned to use a standard biometric technology for customer identification and for automatic debit process of the food cost from the customer's pre-registered account. The analysis method uses shape and size analysis to identify food items that are served on distinct type dishes and color analysis for those items that are served in similar type dishes. The paper explains prototype of a developed system and experimental results for 10 types of meals served at Ritsumeikan Asia Pacific University Cafeteria to justify the validity of this approach. ©2010 IEEE.

In this paper, the administration-driven WEEE management system in Europe and the market-driven end-of-life automobile management system in the US are studied first in this paper. Successful experiences and deficiencies of each system are investigated and compared. Then the potentials for integrating these two systems under each single-instrument-dominant scenario are identified followed by the discussion on key factors for soundly building an integrated system under various scenarios in general. Based on these achievements, the WEEE management system in China is taken as an example to demonstrate how to practically set up an integrated system in such a specific conditions, so that the application of EPR principle could be better promoted in a more economic efficient and ecological effective manner. © 2010 IEEE.

We develop a framework based approach to extract and recognize hand gestures from the video sequence acquired by a dynamic camera, which could be a useful interface between humans and mobile robots. We use Human-Following Local Coordinate (HFLC) System, a very simple and stable method for extracting hand motion trajectories, which is obtained from the located human face and body part. Hand trajectory motion models (HTMM) are constructed by HFLC and hand blob changing factor. In this paper, we apply a principal component analysis (PCA) based approach to improve the recognition accuracy. For further improvement, temporal changes in the observed hand area changing factor are utilized as new image features to be stored in the database after being analyzed by PCA. Each HTMM in the database is classified into gesture categories, or temporal changes in hand blob changes. We demonstrate the effectiveness of the proposed method by conducting experiments on 51 kinds of sign language based Japanese and American Sign Language gestures obtained from 7 people. Our experimental recognition results show better performance is obtained by PCA based approach than the Condensation algorithm based method.

This paper proposes a system that automatically deforms the original video images into line drawing expression transmitted in both directions so as to achieve two goals: easing the concerns of users privacy and ensuring good enough support by visual channel. We examine the effect of line drawing video for support system on the effectiveness and impression of privacy on a collaboration between a novice user and an operator. As a result, it became clear that concern of the user on privacy or security are greatly reduced by using a line drawing image, and there was no significant difference on efficiency with an original image and a line drawing image. © 2010, The Institute of Image Electronics Engineers of Japan. All rights reserved.

This paper describes about the media which touch off informal text communication by black spot stimulus. A black spot stimulus is shown as a background image of the text field in advance of text input. The purpose of black spot stimulus presentation is to touch off conceiving topic based on the cognitive process model of creative thinking. The prototype system based on this media concept was developed, and the communication experiment was conducted. It turned out as a result of the experiment that these media touched off the idea about making topic strongly in the text communication. Moreover, it was implied that the topic touched off deepens the mutual understanding between speakers. © 2010, The Institute of Image Electronics Engineers of Japan. All rights reserved.

Recently, activities about the 3D digital archive of cultural heritages become very popular. The development of this technology aims at satisfying the competing ideals of preserving cultural heritage while at the same time making it available to the public. Rich expression by the stereoscopic 3D is suitable for the viewing of 3D data saved culture heritages. In this paper, we report on the work of archiving heritage of Chinese Maijishan Grotto and the developing of stereoscopic VR content for it. In our content, viewers can experience the precious cultural heritage which can't be usually watched. Representation of cultural heritage can be more lifelike by presenting stereoscopic images and interactions. In addition, the results of the subjective evaluation confirmed VR content's reality and 3D effects.

In this paper, we propose a novel ellipse detection method which is based on a modified RANSAC, with automatic sampling guidance from the edge orientation difference curve. Hough Transform family is one of the most popular and methods for shape detection, but the Standard Hough Transform loses its computation efficiency if the dimension of the parameter space gets high. Randomized Hough Transform, an improved version of Standard Hough Transform has difficulty in detecting shapes from complicated, cluttered scenes because of its random sampling process. As a pre-process for random selection of five pixels to be used to build the ellipse's equation, we propose a two-step algorithm: (1) region segmentation and contour detection by mean shift algorithm (2) contour splitting based on the edge orientation difference curve obtained from the contour of each region. In each contour segment obtained by step (2), 5 pixels are randomly selected and the modified RANSAC is applied to the 5 pixels so that an accurate ellipse model is obtained. Experimental result show that the proposed method can achieve high accuracies and low computation cost in detecting multiple ellipses from an image.

This paper proposes a Tensor Decomposition Based method that can recognize an unknown person's action from a video sequence, where the unknown person is not included in the database (tensor) used for the recognition. The tensor consists of persons, actions and time-series image features. For the observed unknown person's action, one of the actions stored in the tensor is assumed. Using the motion signature obtained from the assumption, the unknown person's actions are synthesized. The actions of one of the persons in the tensor are replaced by the synthesized actions. Then, the core tensor for the replaced tensor is computed. This process is repeated for the actions and persons. For each iteration, the difference between the replaced and original core tensors is computed. The assumption that gives the minimal difference is the action recognition result. For the time-series image features to be stored in the tensor and to be extracted from the observed video sequence, the human body silhouette's contour shape based feature is used. To show the validity of our proposed method, our proposed method is experimentally compared with Nearest Neighbor rule and Principal Component analysis based method. Experiments using 33 persons' seven kinds of action show that our proposed method achieves better recognition accuracies for the seven actions than the other methods.

To achieve environments in which humans and mobile robots co-exist, technologies for recognizing hand gestures from the video sequence acquired by a dynamic camera could be useful for human-to-robot interface systems. Most of conventional hand gesture technologies deal with only still camera images. This paper proposes a very simple and stable method for extracting hand motion trajectories based on the Human-Following Local Coordinate System (HFLC System), which is obtained from the located human face and both hands. Then, we apply Condensation Algorithm to the extracted hand trajectories so that the hand motion is recognized. We demonstrate the effectiveness of the proposed method by conducting experiments on 35 kinds of sign language based hand gestures. © 2009 SPIE-IS&amp
T.

Research into creative thinking-support tools and communication is commonly focused on how to develop and share ideas between participants or with others. In this paper, we proposes a creative thinking support method that utilizes randomly generated visual prompter (black circle) image patterns (VP-patterns) and free hand-drawing and writing functions. Concepts and ideas of the research have been explained together with the development of the systems (CSP1 and CSP2). Experiments have been Conducted in order to evaluate the potentials and effectiveness of the system. From the results, a tendency towards inspiring creative ideas by participants has been observed.

Research into creative thinking-support tools and communication is commonly focused on how to develop and share ideas between participants or with others. In this paper, in order to evaluate the potential variations developmental directions for the visual "·" Prompter Message System (PM-System), we explore a Prompter Message Simulation (PM-Simulation), that uses a printed paper version of visual "·" prompter image patterns (VP-patterns), and allows not only writing texts but also freehand drawing and writing. Experiments have been conducted and analyzed into methods of stimulating the thought process, inspiring ideas, creative and imaginative thinking and enrich self-expression. From the results, a tendency towards inspiring creative ideas by participants has been observed. © 2008 IADIS.

In this paper, we present a framework of body-part segmentation for a real human wearing multiple-colored suit, which has different colors for individual body parts. The segmentation framework would be very useful for estimating postures of a real human. Our work firstly runs a robust background modeling and detection method called codebook so that the human area is extracted from the background. Then, we apply the mean shift segmentation as well as edge detection to the extracted human area. Experimental results show that the proposed method achieves better performances than conventional segmentation methods.

This paper deals with a real-time method for reconstructing 3D posture parameters of the human from orthogonal images by using artificial neural network techniques with higher accuracies than conventional computer vision based methods. To sample the data used for training the neural network, a human wearing the sensing devices of a motion capture system is synchronously observed by orthogonal cameras so that at each frame the motion capture data as well as the positions of the located feature points of the human areas in the camera images are obtained at the same time. During the training phase, the positions of the located feature points and the motion capture data are respectively used for the input and output vectors of the neural network. During the reconstruction phase, the positions of the located feature points at each frame of the video sequence to be reconstructed are inputted into the trained neural network so that the inputted data at each frame are converted to the motion capture data. The proposed method is implemented on a personal computer and runs in real-time (30 frames/sec). Experimental results confirm both the feasibility and the effectiveness of the proposed method for estimating 3D human postures.

This paper proposes a new efficient, robust, BEM (Boundary Element Method) based method for simulating articulated 3D elastic objects' behaviors. The articulated elastic object consists of bones, elastic object primitives and joining surfaces. After transforming the bones by inverse kinematics, the elastic object primitives are deformed basically by the BEM. To reduce the computation cost and memory usage and to achieve robust deformations, this paper improves the BEM based module of the conventional method: more specifically, a new method for computing the displacements of the vertices in the joining surfaces. Experimental results show that the proposed model can be simulated 10 times as fast as the conventional model or faster, and that real time performances such as 60fps can be achieved even for models with 20,000 vertices. High quality simulations of different deformations including expansions and contractions are achieved using character type models.

hi this paper, an efficient approach to segmentation of different independent motion areas from a moving platform is described. This approach is implemented on a stereo vision system, depth information could be computed by matching feature points between stereo images. For two consecutive frames, ego-motion is estimated from the optical flows, including depth information belonging to the background, which has a larger space distribution comparing to those of independent moving objects. In order to distinguish different motion areas, we proposed a modified version of RANSAC mechanism, which could handle the problem of multiple model extraction in a noisy environment.

This paper proposes a new method for enhancing the contrast of color images based on Wavelet Transform and human visual system. The RGB (red, green, and blue) values of each pixel in a color image are converted to HSV (hue, saturation and value) values. To the V (luminance value) components of the color image, Wavelet Transform is applied so that the V components are decomposed into the approximate components and detail components. The obtained coefficients of the approximate components are converted by a grey-level contrast enhancement technique based on human visual system. Then, inverse Wavelet transform is performed for the converted coefficients so that the enhanced V values are obtained. The S components are enhanced by histogram equalization. The H components are not changed, because changes in the H components could degrade the color balance between the HSV components. The enhanced S and V together with H are converted back to RGB values. The effectiveness of the proposed method is demonstrated experimentally.

This paper proposes an efficient and effective computation method for simulating articulated 3D elastic objects. The articulated elastic object consists of bones, elastic object primitives and joining surfaces. After transforming the bones by inverse kinematics, the elastic object primitives are deformed basically by the BEM (Boundary Element Method). To reduce the computation cost and memory usage and to achieve robust deformations, this paper improves the BEM based module of the conventional method: more specifically, a new method for computing the displacements of the vertices in the joining surfaces. Experimental results show that the proposed model can be simulated 10 times as fast as the conventional model or faster, and that real-time performances such as 60fps can be achieved even for models with 20,000 vertices. High quality simulations of different deformations including expansions and contractions are achieved using character type models.

We present a novel skinned skeletal animation system based on spline-aligned deformations for providing high quality and fully designable deformations in real-time. Our ambition is to allow artists the easy creation of abstract, pose-dependent deformation behaviors that might directly be assigned to a large variety of target objects simultaneously To achieve this goal, we introduce the usage of deformation styles and demonstrate their applicability by our animation system. We therefore enhance spline-skinned skeletal animation with two sweep-based free-form-deformation (FFD) variants. The two FFD variants are pose-dependent, driven by three textures and three curves, which can be designed by the artist. As the three textures are similar to height-maps, their creation is very intuitive. Once designed, the deformation styles can be directly applied to any number of targets for imitating material behaviors of cloth, metal or even muscles. Our GPU based implementation shows promising results for real-time usage, as about 30 Million vertices per second can be animated. The basic spline-skinning even reaches more than twice the speed and gets close to the performance of skeletal subspace deformation (SSD). Furthermore, our method can easily be combined along with other existing deformation techniques as pose space deformation or SSD.

This paper proposes a system that automatically deforms the images transmitted in both directions so as to achieve two goals: easing the concerns of users and ensuring well enough support. We examine here the effect of visual information quality on the effectiveness and impression of collaboration between a novice user and an operator.

Research into thinking-support tools is commonly focused on how to develop and share ideas between participants or with others. In this paper, we propose and develop a communication system that stimulates the thought processes and inspires the creative ideas of participants by using a visual "circle" prompt within the framework of a communication pallet. Experiments have been conducted into methods of stimulating the thought process and inspiring ideas during conversation and the results have been analyzed. From the results, a tendency towards inspiring creative ideas by participants has been observed.

Research into thinking-support tools is commonly focused on how to develop and share ideas between participants or with others. In this paper, we propose and develop a communication system that stimulates the thought processes and inspires the creative ideas of participants by using a visual "circle" prompt within the framework of a communication pallet. Experiments have been conducted into methods of stimulating the thought process and inspiring ideas during conversation and the results have been analyzed. From the results, a tendency towards inspiring creative ideas by participants has been observed.

Presented is a novel skeletal animation system for providing high quality geometric deformations in real-time. Each bone of the skeleton is therefore represented by a spline, rather than using conventional matrix rotation. In our approach, each vertex of the animated character can be influenced by a maximum of three spline-curves, which is sufficient for skinned animation. One spline is parameterized by three control points and a possible twist. As opposed to conventional Bezier curves does our arc-spline rely on trigonometric functions for providing better curvatures. The optimized implementation using the OpenGL-shading language shows very promising results for real-time character animation, as even about 1 Million vertices were able to be transformed at interactive 43 frames per second on a GeForce 7800 GTX graphics card.

We have been researching the relationship between human motion and emotion. In this paper, our purpose is to extract motion features specific to each emotion. We propose a new approach for motion data analysis, which applies the higher order Singular Value Decomposition(HOSVD) direct to the motion data and the wavelet analysis to the synthesized data with SVD. The HOSVD models the mapping between persons and emotions. The model can synthesize a complete data acting with each emotion for a given new person. The wavelet analysis extracts each motion feature from the synthesized data for each emotion. Some experimental results using motion capture data for "gait" action and 6 emotions - "angry, joy, sad and so on" show that our method can synthesize novel gait motions for a person by using the extracted motion elements and can extract some features specific to each emotion.

In this paper we describe a mixed reality-supported interactive viewing enhancement museum display system: Visual-dimension Interact System (VIS). With a transparent interactive inter-face, the museum visitor is able to see, manipulate, and interact with the physical exhibit and its virtual information, which are overlapped on one other. Furthermore, this system provides the possibility for visitor to experience the creation process in an environment as close as possible to the real process. This has the function of assisting the viewer in understanding the exhibit and most importantly, gaining a so-to-speak hands-on experience of the creation process itself leading to a deeper understanding of it.

This paper studies how audiences should be expressed in a Cyber-theater, in which remotely located persons can direct plays as directors, perform as performers and/or see the performances as audiences through a networked virtual environment. It is noted that the audience effect has been widely acknowledged in the real-world theater: that is, the audience reaction has a significant effect on the acting of player and performance of the play itself. However, only a few works relevant to audiences in the Cyber Theater can be seen. This paper studies whether the audience effect exists also in the Cyber-theater. By constructing a system in which two actors are displayed a remotely located audience's avatar in which the audience can display his/her emotional actions, we clarified that interactions between the actors and audience are effective.

This paper applies an algorithm, based on Tensor Decomposition, to a new synthesis application: by using sampled motions of people of different ages under different emotional states, new motions for other people are synthesized Human motion is the composite consequence of multiple elements, including the action performed and a motion signature that captures the distinctive pattern of movement of a particular individual. By performing decomposition, based on N-mode SVD (singular value decomposition), the algorithm analyzes motion data spanning multiple subjects performing different actions to extract these motion elements. The analysis yields a generative motion model that can synthesize new motions in the distinctive styles of these individuals. The effectiveness of applying the tensor decomposition approach to our purpose was confirmed by synthesizing novel walking motions for a person by using the extracted signature.

A method for enabling efficient facial data transmission over the communication network and reconstructing realistic 3D face images is proposed. A conventional method which combines the advantages of the model-based and image-based approaches to reconstruct 3D facial images with high efficiency and precision on a standalone environment is extensively used. In our approach, the face image recognition part and the 3D image reconstruction part, which were conventionally carried out on the same PC, are separated and implemented on two network-connected PCs. Also, by using JPEG compression, the size of image data transmitted over the network is reduced. Our method was implemented on network-connected PCs, and its effectiveness was confirmed. Subjective tests showed that image data could be reduced to around 6% of its original size. © 2005, The Institute of Image Electronics Engineers of Japan. All rights reserved.

This paper proposes a method of tracking a human object by using nonsynchronous multiple-viewpoint images. The proposed method tracks human forms efficiently by using a Kalman filter to integrate observed information which is obtained nonsynchronously from multiple viewpoints. The experimental system is composed of multiple observation nodes, which operate nonsynchronously to process the multiviewpoint images, a tracking node, which tracks the human figure, and the discovering node, which finds the human figure. The image features are matched to the tracking model in the observation node based on the prediction of the observed value which is sent from the tracking node. The image features matched to the model are sent to the tracking node, and the tracking model is updated. The image features which are not matched are sent to the discovering node to find a new human figure. With the proposed approach, it is possible to construct a large-scale tracking system while reducing the deterioration of processing efficiency and the redundancy among observations that occur in synchronous systems. The effectiveness of the proposed method is demonstrated by an experiment using real images. (C) 2004 Wiley Periodicals, Inc.

In this paper we describe a mixed reality supported interactive viewing enhancement museum display system. With a transparent interactive interface, the museum visitor is able to see, manipulate, and interact with the physical exhibit and its virtual information, which are overlapped on one other. Furthermore, this system provides the possibility for the visitor to experience the creation process in an environment as close as possible to the real process. This has the function of assisting the viewer in understanding the exhibit and most importantly, gaining a so-to-speak hands-on experience of the creation process itself leading to a deeper understanding of it.

We have been studying about people's visual impression and image features for texture images in order to clarify the human subjective interpretation mechanism for images[1]. In corresponding image features of human impressions for the images, we found that the impressions for material were bottle-necked. We have studied a new analysis method which gives the impression for material from texture images. Especially, we mainly focused on the properties of visual targets which people can feel tactile sense. In this paper, we propose a new texture analysis method which is based on frequency analysis with 3D texture which is designed for photorealistic rendering. We found that our new method can estimate not only the surface roughness but also the surface softness.

The authors have developed a binocular-type display system that allows digital archives of cultural assets to be viewed in their actual environment. The system is designed for installation in locations where such cultural assets were originally present. The viewer sees buildings and other heritage items as they existed historically by looking through the binoculars. Images of the cultural assets are reproduced by stereoscopic 3D CG in cyberspace, and the images are superimposed on actual images in real-time. This system consists of stereoscopic CCD cameras that capture a stereo view of the landscape and LCDs for presentation to the viewer. Virtual cameras. used to render CG images from digital archives. move in synchrony with the actual cameras, so the relative position of the CG images and the landscape on which they are superimposed is always fixed. The system has manual controls for digital zoom. Furthermore, the transparency of the CG images can be altered by the viewer. As a case study for the effectiveness of this system, the authors chose the Heijyoukyou ruins in Nara, Japan. The authors evaluate the sense of immersion, stereoscopic effect, and usability of the system.

A Dynamic Programming (DP) based algorithm that achieves real-time, high quality stereo-matching is presented. A special coarse to fine approach as well as the MMX based assembler implementation mainly contribute to the reached computation speed. A novel vertical smoothing approach by re-integrating paths inside the DP matching allows the avoidance of noisy horizontal strokes, so that high quality stereo-matching is achieved. The current implementation capable of running at about 30 FPS on an 2.2GHz PC, which is sufficient for the utilization in real-time applications.

This is a position paper reporting an on-going collaboration project between SUNY Binghamton, USA, and Waseda University, Japan, on multimodal information retrieval through exploiting the cognitive synergy across the different modalities of the information to facilitate an effective retrieval. Specifically we focus on image retrieval in the applications where imagery data appear along with collateral text. It is noted that these applications are ubiquitous. We have proposed the Synergistic Indexing Scheme (SIS) to explicitly exploit the synergy between the information of imagery and text modalities. Since the synergy we have exploited between the information of imagery and text modalities is subjective and depends on specific cognitive context, we call this type of synergy as cognitive synergy. We have reported part of the empirical evaluation and are in the process to fully implement the SIS prototype for an extensive evaluation.

As a step towards subjective image retrieval, this paper reports an on-going collaboration project between Waseda University and SUNY Binghamton, on relating texture images to haptic impressions. To grasp the surface height variations, texture images are taken under different illuminations and viewing conditions. Our method applies a new frequency analysis method to the texture images. We evaluate the performances of our feature and other typical conventional features by checking whether texture images are correctly classified into "soft" or "hard" by the SVM (support vector machine) method, where the training data for the SVM are collected by subjective tests. Experimental results show that our texture feature can classify "soft" or "hard" better than the other features.

This paper deals with a method that analyzes a botanical tree's behaviors in real space by a computer vision approach so as to reproduce the analyzed behaviors in virtual space. Instead of applying unstable local tracking to the tree in a video sequence, we estimate the direction and strength of the wind that shakes the tree by a learning based method that classifies the input video sequence into one of the stored winds with different directions and strengths. In the learning phase, sample video sequences are used for constructing the Eigenspace and Fisherspace, which is obtained from Fisher discriminant analysis. In the classification phase, the input video sequence is compared with each of the stored sample sequences so that the direction and strength of the wind are estimated. An interpolation method improves the estimation accuracy. Experimental results demonstrate the effectiveness of the proposed method.

A method for constructing 3D human facial images which maintains the chararcteristics of video input image is proposed. First, a personalized 3D polygonal facial model is constructed from two orthogonal facial images. Video input frames are modified, and the vertices composing the 3D polygons are altered according to the contraction rule of facial muscles. Moreover, the modified input frame is texture mapped directly onto to the forementioned model to improve the quality of the reconstructed image. This method was implemented onto an ordinary PC, and an experiment using five persons' face data showed the effectiveness of the method both quantitatively and subjectively.

This paper suggested an image processing method using a multiple-colored suit for estimating complicated postures of a human body. One of the major problems on the posture estimation when using the silhouette image analysis is the overlap of the body parts' silhouettes. In order to solve this problem, the human wears the multiple-colored suit so that each body part has a different color from the other body parts. Experimental results show the effectiveness of the proposed method. It has confirmed that the complicated overlapped postures can be analyzed through the contour of the human's silhouette and the color information from the body segments that are taken using three directional cameras which are located at the front, left, and right side of the subject.

This paper proposes a computer vision based method that recognizes interactions between human body and object. In two successive frames in a video sequence, our function based contour analysis method checks whether the silhouettes of the human body and object overlap. If they do not overlap, our method judges that the human body and object does not interact. If the two silhouettes overlap, our method checks whether the motion vectors obtained from the areas for the human body and object coincide. If they coincide, our method judges that the human body and object are interacting. Some experimental results show the effectiveness of the proposed method. © 2003 by Springer Science+Business Media New York.

This paper proposes a non-contact type method for estimating human body postures. One of the major problems on the posture estimation using the silhouette image analysis is the overlapping of the body parts' silhouettes. In order to solve this problem, this paper proposes a method for estimating the posture of a human wearing a multiple-colored suit based on color information processing. By analyzing the contour of the human's silhouette, the method judges whether feature points are occluded by another body parts. If the occlusions occurs, color region segmentation is performed in order to know which region is frontal. The feature point in the frontal region is located in the skeleton of the region. Experimental results show the effectiveness of the proposed method.

This paper proposes anew efficient method that can animate botanical trees in 3D realistically. In this paper, a 3D botanical tree model consists of a set of branch segments, to which leaf models are attached. To reduce the amount of computation, instead of calculating the motions of all the branch segments, only the representative segment in each branch is numerically analyzed. The numerical analysis is constrained to a 2D plane so that 3D numerical analysis need not be performed. Concerning the leaf model, a set of four leaves is systematically attached to each branch segment. Experimental results clarify the conditions for real-time, realistic animations of dynamic behaviors of trees.

A method for constructing 3D human facial images which maintains the chararcteristics of video input image is proposed. First, a personalized 3D polygonal facial model is constructed from two orthogonal facial images. Video input frames are modified, and the movements of facial feature points are tracked and measured based on the the contraction rule of facial muscles. The vertices of the 3D polygons are altered accordingly, and the modified input frame is texture mapped directly onto to the forementioned model. This method was implemented using an ordinary CCD camera and computer, and an experiment using one person's data showed that realistic facial images could be reconstructed with high speed and precision.

In order to correspond image features to human impressions for images, the impression for material is a bottle-neck. We have studied the new analysis method which gives the impression for material from a texture image. In this report, we propose a new material analysis method based on frequency analysis. Our method can classify texture images into the material which has similar surface structure.

An efficient method for constructing facial images for use in telecommunication applications is proposed. This method uses a simple 3D feature model, which consists of polygons, which describe the shape of the face, and elastic linear springs, which simulate the natural movements of facial muscles. This method requires only two orthogonal facial images, and could easily be implemented on a relatively low-spec PC. Experimental results showed good results that various facial expressions could be synthesized and displayed from arbitrary directions.

This paper describes the author's recent activities related to computer vision based methods for analyzing human behaviors: more specifically, posture estimation and recognizing interactions between a human body and object. For estimating postures in 3D from multiple camera images, the authors developed a heuristic based method and non-heuristic method. The heuristic based method heuristically analyzes the contour of a human silhouette so that significant points of a human body can be located in each image. The non-heuristic method utilizes a function for analyzing contours without using heuristic rules. Recognizing the interactions exploits the function based contour analysis and motion vector based analysis so that the system can judge whether the human body interacts with the object.

In this paper, we present a novel representation of the human face for estimating the orientation of the human head in a two dimensional intensity image. The method combines the use of the much familiar eigenvalue based dissimilarity measure with image based rendering. There are two main components of the algorith described here: The offline hierarchical image database generation and organization, and the online pose estimation stage. The synthetic images of the subject's face are automatically generated offline, for a large set of pose parameter values, using an affine coordinate based image reprojection technique. The resulting database is formally called as the IBR (or image based rendered) database. This is followed by the hierarchical organization of the database, which is driven by the eigenvalue based dissimilarity measure between any two synthetic image pair. This hierarchically organized database is a detailed, yet structured, representation of the subject's face. During the pose estimation of a subject in an image, the eigenvalue based measure is invoked again to search the synthetic (IBR) image closest to the real image. This approach provides a relatively easy first step to narrow down the search space for complex feature detection and tracking algorithms in potential applications like virtual reality and video-teleconferencing applications. © 2001 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.

Traditionally most camera-based position estimation systems use only a few points to calibrate cameras. In this paper, we investigate a novel and alternate approach for 3D position estimation by using a larger number of points arranged in a 3D grid. We present an implementation of the active-space indexing mechanism which uses three cameras. Given the corresponding points in camera images, a precise estimation of the position can be obtained. The active-space indexing method can be also used as a spatial filter to eliminate the large number of possible corresponding pairs from consideration. This capability, unique only to the active-space indexing method, provides a tractable algorithm to the otherwise intractable situation. (C) 2001 Academic Press.

This paper overviews our research activities on computer vision based non-verbal information analysis that can be applied to virtual communication environments and human computer interactions. In virtual communication environments, a user's facial expressions and body motions are estimated by computer vision approaches, and the estimated non-verbal information is reproduced in the user's avatar. For human computer interfaces, hand gestures are recognized as pre-defined commands by analyzing multiple camera images that observe the hand. In addition, facial expressions and body gestures are recognized from a time-sequential images by HMM (Hidden Markov Models).

This paper proposes a new real-time method of estimating human postures in three dimensions from trinocular images. The proposed method extracts feature points of the human body by analyzing contours of human silhouettes. The feature points are extracted by using the subtraction images when self-occlusions occur in the silhouette images. Dynamic compensation is carried out with a Kalman filter so that all feature points are tracked. The 3D coordinates of the feature points are reconstructed by considering the geometrical relationship between the three cameras. Experimental results confirm both the feasibility and the effectiveness of the proposed method.

This paper has two parts. In the first part of the paper, we note the property that under the para perspective camera projection model of a camera, the set of 2D images produced by a 3D point can be optimally represented by two lines in the affine space (alpha - beta space). The slope of these two lines are same, and we observe that this constraint is exactly the same as the epipolar line constraint. Using this constraint, the equation of the epipolar line can be derived. In the second part of the paper, we use the "same slope" property of the lines in the alpha - beta space to derive the affine structure of the human face. The input to the algorithm is not limited to an image sequence of a human head under rigid motion. It can be snapshots of the human face taken by the same or different cameras, over different periods of time. Since the depth variation of the human face is not very large, we use the para perspective camera projection model. Using this property, we reformulate the (human) face structure reconstruction problem in terms of the much familiar multiple baseline stereo matching problem. Apart from the face modeling aspect. we also show how we use the results for reprojecting human faces in identification tasks.

This paper has two parts. In the first part of the paper, we note the property that under the para perspective camera projection model of a camera, the set of 2D images produced by a 3D point can be optimally represented by two lines in the affine space (α β space). The slope of these two lines are same, and we observe that this constraint is exactly the same as the epipolar line constraint. Using this constraint, the equation of the epipolar line can be derived. In the second part of the paper, we use the "same slope" property of the lines in the α β space to derive the affine structure of the human face. The input to the algorithm is not limited to an image sequence of a human head under rigid motion. It can be snapshots of the human face taken by the same or different cameras, over different periods of time. Since the depth variation of the human face is not very large, we use the para perspective camera projection model. Using this property, we reformulate the (human) face structure reconstruction problem in terms of the much familiar multiple baseline stereo matching problem. Apart from the face modeling aspect, we also show how we use the results for reprojecting human faces in identification tasks.

This paper proposes a wavelet-based asker identification system for smart multi-point tele-conferences. Our system is composed of three processors. The first processor is called an "ask" action detector, which is used to decide whether an "ask" action is occurring. If an "ask" action is detected, the input image is sent to the face detector that detects the human face from the complex background. The detected face is fed to a wavelet-based face identification processor to recognize the asker.

Head and face detection and eye tracking in realtime are the first steps for head gesture recognition and/or face expression recognition for a human-computer interaction interface. We propose a two-step approach for eye tracking in video streams.
First, we detect or track a point between the eyes. For this task, we apply a special filter rye proposed before. Once we detect the point between the eyes, it is fairly easy to locate the eyes, which are the two small darkest parts on each side of this point. Because defecting the point between the eves is easier and more stable than directly detecting the eyes, the system can robustly track the eyes
We implement the system on a SGI O2,workstation. The video image size is 320x240 pixels. The system processes images at seven frames per second in the detection mode, and 13 frames per second in the tracking mode without arty special hardware.

This paper proposes a skin color information based face location algorithm that can locate the human face in a complex background including non-face skin color regions, such as hands. Our algorithm is divided into three steps. The first step is to use color components of the input image to segment the skin colors from the complex background. The second step is to detect a curve, which is called the division curve, between the hair region and face region. The last step is to use the division curve to estimate the face location. Our algorithm can be applied to some gesture analysis and understanding systems that combine the gestures of the head and hands.

For face detection and tracking in a video sequence, many systems are now applying a technique to extract skin-color areas effaces. In these systems, the skin-color model should be built beforehand. However skin-color changes when the lighting condition varies when an input video camera is changed, when the white-balance of the camera is readjusted, etc.
In this paper we describe a method to build a skin-color model for operation while a face detection system is running. When it has no skin-color model, it uses the differences between adjacent frames to extract moving objects. Once the system recognizes that the moving object is a face, it analyses the color histograms of the moving urea and extract skin-color distribution
We applied this method to our face detection and tracking system. Experiments showed good results.

In this paper, we propose and adaptive human tracking system with non-synchronous multiple observations. Our system consists of three types of processes, discovering node for detecting newly appeared person, tracking node for tracking each target person, and observation node for processing one viewpoint (camera) images. We have multiple observation nodes and each node works fully independently. The tracking node integrates observed information based on reliability evaluation. Both observation conditions (number of cameras), relative distance between a human and cameras, extent of occlusion, etc.) and human motion states (walking, standing, sitting) are considered in the evaluation. Matching between tracking models and observed image features are performed in each observation node based on the position, size and color similarities of each 2D image. Due to the non-synchronous property, this system is highly scalable for increasing the detection area and number of observing nodes. Experimental results for some indoor scenes are also described.

This paper proposes a new real-time method for estimating human postures in 3D from trinocular images. In this method, an upper body orientation detection and a heuristic contour analysis are performed on the human silhouettes extracted from the trinocular images so that representative points such as the top of the head can be located. The major joint positions are estimated based on a genetic algorithm-based learning procedure. 3D coordinates of the representative points and joints are then obtained from the two views by evaluating the appropriateness of the three views. The proposed method implemented on a personal computer runs in real-time. Experimental results show high estimation accuracies and the effectiveness of the view selection process. © 2000 IEEE.

Among head gestures, nodding and head-shaking are very common and used often. Thus the detection of such gestures is basic to a visual understanding of human responses. However it is difficult to detect them in real-time, because nodding and head-shaking are fairly small and fast head movements. We propose an approach for detecting nodding and head-shaking in real time from a single color video stream by directly detecting and tracking a point between the eyes, or what we call the "between-eyes". Along a circle of a certain radius centered at the "between-eyes", the pixel value has two cycles of bright parts (forehead and nose bridge) and dark parts (eyes and brows). The output of the proposed circle-frequency filter has a local maximum at these characteristic points. To distinguish the true "between-eyes" from similar characteristic points in other face parts, we do a confirmation with eye detection. Once the "between-eyes" is detected, a small area around it is copied as a template and the system enters the tracking mode.combining with the circle-frequency filtering and the template, the tracking is done not by searching around but by selecting candidates using the template
the template is then updated. Due to this special tracking algorithm, the system can track the "between-eyes" stably and accurately. It runs at 13 frames/s rate without special hardware. By analyzing the movement of the point, we can detect nodding and head-shaking. Some experimental results are shown. © 2000 IEEE.

This paper proposes a real-time human body posture estimation method using ANNs. The network is composed of three ANNs and a decision logic unit. The ANNs' input is the result of a function analysis on a human silhouette's contour extracted from camera images and the ANNs' output indicates the feature points' positions on the contour. The decision logic unit synthesizes each of the ANNs' output vectors and then the 2D coordinates of the human body's feature points are calculated. The proposed method is implemented on a personal computer and runs in real-time (17-20 frames/sec). Experimental results confirm both the feasibility and the effectiveness of the proposed method for estimating human body postures.

This paper proposes a new real-time method of estimating human postures in 3D from trinocular images. The proposed method extracts feature points Of the human body by applying a type of function analysis to contours of human silhouettes. To overcome self-occlusion problems, dynamic compensation is carried out using the Kalman filter and all feature points are tracked. The 3D coordinates of the feature points are reconstructed by considering the geometrical relationship between the three cameras. Experimental results confirm both the feasibility and the effectiveness of the proposed method, and an application example of the 3D human body posture estimation to a motion recognition system is presented.

This paper proposes a new communication method, called "Virtual Me", that enables a user to simultaneously participate in multiple Events such as meetings or lectures as a virtual-metamorphosed avatar controlled by the user and/or as a computer-controlled autonomous agents. By interpreting non-verbal data generated by the other participants at each event, the system determines when to switch between using virtual metamorphosis or employing agents. We demonstrate the effectiveness of this method in a prototype of the Virtual Me system. It is our hope that this technology may be in the interest not only of time efficiency but also of enriched human communication.

Indexing techniques usually are used in model-based object recognition and ray tracing algorithms. In this paper we present a new method for estimating the three-dimensional position of a subject (resp. object) in a circumscribed space based on an indexing method. We construct two and three-dimensional indices of a space, which are used to estimate the three-dimensional position by an interpolation technique. There are two processes in estimating the three-dimensional position of a subject (resp. object): preprocessing and three-dimensional position estimation. We have implemented this idea using stereo camera, and tested by using two different sizes of a grid pattern. Promising results for preprocessing and 3D position estimation are presented. Moreover, we show that this approach can also be extended for multiple cameras.

We propose a method of tracking 3D position, posture, and shapes of human hands from multiple-viewpoint images. Self-occlusion and hand-hand occlusion are serious problems in the vision-based hand tracking. Our system employs multiple-viewpoint and viewpoint selection mechanism to reduce these problems. Each hand position is tracked with a Kalman filter and the motion vectors are updated with image features in selected images that do not include hand-hand occlusion. 3D hand postures are estimated with a small number of reliable image features. These features are extracted based on distance transformation, and they are robust against changes in hand shape and self-occlusion. Finally, a "best view" image is selected for each hand for shape recognition. The shape recognition process is based on a Fourier descriptor. Our system can be used as a user interface device in a virtual environment, replacing glove-type devices and overcoming most of the disadvantages of contact-type devices.

In this paper, a new method was proposed for implementing 3-D facial image models designed to allow faithful reconstruction of facial images in virtual space teleconferencing by using 3-D measurement while detecting various facial expressions. In the method proposed, first many dots are painted on the face. Then, for a variety of facial expressions (eight expressions in this study) selected with regard to the action of the major facial muscles, 3-D displacement vectors for the normal (neutral) face are measured at each dot and recorded in the facial image plane as reference vectors. When the facial image is reconstructed, 2-D displacement vectors are detected by tracking markers that were previously placed on the face and are represented as the sum of two enclosing reference vectors. Based on these data, the vertices of a 3-D wireframe face model (WFM) are moved appropriately, and the facial expression is reconstructed. (C) 1999 Scripta Technica.

The virtual metamorphosis system lets people change their forms into any other form in a virtual scene. To realize these changes, a computer Vision system estimates facial expressions and body postures and reproduces them in a computer graphics avatar in real time. We introduce three systems in order of their development: the Virtual Kabuki system, Networked Theater, and "Shall We Dance?".

In this paper, we propose an algorithm to extract and track complex contours with occlusions such as the contours of the human body. In this algorithm, complex contours are assumed to be composed of contour segments which have smooth variation of the intensity gradient normal to the contour direction. Moreover, the contour segments are modeled using B splines, and contours in the image are extracted and tracked by the contour segments. In the first frame, contour segments are extracted using only the edge pixels. In the tracking of contour segments of the next frame, we use Snakes. For frames after the second frame, the contour segments are redetected using the tracking result and edge pixels. After that, extraction and tracking are repeatedly applied to each frame. Moreover, a method of classification of contour segments which is based on movement is described and the possibility of human action description is investigated. Finally, experimental and classification results obtained by means of this algorithm on synthesized and real images are given. © 1999 Scripta Technica.

A method for extracting facial motion parameters is proposed. The method consists of three steps. First, the feature points of the face, selected automatically in the first frame, are tracked in successive frames. Then, the feature points are connected with Delaunay triangulation so that the motion of each point relative to the surrounding points can be computed. Finally, muscle motions are estimated based on motions of the feature points placed near each muscle. The experiments showed that the proposed method can extract facial motion parameters accurately. In addition, the facial motion parameters are used to render a facial animation sequence.

This paper presents a hybrid estimation method of human body postures from CCD camera images. In the hybrid estimation method, the feature points of the human body (top of the head, tips of the hands, and feet, and elbow joints) are obtained from the results of heuristic contour analyses of human silhouettes or those of a time subtraction image depending on the reliability of the silhouette information. A dynamic compensation is then carried out by tracking all feature points using the AR model in order to obtain their optimal position and to overcome self-occlusion problems. The AR model's parameters are estimated through on-line processing by the Kalman filter. The proposed method is implemented on a personal computer and the process runs in real-time. Experimental results show high estimation accuracy and the feasibility of the proposed method.

This paper proposes a new real-time method for estimating human postures in 3D from trinocular images. In this method, an upper body orientation detection and a heuristic contour analysis are performed on the human silhouettes extracted from the trinocular images so that representative points such as the top of the head can be located. The major joint positions are estimated based on a genetic algorithm based learning procedure. 3D coordinates of the representative points and joints are then obtained from the two views by evaluating the appropriateness of the three views. The proposed method implemented on a personal computer runs in real-time (30 frames/second). Experimental results show high estimation accuracies and the effectiveness of the view selection process.

In this paper, we describe a visual surveillance system for evaluating the audience's reaction in meeting sessions. The system, which works in real-time, can recognize and evaluate the reaction of the audience. It is mainly composed of three subsystems. The first subsystem is a face detection and head motion segmentation system which is used to detect the face from complex background and segment the head motion into different units, with each unit including different information about the audience's reaction. The second subsystem is a gesture and pose recognition system which can recognize the gestures and the poses of human head. The third subsystem is an evaluation system which is used to evaluate the reaction of the audience by using the recognition results in the second system. Our system has been tested for determining the concentration of the audience in meeting sessions and the experimental results are good.

This paper proposes a new real-time method for reproducing facial expressions in 3D face models realistically based on anatomy for artists. To reproduce facial expressions in a face model, the detected expressions need to be converted to the data for deforming the face model. In the proposed method, an artist who has learned anatomy for artists creates arbitrary facial expressions in the 3D face model by mixing the reference expressions chosen by the artist so that the synthesized expressions realistically represent the respective expressions displayed by real persons. The parameters obtained by this manual operations are used to construct the equations that convert the expression features obtained by the detection module to the displacement vectors of the vertices of the face model. During human communications through face models, the equations are used to reproduce the detected expressions in real-time. The effectiveness and robustness of the proposed method were demonstrated by experimental results and demonstration systems.

In this paper we present a unified theoretical framework for novel scene synthesis, merging real and virtual worlds, and view stitching. To start with, we have a set of real images from weakly calibrated cameras, for which we compute the dense point match correspondences. For applications like novel view synthesis, one may first solve the 3D scene reconstruction problem, followed by a view rendering process. However, errors in 3D scene reconstruction usually gets reflected in the quality of the new scene generated, so we seek. a more direct method. In this paper, we use the knowledge of dense point matches and their affine coordinate values to estimate the corresponding affine coordinate values in the new scene. Our technique of reprojection is extended for other applications like merging real and synthetic worlds, and view stitching. (C) 1998 Elsevier Science B.V. All rights reserved.

This paper describes the vutual metamorphosis system, whtch was originally proposed by ATR Media Integratton &amp
Communicattons Research Laboratories, Kyoto, Japan. In the vntual metamorphosis system, people can change their forms mto any other form in a vutual scene. To realize. this changes, facial expressions and body postures are estimated by computer vision and reproduced m a computer graphics avatar m real-ume. We developed computer vision based technologies for automatic face trackmg, estimatmg deformations of factal components,and estimatmg body postures in real-time. We also developed real-time facial expresston reproduction method based on "Anatomy for artists". Usmg these technologies, prototyping systems for virtual metamorpho-SIS were constructed. This paper introduces two systems in order of their development: the Virtual Kabuki system and the "Shall We Dance" system.

A new method for extracting human silhouettes from color images in real-time regardless of changes in lighting conditions and backgrounds is proposed. In the method, RGB values of each pixel in color images acquired by a CCD camera are converted to YIQ values. Before extracting human silhouettes, the mean YIQ values of each pixel and the threshold values for the Y, I, Q components are obtained in advance from the image sequence acquired by the camera that observed the background. Based on the mean values and the threshold values, each pixel is judged whether it belongs to the human silhouette or not. The proposed method was used with the virtual metamorphosis system, in which anyone can change his or her form to any other form, and the robustness and real-time-ness were confirmed. © 1998 IEEE.

A method that can be used for spotting segments that display facial expression is proposed. The motion of the face is modeled by HMM in such a way that each state corresponds to the conditions of facial muscles, e.g., relaxed, contracting, apex and relaxing. The probability assigned to each state is updated iteratively as the feature vector is obtained from image processing. A spotted segment is placed into a certain category when the probability of that category exceeds a threshold value. Experiments show that the segments for the six basic expressions can be spotted accurately in near real time. © 1998 IEEE.

In this paper we present a new camera projection model, which is intermediate between the affine camera mode and the pin hole projection model. It is modeled as a perspective projection of 3D points into an arbitrary plane, followed by an affine transform of these projected points. We observe that the reprojection of a point into a navel image can be achieved uniquely provided that we have located a set of Jive reference points over four images (of which three are input images, and the fourth is the novel image). Also, the reprojection, theory; does not assume that the input images are captured from cameras with identical internal calibration parameters. Thus, we apply our technique two different domain: (a) generation of novel images from a stereo pair (b) generation of virtual walkthrough sequence with a monocular image sequence as input.

We have implemented a new algorithm, called the geometric-imprints method for extracting key-features of a given contour. Series of points on the curve are used to find a geometric-imprint set of the contour. The algorithm is based upon the assumption that an articulated human and other figures have mostly cylindrical body-parts. The algorithm recursively partitions the contour based upon the cylindrical nature of human-silhouettes, and a starting point. Similar topological contours are expected to produce similar geometric-imprints. Thus the correspondence problem is somewhat simplified. Results of our implementation are presented.

In this paper we present an algorithm to estimate the human face structure. The input to the algorithm is not limited to an image sequence of a human head under rigid motion. It can be snapshots of the human face taken by the same or different cameras, over different periods of time. Since the depth variation of the human face is not very large, we use the affine camera projection model. Under this assumption, it can be shown that the set of 2D images produced by a 3D point feature of a rigid object can be optimally represented by two lines in the affine space. Using this property, we reformulate the (human) face structure reconstruction problem in terms of the much familiar multiple baseline stereo matching problem [8]. Apart from the face modeling aspect, we also show how we use the results for reprojecting human faces in identification tasks.

At ATR Media Integration & Communications Research Laboratories, our research group is aiming at realizing virtual communication environments in which the users at remote sites can feel that they are virtually co-located. This paper presents the ongoing research in two subareas chosen to address this problem: (a) Generating human images in virtual scenes (b) Act image based approach for generation of (background) scenes. In the virtual metamorphosis system, 3D models of characters are created in advance. For metamorphosis, the facial expressions and body posture of a person are detected in real-time fr om the face images and monocular thermal images of the person, respectively. The detected expressions and body movements are reproduced by deforming the 3D models. Demonstrations using SD models of a Kabuki actor and a dinosaur show good performance. for novel view generation, we use as affine coordinate based re-projection scheme, which is also extended to merging real and synthetic objects.

This paper introduces a new real-lime method to estimate the posture of a human from thermal images acquired by an infrared camera regardless of the background and lighting conditions. Distance transformation is performed for the human body area extracted from the thresholded thermal image for the calculation of the center of gravity. After the orientation of the zipper half of the body is obtained by calculating the moment of inertia, significant points such as the top of the head, the tips oft he hands and foot are heuristically located. In addition, the elbow and knee positions are estimated from the detected (significant) points using a genetic algorithm based learning procedure.
The experimental results demonstrate the robustness of the proposed algorithm and real-time (faster than 20 frames per second) performance.

We propose a novel method of extracting a moving object region from each frame in a series of images regardless of complex, changing background using statistical knowledge about the target. In vision systems for 'real worlds' like a human motion tracker, a priori knowledge about the target and environment is often limited (e.g., only the approximate size of the target is known) and is insufficient for extracting the target motion directly. In our approach, information about both target object and environment is extracted with a small amount of given knowledge about the target object. Pixel value (color, intensity, etc.) distributions for both the target object and background region are adaptively estimated from the input image sequence based on the knowledge. Then, the probability of each pixel being associated with the target object is calculated. The target motion can be extracted from the calculated stochastic image. We confirmed the stability of this approach through experiments.

We describe a method for detecting hand position, posture, and finger bendings using multiple camera images. Stable detection can be achieved using distance transformed images. We detect the maximum point in each distance transformed image as the center of gravity (COG) point of the hand region and calculate its 3D position by stereo matching. The distance value of a COG point varies according to the angle between the camera axis and normal axis of the hand plane. Hand rotation angle can be determined in maximum likelihood estimation from the distance values in all camera images. Using the detected position and posture, the best camera for hand shape detection can be selected. This camera selection makes the hand shape detection simple and stable. This system can be used as a user interface device in a virtual environment, replacing glove-type devices and overcoming most of the disadvantages of contact-type devices.

A new method for estimating human posture from multiple images using a genetic algorithm is proposed. In the proposed algorithm, the posture parameters to be estimated are assigned to the genes of an individual in the population. For each individual, its fitness evaluates to what extent the human multiple images synthesized by deforming a 3D human model according to the values of the genes are registered to the real human multiple images. Genetic operations such as natural selection, crossover and mutation are performed so that individuals in the next generation are generated. After a certain number of repetitions of these processes, the estimated parameter values are obtained from the individual with the best fitness. Experiments using multiple synthesized images show promising results for estimating 17 joint angle values for each degree of freedom of the joints and also the three translational and three rotational degrees of freedom.

This paper proposes a new real-time method for estimating human body postures from thermal images acquired by an infrared camera, regardless of the background and lighting conditions. Distance transformation is performed for the human body area extracted from the thresholded thermal image, in order to calculate the center of gravity. After the orientation of the upper half of the body is obtained by calculating the moment of inertia, significant points such as the top of the head and the ends of the hands and feet are heuristically located. In addition, the elbow and knee positions are estimated from the detected (significant) points, using a genetic-algorithm-based learning procedure. The experimental results demonstrate the robustness of the proposed algorithm and real-time performance (faster than 20 frames per second).

In our "Virtual KABUKI System", to estimate human body postures, a human silhouette must be segmented from a background image. We had used an infrared camera to segment it. But clothes which are hard to have a heat, such as a skirt and a jacket, obstruct a detection of a complete human silhouette. To solve this problem, we developed the new method which uses a difference between a current image and a background image. This paper introduce a new real-time method to estimate human body postures from color images acquired by a normal CCD camera.

This paper describes the Virtual Kabuki Theater the authors have recently developed. In the Virtual Kabuki Theater, people at different locations can be Kabuki actors in a Kabuki scene. In our system, Kabuki actors'3D models are created in advance. Facial expressions of a person are detected in real-time in the face images from the small camera fixed to the helmet worn by the person. Body movements of the person are estimated in real-time from the thermal images acquired by the infrared camera that observes the person. The detected expressions and body movements are reproduced in the Kabuki actor's model. Our implementation shows good performance. The Virtual Kabuki Theater is a first step towards human metamorphosis systems, in which anyone can change (metamorphose) his/her form into any other characters.

A method for converting one person’s facial expression into another person’s is proposed. The sequence of the feature vector for each expression is modeled by using HMM with the hidden states corresponding to the different muscle conditions (relaxed, contracting, and the end of contraction). The probability of each state is evaluated for each frame and the contraction rate of each muscle is obtained from the probability of each state using a matrix representing the characteristics of other people’s expressions. The experiments showed the superior realism of the expression generated by our proposed method.

A method that can be used for for recognizing facial expressions of multiple persons is proposed. In this method, the condition of facial muscles is assigned to a hidden state of a HMM for each expression. Then, the probability of the state is updated according to a feature vector obtained from image processing. Image processing is performed in two steps. First, a velocity vector is estimated from every two successive frames by using an optical flow algorithm. Then, a two-dimensional Fourier transform is applied to a velocity vector field at the regions around an eye and the mouth. The coefficients for lower frequencies are selected to form a feature vector. A mixture density is used for approximating the output probability of the HMM so as to represent a variation in facial expressions among persons. To cope with the case when two expressions are displayed contiguously, the HMM computation is modified such that when the peak of a facial motion is detected, a new sequence of facial expressions is assumed to start from the previous frame with minimal facial motion. Experiments show that a mixture density is effective because recognition accuracy improves as the number of mixtures increases. In addition, the method correctly recognizes a facial expression that contiguously follows another one.

This paper introduces a new;real-time method to estimate the posture of a human from thermal images acquired by an infrared camera regardless of the background and lighting conditions. Distance transformation is performed for the human body area extracted from the thresholded thermal image for the calculation of the center of gravity. After the orientation of the upper half of the body is obtained by calculating the moment of inertia, significant points such as the top of the head, the tips of the hands and foot are heuristically located. In addition, the elbow and knee positions are estimated from the detected (significant) points using a genetic algorithm based learning procedure.
The experimental results demonstrate the robustness of the proposed algorithm and rear-time (faster than 20 frames per second) performance.

A method for recognizing facial expressions in which a feature vector is obtained from Fiourier transform coefficients of velocity vector on a face while an expressions is displayed. As the feature vector is computed from the gradient based optical flow algorithm without time-consuming iteration, the method makes a real-time processing possible.

In this paper we present an algorithm to generate new views of a human face, starting with at least two other views of the face. In a typical face recognition system, the task of comparison becomes easier if the faces have similar orientation with respect to the camera. The affine coordinate based reprojection algorithm presented in the paper enables us to do that. Dense point matches between the two input faces of the same individual are computed using an affine coordinate based reprojection framework. This is followed by the reprojection of one of these to faces to the target face once the user has matched four feature points across two input face images and the target face image.

In this paper we present an algorithm to generate new views of a scene, starting from stereo images. Errors in 3D scene reconstruction usually get reflected in the quality of the new scene generated, so we seek a direct method for reprojection. In this paper, we use the knowledge of dense point matches and their affine coordinate values to estimate the corresponding affine coordinate values in the new scene.

A new method for the real-time detection of facial expressions from time-sequential images is proposed. The proposed method does not require the use of tape marks that are pasted to the face to detect expressions in real-time in the current implementation for virtual space teleconferencing. In the proposed method, four windows are applied to four areas of a facial image : left and right eyes, mouth and forehead. Each window is divided into blocks consisting of 8 by 8 pixels. Discrete cosine transform (DCT) is applied to each block, and the feature vector of each window is obtained by taking the summations of the DCT energies in the horizontal, vertical and diagonal directions. To convert the DCT features to virtual tape mark movements, we represent the displacement of a virtual tape mark by a polynomial of the DCT features for the three directions. We apply a genetic algorithm to train facial expression image sequences to find the optimal set of coefficients that minimizes the difference between the real and converted displacements of the virtual tape marks. Experimental results shows the effectiveness of the proposed method.

We propose a shape reconstruction and color inferring method for 3D objects based on planar symmetry using an active camera. This efficiently creates 3-D models of objects from the binary silhouettes in some views acquired by the active camera whose viewpoints are selected recursively based on the symmetry planes of observed objects. And the lighting condition is measured using color samples before inferring the color of observed objects. The original color data of the plane is inferred from the observed RGB data of the color samples, RGB data of the object observed by the CCD camera, and the surface normal to the point on the surface of the object.

This paper describes realistic, real-time 3-D reproduction of human body movements based on detection data from a small number of sensors attached to a participant for virtual space teleconferencing. A 3-D human model is created by a skeletal link model. WFM (Wire Frame Model), body segments for FFD (Free Form Deformations), and IFP (Implicit Function Primitives) for body parts. In this model WFM provides fast rendering ability and FFD and IFP give natural deformation representation. A reconstruction algorithm that can deal with human movements in teleconference situations is proposed
sensed data are used to reconstruct whole body movements by utilizing motion constraints. Experiments, in which four magnetic sensors for detecting the six pose parameters are attached to the upper half of a human body, show the effectiveness of the proposed 3-D human model and reconstruction method, in which human movements are reproduced at a speed of 8 frames/sec.

This paper describes a virtual space teleconferencing that proposed as an human-oriented telecommunication system. By using this system, participants at different sites can engage in a conference with the sensation of sharing the same space, and cooperative work can be performed among remotely located participants. More specifically, our system connects three different sites via a 1.5 Mbps ISDN now in commercial use. The system has two large screens that can provide real-time reproduction of 3-D whole body human images. Participants at the three different sites are able to feel as if they are all at the same site. In this experimental system, to achieve real-time reproduction, tape marks, a data glove and magnetic sensors are used. However, these tools are not appropriate for natural human communications. It is a final goal to develop a system that does not require such tools. We also describe a new facial expression detection method that does not need the tape marks, and real-time detection is realized by exploiting frequency domain data from the face image acquired by the CCD camera. Experiments show very promising results for accurate expression detection and for implementation of real-time hardware of the using propose method.

Facial expression recognition is an important technology fundamental to realize intelligent image coding systems and advanced man-machine interfaces in visual communication systems. In computer vision field, many techniques have been developed to recognize facial expressions. However, most of those techniques are based on static features extracted from one or two still images. Those techniques are not robust against noise and cannot recognize subtle changes in facial expressions. In this paper we use hidden Markov models (HMM) with continuous output probabilities to extract a temporal pattern of facial motion. In order to improve the recognition performance, we propose a new feature obtained from wavelet transform coefficients. For the evaluation, we use 180 image sequences taken from three male subjects. Using these image sequences, the recognition rate for user trained mode achieves 98% compared with 84% using our previous method. The recognition rate for user independent mode achieves 84% when the expressions are restricted to four expressions.

A new method to detect deformations of facial parts from a face image regardless of changes in the position and orientation of a face using the genetic algorithm is proposed. Facial expression parameters that are used to deform and position a 3D face model are assigned to the genes of an individual in a population. The face model is deformed and positioned according to the gene values of each individual and is observed by a virtual camera, and a face image is synthesized. The fitness which evaluates to what extent the real and synthesized face images are similar to each other is calculated. After this process is repeated for sufficient generations, the parameter estimation is obtained from the genes of the individual with the best fitness. Experimental results demonstrate the effectiveness of the method. © 1996 IEEE.

This paper considers the problem of extracting and tracking complex contours without user interaction. We assume that a complex contour consists of contour segments whose spatial coordinates and intensity gradient vary smoothly in the direction normal to themselves. In our algorithm, digital curves that could correspond to contour segments are extracted by connecting edge pixels using a B-spline based contour segment model. The extracted curves trace the contour segments at the next frame by using the active contour model technique. Experimental results show even occluded contours can be tracked automatically. © 1996 IEEE.

This paper describes the Virtual Kabuki Theater the authors have recently developed. In the Virtual Kabuki Theater, people at different locations can be Kabuki actors in a Kabuki scene. In our system, Kabuki actors' SD models are created in advance. Facial expressions of a person are detected an real-tame in the face images from the small camera fixed to the helmet worn by the person. Body movements of the person are estimated in real-time from the thermal images acquired by the infrared camera that observes the person. The detected expressions and body movements are reproduced in. the Kabuki actor's model. Our implementation shows good performance. The Virtual Kabuki Theater is a first step towards human metamorphosis systems, in which anyone can change (metamorphose) his/her form into any other characters.

In this paper, we present a method of obtaining the approximate transformation parameter values as a starting point in estimating the pose of rigid 3D free form objects using multiple 2D images. We back project the edge silhouettes in the images, and obtain the approximate volume in the 3D space containing the object. Next, for a point selected in the volume, we hypothesize a set of points within the 3D CAD model of the object it can possibly correspond to, using the spatial extent function introduced in this paper. This is repeated for three arbitrarily chosen point in the volume. The hypothesized (match point) lists of these three points are next used to derive the pose parameter by enforcing the conditions of rigidity. Our initial experiments demonstrate the potential of this idea, and the pose parameters estimated using this method can be refined using the standard methods available in the literature.

A method for recognizing facial expressions from time-sequential images by using Hidden Markov Models (HMM) is proposed. HMM has the advantage that it can process time-sequential infomation. Moreover we can expect the HMM to make generalizations from the training data because of its learning procedure. Each image of a facial expression is transformed into an image feature vector. Each element of the feature vector consists of the average power from a distinct frequency band obtained by applying the Wavelet transformation to the image. The sequence is converted into a symbol sequence by using a new category-separated vector quantization. The codebook is constructed by appending codewords selected from other categories to each category to reduce the probability of wrong symbolization for similar facial expressions. To recognize an observed sequence, the HMM that best matches the sequence is chosen, and the category of the HMM is the recognized expression. Experiments for recognizing 4 expressions result in a promising recognition rate of 93.7%.

Real-time reproduction of 3D human images is realized by an experimental system recently built as a prototype for virtual space teleconferencing, in which participants at different sites can feel as if they are colocated and can work cooperatively, At each sending and receiving site of the teleconferencing system, a 3D model of each participant is constructed from a wire frame model mapped by color texture and is rendered on a 3D display, In the current experimental system, real-time detection of facial features at the sending site is achieved by visually tracking tape marks pasted to the participant's face. Movements of the head, body, hands, and fingers are detected in real time using magnetic sensors and data gloves. At the receiving site, the detected motion parameters are used to move nodes in the wire frame model to reproduce the movements of the participants at each sending site. Realistic facial expressions are reproduced through the-use of simple motion rules applied to the tape mark tracking information, Through experimental evaluation, the optimal number of nodes for best quality has been obtained. Reproduction of facial expressions and synthesis of arm movements are examined, The reproduction speed using the optimal human model is approximately 6 frames/s. Examples of cooperative work by participants using the experimental system illustrate the effectiveness of virtual space teleconferencing. (C) 1995 Academic Press, Inc.

Human figure animation is a widely researched area with many applications. This paper addresses specific issues that deal with the synthesis, animation and environmental interaction of human figures within a virtual space teleconferencing system. A layered representation of the human figure is adopted. Skeletal posture is determined from magnetic sensors on the body, using heuristics and inverse kinematics. This paper describes the use of implicit function techniques in the synthesis and animation of a polymesh geometric skin over the skeletal structure. Implicit functions perform detection and handling of collisions with an optimal worst case time complexity that is linear in the number polymesh vertices. Body deformations resulting from auto-collisions are handled elegantly and homogeneously as part of the environment. Further, implicit functions generate precise collision contact surfaces and have the capability to model the physical characteristics of muscles in systems that employ force feedback. The real time implementation within a virtual space teleconferencing system, illustrates this new approach, coupling polymesh and implicit surface based modeling and animation techniques.

An effective algorithm for character recognition in scene images is studied. Scene images are segmented into regions by an image segmentation method based on adaptive thresholding. Character candidate regions are detected by observing gray-level differences between adjacent regions. To ensure extraction of multisegment characters as well as single-segment characters, character pattern candidates are obtained by associating the detected regions according to their positions and gray levels. A character recognition process selects patterns with high similarities by calculating the similarities between character pattern candidates and the standard patterns in a dictionary and then comparing the similarities to the thresholds. A relaxational approach to determine character patterns updates the similarities by evaluating the interactions between categories of patterns, and finally character patterns and their recognition results are obtained. Highly promising experimental results have been obtained using the method on 100 images involving characters of different sizes and formats under uncontrolled lighting.

This paper discusses a similarity image retrieval method in an image database which corresponds to diversified human similarity sensation. In the proposed similarity image retrieval, a similarity between images is examined by subjective evaluation, and a multidimensional psychological space is constructed using multidimensional scaling. Each of the coordinate axes in the multidimensional psychological space is represented by a linear combination of image feature parameters using multiple regression analysis. The image feature parameters extracted from a retrieval key image are converted into a point in the multidimensional psychological space using part or all of the forementioned linear combination expressions. The image with the minimum distance among the stored images is given as the retrieval output. Two-hundred sixty binary patterns of a butterfly are evaluated by a subjective test, and obtained similarities between patterns are used as the true values. The error is examined between the forementioned result and the value estimated in the multidimensional psychological space calculated from a linear combination expression of image feature parameters. As a result, the optimal dimension of the multidimensional psychological space is determined as 15. Retrieval performance in the space with the optimal dimension is evaluated from viewpoints of retrieval rate and mixture of dissimilar patterns into the retrieval result. Usefulness of the method is verified. Axis implications in the multidimensional psychological space are examined, and the possibility to retrieve an image from an individual viewpoint is indicated.

We present a new method for the control of a robot manipulator. The method assumes virtually no prior Information about the robot's kinematic behavior and therefore avoids the costly and time-consuming calibration required by many existing methods. Instead, it relies on continuous sensory feedback, integrating visual data and kinematics into a single mapping, called the perceptual kinematic map (PKM), from the robot's joint space to the camera space. We present results from a real-time, software simulation of a manipulator using a PKM controller.

A homotopy describes the transformation of one arbitrary curve into another that shares the same endpoints. In this paper, we propose a deformable cylinder model, based on homotopy, in which an arbitrary surface interpolated between two contours via a blending function is transformed into another surface over time. We then show how this homotopic deformation can be applied to the realistic animation of human faces in a virtual space teleconferencing system. Specifically, we show that facial expressions such as wrinkling of the forehead and opening and closing of the mouth can be synthesized and animated in real time through 3D homotopic deformations.

A human action recognition method based on a hidden Markov model (HMM) is proposed. It is a feature-based bottom-up approach that is characterized by its learning capability and time-scale invariability. To apply HMMs, one set of time-sequential images is transformed into an image feature vector sequence, and the sequence is converted into a symbol sequence by vector quantization. In learning human action categories, the parameters of the HMMs, one per category, are optimized so as to best describe the training sequences from the category. To recognize an observed sequence, the HMM which best matches the sequence is chosen. Experimental results for real time-sequential images of sports scenes show recognition rates higher than 90%. The recognition rate is improved by increasing the number of people used to generate the training data, indicating the possibility of establishing a person-independent action recognizer.

We present a method for recognizing polyhedral objects from range images. An object is said to be recognized as one of the models of a library of object models when many features of the model can be made to match the features of the observed object by the same rotation-translation transformation (the object pose). In the proposed approach, the number of considered pairs of image and model features is reduced by selecting at random only a few of all the possible image features and matching them to appropriate model features. The rotation and translation required for each match are computed, and a robust LMS (Least Median of Squares) method is applied to determine clusters in translation and rotation spaces. The validity of the object pose suggested by the clusters is verified by a similarity measure which evaluates how well a model in the suggested pose would fit the original range image. The pose estimation and verification are performed for all models in the model library. The recognized model is the model which yields the smallest value of the similarity measure, and the pose of the object is found in the process.