研究者詳細 - 斎藤　英雄

論文 - 斎藤　英雄

分割表示 >> ／全件表示 356 件中 1 - 356 件目

InpaintFusion: Incremental RGB-D Inpainting for 3D Scenes

Mori S., Erat O., Broll W., Saito H., Schmalstieg D., Kalkofen D.

IEEE Transactions on Visualization and Computer Graphics （IEEE Transactions on Visualization and Computer Graphics） 26 （ 10 ） 2994 - 3007 2020年10月

ISSN 10772626

© 1995-2012 IEEE. State-of-the-art methods for diminished reality propagate pixel information from a keyframe to subsequent frames for real-time inpainting. However, these approaches produce artifacts, if the scene geometry is not sufficiently planar. In this article, we present InpaintFusion, a new real-time method that extends inpainting to non-planar scenes by considering both color and depth information in the inpainting process. We use an RGB-D sensor for simultaneous localization and mapping, in order to both track the camera and obtain a surfel map in addition to RGB images. We use the RGB-D information in a cost function for both the color and the geometric appearance to derive a global optimization for simultaneous inpainting of color and depth. The inpainted depth is merged in a global map by depth fusion. For the final rendering, we project the map model into image space, where we can use it for effects such as relighting and stereo rendering of otherwise hidden structures. We demonstrate the capabilities of our method by comparing it to inpainting results with methods using planar geometric proxies.

Access to Document (DOI)

Resolving position ambiguity of imu-based human pose with a single RGB camera

Kaichi T., Maruyama T., Tada M., Saito H.

Sensors (Switzerland) （Sensors (Switzerland)） 20 （ 19 ） 1 - 12 2020年10月

ISSN 14248220

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. Human motion capture (MoCap) plays a key role in healthcare and human–robot collaboration. Some researchers have combined orientation measurements from inertial measurement units (IMUs) and positional inference from cameras to reconstruct the 3D human motion. Their works utilize multiple cameras or depth sensors to localize the human in three dimensions. Such multiple cameras are not always available in our daily life, but just a single camera attached in a smart IP devices has recently been popular. Therefore, we present a 3D pose estimation approach from IMUs and a single camera. In order to resolve the depth ambiguity of the single camera configuration and localize the global position of the subject, we present a constraint which optimizes the foot-ground contact points. The timing and 3D positions of the ground contact are calculated from the acceleration of IMUs on foot and geometric transformation of foot position detected on image, respectively. Since the results of pose estimation is greatly affected by the failure of the detection, we design the image-based constraints to handle the outliers of positional estimates. We evaluated the performance of our approach on public 3D human pose dataset. The experiments demonstrated that the proposed constraints contributed to improve the accuracy of pose estimation in single and multiple camera setting.

Access to Document (DOI)

Pose estimation of primitive-shaped objects from a depth image using superquadric representation

Hachiuma R., Saito H.

Applied Sciences (Switzerland) （Applied Sciences (Switzerland)） 10 （ 16 ） 2020年08月

© 2020 by the authors. This paper presents a method for estimating the six Degrees of Freedom (6DoF) pose of texture-less primitive-shaped objects from depth images. As the conventional methods for object pose estimation require rich texture or geometric features to the target objects, these methods are not suitable for texture-less and geometrically simple shaped objects. In order to estimate the pose of the primitive-shaped object, the parameters that represent primitive shapes are estimated. However, these methods explicitly limit the number of types of primitive shapes that can be estimated. We employ superquadrics as a primitive shape representation that can represent various types of primitive shapes with only a few parameters. In order to estimate the superquadric parameters of primitive-shaped objects, the point cloud of the object must be segmented from a depth image. It is known that the parameter estimation is sensitive to outliers, which are caused by the miss-segmentation of the depth image. Therefore, we propose a novel estimation method for superquadric parameters that are robust to outliers. In the experiment, we constructed a dataset in which the person grasps and moves the primitive-shaped objects. The experimental results show that our estimation method outperformed three conventional methods and the baseline method.

Access to Document (DOI)

CorsNet: 3D Point Cloud Registration by Deep Neural Network

Kurobe A., Sekikawa Y., Ishikawa K., Saito H.

IEEE Robotics and Automation Letters （IEEE Robotics and Automation Letters） 5 （ 3 ） 3960 - 3966 2020年07月

© 2016 IEEE. Point cloud registration is a key problem for robotics and computer vision communities. This represents estimating a rigid transform which aligns one point cloud to another. Iterative closest point (ICP) is a well-known classical method for this problem, yet it generally achieves high alignment only when the source and template point cloud are mostly pre-aligned. If each point cloud is far away or contains a repeating structure, the registration often fails because of being fallen into a local minimum. Recently, inspired by PointNet, several deep learning-based methods have been developed. PointNetLK is a representative approach, which directly optimizes the distance of aggregated features using gradient method by Jacobian. In this paper, we propose a point cloud registration system based on deep learning: CorsNet. Since CorsNet concatenates the local features with the global features and regresses correspondences between point clouds, not directly pose or aggregated features, more useful information is integrated than the conventional approaches. For comparison, we also developed a novel deep learning approach (DirectNet) that directly regresses the pose between point clouds. Through our experiments, we show that CorsNet achieves higher accuracy than not only the classic ICP method, but also the recently proposed learning-based proposal PointNetLK and DirectNet, including on seen and unseen categories.

Access to Document (DOI)

Pose Estimation of Stacked Rectangular Objects from Depth Images

Matsuno D., Hachiuma R., Saito H., Sugano J., Adachi H.

IEEE International Symposium on Industrial Electronics （IEEE International Symposium on Industrial Electronics） 2020-June 1409 - 1414 2020年06月

ISSN 9781728156354

© 2020 IEEE. This paper addresses the task of six degrees of freedom (6-DoF) pose estimation of stacked rectangular objects from depth images. Object pose estimation is one of the key challenges for visual processing systems since it plays a vital role in many situations such as warehouse/factory automation, robotic manipulation, and augmented reality. Many recent approaches to object pose estimation use RGB information for detecting and estimating the pose of objects. However, in warehouse/factory automation, objects are often small, occluded, cluttered, and texture-less which makes it difficult to utilize RGB features for detection and pose estimation. In order to overcome this restriction, we use only the depth information (without RGB information) and its geometric features to segment each object and to estimate the 6-DoF (position and orientation) in a stacked scene. We segment the rectangular objects in each scene from the depth and surface normal discontinuities (geometric segmentation). From the geometrically segmented image, four object corner points can be estimated using the convex hull detection and the eight corner points, which are required for the 6-DoF pose estimation, can be calculated. To improve the accuracy of orientation estimation, we estimate four orientation candidates and select the best among them. Experimental results using two evaluation methods show that our method outperformed the baseline method.

Access to Document (DOI)

Detection of cervical myelopathy with Leap Motion Sensor by random forests

Watanabe M., Sugiura Y., Saito H., Koyama T., Fujita K.

LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies （LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies） 214 - 216 2020年03月

ISSN 9781728170633

© 2020 IEEE. This paper presents a method for automatic detection of cervical myelopathy by using Leap Motion Sensor for sensing hand motion to apply a method called the 10-s grip and release test. Leap Motion data is analyzed for computing feature of hand motion, which is used for automatic detection of patient or healthy people by random forests algorithm for machine learning. For evaluating automatic detection performance, we collect real data of hand motion 54 of patient and healthy people, which is used for validating accuracy of detection.

Access to Document (DOI)

Gait Measurement at Home Using A Single RGB Camera

Yagi K., Sugiura Y., Hasegawa K., Saito H.

Gait and Posture （Gait and Posture） 76 136 - 140 2020年02月

ISSN 09666362

© 2019 Elsevier B.V. Background: An individual's gait is a key factor for consideration in evaluating their overall health. Several medical studies have demonstrated the correlation between gait and incidence rate of diseases, mortality, and risk of fall. However, gait is only occasionally evaluated during medical visits, which may delay the detection of health problems. Methods: In this paper, we propose a gait measurement system that is suitable for use at home. Our method requires only a single RGB camera, whereas other visionary sensor-based methods require depth sensors or multiple RGB cameras. In addition, the setup for the measurement is easy. What the user has to do is only putting a single camera in a room and choosing four location known points on the floor. Our method can measure step positions and step timings, and therefore, other important parameters such as stride length, step width, walking speed, and cadence may also be captured. The individual's gait is captured by the camera, and therefore the user is not required to wear any devices. Results: In the experiment described herein, we demonstrate our method's accuracy by comparing it with the motion capture system. The results indicate that our method can measure walking speed with an error of 3.62 cm/s from the side view, and which is too small a change to be clinically meaningful.

Access to Document (DOI)

Rgb-d image inpainting using generative adversarial network with a late fusion approach

Fujii R., Hachiuma R., Saito H.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) （Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)） 12242 LNCS 440 - 451 2020年

ISSN 9783030584641

© Springer Nature Switzerland AG 2020. Diminished reality is a technology that aims to remove objects from video images and fills in the missing region with plausible pixels. Most conventional methods utilize the different cameras that capture the same scene from different viewpoints to allow regions to be removed and restored. In this paper, we propose an RGB-D image inpainting method using generative adversarial network, which does not require multiple cameras. Recently, an RGB image inpainting method has achieved outstanding results by employing a generative adversarial network. However, RGB inpainting methods aim to restore only the texture of the missing region and, therefore, does not recover geometric information (i.e, 3D structure of the scene). We expand conventional image inpainting method to RGB-D image inpainting to jointly restore the texture and geometry of missing regions from a pair of RGB and depth images. Inspired by other tasks that use RGB and depth images (e.g., semantic segmentation and object detection), we propose late fusion approach that exploits the advantage of RGB and depth information each other. The experimental results verify the effectiveness of our proposed method.

Access to Document (DOI)

In-Plane Rotation-Aware Monocular Depth Estimation Using SLAM

Saito Y., Hachiuma R., Yamaguchi M., Saito H.

Communications in Computer and Information Science （Communications in Computer and Information Science） 1212 CCIS 305 - 317 2020年

ISSN 9789811548178

© 2020, Springer Nature Singapore Pte Ltd. Estimating accurate depth from an RGB image in any environment is challenging task in computer vision. Recent learning based method using deep Convolutional Neural Networks (CNNs) have driven plausible appearance, but these conventional methods are not good at estimating scenes that have a pure rotation of camera, such as in-plane rolling. This movement imposes perturbations on learning-based methods because gravity direction is considered to be strong prior to CNN depth estimation (i.e., the top region of an image has a relatively large depth, whereas bottom region tends to have a small depth). To overcome this crucial weakness in depth estimation with CNN, we propose a simple but effective refining method that incorporates in-plane roll alignment using camera poses of monocular Simultaneous Localization and Mapping (SLAM). For the experiment, we used public datasets and also created our own dataset composed of mostly in-plane roll camera movements. Evaluation results on these datasets show the effectiveness of our approach.

Access to Document (DOI)

Global-map-registered local visual odometry using on-the-fly pose graph updates

Yamaguchi M., Mori S., Saito H., Yachida S., Shibata T.

ISSN 9783030584641

© Springer Nature Switzerland AG 2020. Real-time camera pose estimation is one of the indispensable technologies for Augmented Reality (AR). While a large body of work in Visual Odometry (VO) has been proposed for AR, practical challenges such as scale ambiguities and accumulative errors still remain especially when we apply VO to large-scale scenes due to limited hardware and resources. We propose a camera pose registration method, where a local VO is consecutively optimized with respect to a large-scale scene map on the fly. This framework enables the scale estimation between a VO map and a scene map and reduces accumulative errors by finding corresponding locations in the map to the current frame and by on-the-fly pose graph optimization. The results using public datasets demonstrated that our approach reduces the accumulative errors of naïve VO.

Access to Document (DOI)

DetectFusion: Detecting and segmenting both known and unknown dynamic objects in real-time slam

Hachiuma R., Pirchheim C., Schmalstieg D., Saito H.

30th British Machine Vision Conference 2019, BMVC 2019 （30th British Machine Vision Conference 2019, BMVC 2019） 2020年

© 2019. The copyright of this document resides with its authors. We present DetectFusion, an RGB-D SLAM system that runs in real time and can robustly handle semantically known and unknown objects that can move dynamically in the scene. Our system detects, segments and assigns semantic class labels to known objects in the scene, while tracking and reconstructing them even when they move independently in front of the monocular camera. In contrast to related work, we achieve real-time computational performance on semantic instance segmentation with a novel method combining 2D object detection and 3D geometric segmentation. In addition, we propose a method for detecting and segmenting the motion of semantically unknown objects, thus further improving the accuracy of camera tracking and map reconstruction. We show that our method performs on par or better than previous work in terms of localization and object reconstruction accuracy, while achieving about 20 fps even if the objects are segmented in each frame.

Deep Selection: A Fully Supervised Camera Selection Network for Surgery Recordings

Hachiuma R., Shimizu T., Saito H., Kajita H., Takatsume Y.

ISSN 9783030597153

© 2020, Springer Nature Switzerland AG. Recording surgery in operating rooms is an essential task for education and evaluation of medical treatment. However, recording the desired targets, such as the surgery field, surgical tools, or doctor’s hands, is difficult because the targets are heavily occluded during surgery. We use a recording system in which multiple cameras are embedded in the surgical lamp, and we assume that at least one camera is recording the target without occlusion at any given time. As the embedded cameras obtain multiple video sequences, we address the task of selecting the camera with the best view of the surgery. Unlike the conventional method, which selects the camera based on the area size of the surgery field, we propose a deep neural network that predicts the camera selection probability from multiple video sequences by learning the supervision of the expert annotation. We created a dataset in which six different types of plastic surgery are recorded, and we provided the annotation of camera switching. Our experiments show that our approach successfully switched between cameras and outperformed three baseline methods.

Access to Document (DOI)

Surgery recording without occlusions by multi-view surgical videos

Shimizu T., Oishi K., Hachiuma R., Kajita H., Takatsume Y., Saito H.

VISIGRAPP 2020 - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications （VISIGRAPP 2020 - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications） 5 837 - 844 2020年

ISSN 9789897584022

Copyright © 2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved. Recording surgery is important for sharing various operating techniques. In most surgical rooms, fixed surgical cameras are already installed, but it is almost impossible to capture the surgical field because of occlusion by the surgeon’s head and body. In order to capture the surgical field, we propose the installation of multiple cameras in a surgical lamp system, so that at least one camera can capture the surgical field even when the surgeon’s head and body occlude other cameras. In this paper, we present a method for automatic viewpoint switching from multi-view surgical videos, so that the surgical field can always be recorded. We employ a method for learning-based object detection from videos for automatic evaluation of the surgical field from multi-view surgical videos. In general, frequent camera switching degrades the video quality of view (QoV). Therefore, we apply Dijkstra’s algorithm, widely used in the shortest path problem, as an optimization method for this problem. Our camera scheduling method works so that camera switching is not performed for the minimum frame we specified, and therefore the surgical field observed in the entire video is maximized.

MMSports'19: 2nd ACM international workshop on multimedia content analysis in sports

Lienhart R., Moeslund T.B., Saito H.

MM 2019 - Proceedings of the 27th ACM International Conference on Multimedia （MM 2019 - Proceedings of the 27th ACM International Conference on Multimedia） 2732 - 2733 2019年10月

ISSN 9781450368896

© 2019 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-6889-6/19/10. The second ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'19) is held in Nice, France on October 25th, 2019 co-located with the ACM International Conference on Multimedia 2019 (ACM Multimedia 2019). The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding and visualizing the multimedia/multimodal data in sports. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation, understanding, statistical analysis and evaluation, and sensor fusion. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports.

Access to Document (DOI)

Prediction of future shot direction using pose and position of tennis player

Shimizu T., Hachiuma R., Saito H., Yoshikawa T., Lee C.

MMSports 2019 - Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports, co-located with MM 2019 （MMSports 2019 - Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports, co-located with MM 2019） 59 - 66 2019年10月

ISSN 9781450369114

© 2019 Copyright held by the owner/author(s). In this paper, we propose a method to predict the future shot direction in a tennis match using pose information and player position. As far as we know, there is no work that deals with such a predictive task, so there is no shot direction dataset as yet. Therefore, using a YouTube tennis match video, we construct an time of impact and shot direction dataset. To reduce annotation costs, we propose a method to automatically label the shot direction. Moreover, we propose a method to predict the future shot direction using the constructed dataset. The shot direction is predicted using LSTM(long short-time memory), from sequential pose information up to the time of impact and the player position. We employ OpenPose to extract the position of skeleton joints. In the experiment, we evaluate the accuracy of shot direction prediction and verify the effectiveness of the proposed method. Since there are no studies that predict future shot direction, we set four baseline methods to evaluate the effectiveness of our proposed method.

Access to Document (DOI)

Volumetric representation of human body parts using superquadrics

Hachiuma R., Saito H.

Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019 （Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019） 187 - 188 2019年10月

ISSN 9781728147659

© 2019 IEEE. Superquadrics are one of the ideal shape representations for adapting various kinds of primitive shapes with a single equation. This paper revisits the task of representing a 3D human body with multiple superquadrics. As a single superquadric surface can only represent symmetric primitive shapes, we present a method that segments the human body into body parts to estimate their superquadric parameters. Moreover, we propose a novel initial parameter estimation method by using 3D skeleton joints. The results show that superquadric parameters are estimated, which represent human body parts volumetrically.

Access to Document (DOI)

Automatic viewpoint switching for multi-view surgical videos

Shimizu T., Oishi K., Saito H., Kajita H., Takatsume Y.

ISSN 9781728147659

© 2019 IEEE. Recording medical surgery operations is important for sharing the various operating techniques. In most operating rooms, fixed surgery cameras are already installed, but it is almost impossible to capture the surgical field because of occlusion by the surgeon's head and body. In order to capture the surgical field in real surgery operations, we proposed the installation of multiple cameras in a surgical lighting system, so that at least one camera can capture the target surgical field even when the surgeon's head and body occlude other cameras. In this paper, we present a method for automatic viewpoint selection from multi-view surgical videos, so that the surgical field can always be recorded in the output video. We employ a method for learning-based object detection from videos for automatic evaluation of the surgical field area from multiple input videos. By selecting the viewpoint with the largest area of the surgical field, we can virtually reduce the area of the surgeon's head and hands. In general, frequent camera switching degrades the video quality of view (QoV). Therefore, we apply the Dijkstra method widely used in the shortest path problem as a combinatorial optimization method for this problem. Our camera scheduling method is that the camera switching is not performed for a certain period of time, and the surgical field observed in the entire video is maximized.

Access to Document (DOI)

Joint inpainting of RGB and depth images by generative adversarial network with a late fusion approach

Fujii R., Hachiuma R., Saito H.

ISSN 9781728147659

© 2019 IEEE. Image inpainting aims to restore texture of missing regions in scene from an RGB image. In this paper, we aim to restore not only the texture but also the geometry of the missing regions in scene from a pair of RGB and depth images. Inspired by the recent development of generative adversarial network, we employ an encoder-decoderbased generative adversarial network with the input of RGB and depth image. The experimental results show that our method restores the missing region of both RGB and depth image.

Access to Document (DOI)

Incremental class discovery for semantic segmentation with rgbd sensing

Nakajima Y., Kang B., Saito H., Kitani K.

Proceedings of the IEEE International Conference on Computer Vision （Proceedings of the IEEE International Conference on Computer Vision） 2019-October 972 - 981 2019年10月

ISSN 9781728148038

© 2019 IEEE. This work addresses the task of open world semantic segmentation using RGBD sensing to discover new semantic classes over time. Although there are many types of objects in the real-word, current semantic segmentation methods make a closed world assumption and are trained only to segment a limited number of object classes. Towards a more open world approach, we propose a novel method that incrementally learns new classes for image segmentation. The proposed system first segments each RGBD frame using both color and geometric information, and then aggregates that information to build a single segmented dense 3D map of the environment. The segmented 3D map representation is a key component of our approach as it is used to discover new object classes by identifying coherent regions in the 3D map that have no semantic label. The use of coherent region in the 3D map as a primitive element, rather than traditional elements such as surfels or voxels, also significantly reduces the computational complexity and memory use of our method. It thus leads to semi-real-time performance at 10.7 Hz when incrementally updating the dense 3D map at every frame. Through experiments on the NYUDv2 dataset, we demonstrate that the proposed method is able to correctly cluster objects of both known and unseen classes. We also show the quantitative comparison with the state-of-the-art supervised methods, the processing time of each step, and the influences of each component.

Access to Document (DOI)

Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction

Maxence R., Uchiyama H., Kawasaki H., Thomas D., Nozick V., Saito H.

Proceedings - 2019 International Conference on 3D Vision, 3DV 2019 （Proceedings - 2019 International Conference on 3D Vision, 3DV 2019） 574 - 582 2019年09月

ISSN 9781728131313

© 2019 IEEE. The standard photometric stereo is a technique to densely reconstruct objects' surfaces using light variation under the assumption of a static camera with a moving light source. In this work, we use photometric stereo to reconstruct dense 3D scenes while moving the camera and the light altogether. In such non-static case, camera poses as well as correspondences between pixels of each frame to apply photometric stereo are required. ORB-SLAM is a technique that can be used to estimate camera poses. To retrieve correspondences, our idea is to start from a sparse 3D mesh obtained with ORB SLAM and then densify the mesh by a plane sweep method using a multi-view photometric consistency. By combining ORB-SLAM and photometric stereo, it is possible to reconstruct dense 3D scenes with a off-the-shelf smartphone and its embedded torchlight. Note that SLAM systems usually struggle with textureless object, which is effectively compensated by the photometric stereo in our method. Experiments are conducted to show that our proposed method gives better results than SLAM alone or COLMAP, especially for partially textureless surfaces.

Access to Document (DOI)

Eventnet: Asynchronous recursive event processing

Sekikawa Y., Hara K., Saito H.

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition （Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition） 2019-June 3882 - 3891 2019年06月

ISSN 9781728132938

© 2019 IEEE. Event cameras are bio-inspired vision sensors that mimic retinas to asynchronously report per-pixel intensity changes rather than outputting an actual intensity image at regular intervals. This new paradigm of image sensor offers significant potential advantages; namely, sparse and non-redundant data representation. Unfortunately, however, most of the existing artificial neural network architectures, such as a CNN, require dense synchronous input data, and therefore, cannot make use of the sparseness of the data. We propose EventNet, a neural network designed for real-time processing of asynchronous event streams in a recursive and event-wise manner. EventNet models dependence of the output on tens of thousands of causal events recursively using a novel temporal coding scheme. As a result, at inference time, our network operates in an event-wise manner that is realized with very few sum-of-the-product operations--look-up table and temporal feature aggregation--which enables processing of 1 mega or more events per second on standard CPU. In experiments using real data, we demonstrated the real-time performance and robustness of our framework.

Access to Document (DOI)

Simultaneous 3D Tracking and Reconstruction of Multiple Moving Rigid Objects

Ozawa T., Nakajima Y., Saito H.

Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019 （Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019） 2019年05月

ISSN 9781728115719

© 2019 IEEE. Most SLAM works based on the assumption of static scene, so localization of the camera and mapping of the scene can fail and lose accuracy of the scene, including moving objects. This paper presents a method for simultaneous mapping of moving objects in the target scene and localization of the moving camera, based on geometrical segmentation of each temporal frame. Taking advantage of segmentation of the target scene, using only the geometric structure of the scene, our method can estimate relative pose for camera and every geometrically segmented areas even without recognizing each object. For confirming the effectiveness of the proposed method, we experimentally show that our method can estimate relative poses for all segmented areas in the scene, so that we can achieve SLAM for the scene, including multiple moving objects.

Access to Document (DOI)

Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space

Ishikawa Y., Hachiuma R., Ienaga N., Kuno W., Sugiura Y., Saito H.

ISSN 9781728115719

© 2019 IEEE. This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.

Access to Document (DOI)

Automatic eyeglasses replacement for a 3d virtual try-on system

Kobayashi T., Sugiura Y., Saito H., Uema Y.

ACM International Conference Proceeding Series （ACM International Conference Proceeding Series） 2019年03月

ISSN 9781450365475

©2019 Association for Computing Machinery. This paper presents a 3D virtual eyeglasses try-on system for practical use. For fitting eyeglasses in a shop, consumers wish to look at themselves in a mirror while trying on various eyeglass styles. However, for people who need to wear eyeglasses for correcting problems with eyesight, it is impossible for them to clearly observe their face in the mirror without wearing eyeglasses. This makes fitting them for new eyeglasses difficult. This research proposes a virtual try-on system that can be used while wearing eyeglasses.We replace the user's eyeglasses in the input video with new eyeglasses virtually. Moreover, a fast and accurate face tracking tool enables our system to automatically display 3D virtual glasses following a user's head motion. Experimental results demonstrate that the proposed method can render virtual glasses naturally while the user is wearing real eyeglasses.

Access to Document (DOI)

Removing fences from sweep motion videos using global 3D reconstruction and fence-aware light field rendering

Lueangwattana C., Mori S., Saito H.

Computational Visual Media （Computational Visual Media） 5 （ 1 ） 21 - 32 2019年03月

ISSN 20960433

© 2018, The Author(s). Diminishing the appearance of a fence in an image is a challenging research area due to the characteristics of fences (thinness, lack of texture, etc.) and the need for occluded background restoration. In this paper, we describe a fence removal method for an image sequence captured by a user making a sweep motion, in which occluded background is potentially observed. To make use of geometric and appearance information such as consecutive images, we use two well-known approaches: structure from motion and light field rendering. Results using real image sequences show that our method can stably segment fences and preserve background details for various fence and background combinations. A new video without the fence, with frame coherence, can be successfully provided.

Access to Document (DOI)

Self-assessment application of flexion and extension

Ienaga N., Sugiura Y., Saito H., Fujita K.

2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019 （2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019） 150 - 152 2019年03月

ISSN 9781728105437

© 2019 IEEE. The measurement of range of motion during medical examination is a basic procedure in diagnosis and treatment evaluation. However, accurate measurement requires experience, and it is difficult for patients to measure range of motion by themselves. If patients can easily and accurately measure the range by their own, it will encourage patients to continue rehabilitation and will be useful for helping patients to determine when they should see a doctor. We developed a wrist flexion and extension measurement system using images captured by a smart phone so that patients can measure range of motion without expert knowledge or experience. First, the system calculates a middle line for the forearm and an outside line for the dorsal hand based on user inputs. Afterward, the angle formed by these lines is calculated. The experimental results show that the average absolute errors are 4.82° for flexion and 8.74° for extension.

Access to Document (DOI)

Volumetric Representation of Semantically Segmented Human Body Parts Using Superquadrics

Hachiuma R., Saito H.

ISSN 9783030319076

© Springer Nature Switzerland AG 2019. Superquadrics are one of the ideal shape representations for adapting various kinds of primitive shapes with a single equation. This paper revisits the task of representing a 3D human body with multiple superquadrics. As a single superquadric surface can only represent symmetric primitive shapes, we present a method that segments the human body into body parts to estimate their superquadric parameters. Moreover, we propose a novel initial parameter estimation method by using 3D skeleton joints. The results show that superquadric parameters are estimated, which represent human body parts volumetrically.

Access to Document (DOI)

Vehicle trajectory estimation method by drive recorder images and point cloud of surrounding environment

Kurobe A., Kinoshita H., Saito H.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering （Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering） 85 （ 3 ） 274 - 281 2019年

ISSN 09120289

© 2019 Japan Society for Precision Engineering. All rights reserved. The on-the-spot inspections after accident are based on information obtained from the brake marks left on the road and the damage situation of the surroundings. However, this method is affected by the situation before the accident and cannot estimate quantitative guessing. Moreover, technologies such as automatic braking system and lane keeping system, have been developed actively. In developing these technologies, vehicle trajectory is one of the most important data. In this paper, we propose a novel method of estimating vehicle trajectory by using drive recorder images and point cloud which is measured by Mobile Mapping System and LiDAR. In the evaluation experiments, we confirmed the effectiveness of our method by comparing the vehicle poses estimated with RTK-GPS and showing the average errors are within one meter.

Access to Document (DOI)

A Thumb Tip Wearable Device Consisting of Multiple Cameras to Measure Thumb Posture

Ienaga N., Kawai W., Fujita K., Miyata N., Sugiura Y., Saito H.

ISSN 9783030210731

© 2019, Springer Nature Switzerland AG. Today, cameras have become smaller and cheaper and can be utilized in various scenes. We took advantage of that to develop a thumb tip wearable device to estimate joint angles of a thumb as measuring human finger postures is important in terms of human-computer interface and to analyze human behavior. The device we developed consists of three small cameras attached at different angles so the cameras can capture the four fingers. We assumed that the appearance of the four fingers would change depending on the joint angles of the thumb. We made a convolutional neural network learn a regression relationship between the joint angles of the thumb and the images taken by the cameras. In this paper, we captured the keypoint positions of the thumb with a USB sensor device and calculated the joint angles to construct a dataset. The root mean squared error of the test data was 6.23° and 4.75° .

Access to Document (DOI)

Semi-independent stereo visual odometry for different field of view cameras

Truong T.P., Nozick V., Saito H.

ISSN 9783030110086

© Springer Nature Switzerland AG 2019. This paper presents a pipeline for stereo visual odometry using cameras with different fields of view. It gives a proof of concept about how a constraint on the respective field of view of each camera can lead to both an accurate 3D reconstruction and a robust pose estimation. Indeed, when considering a fixed resolution, a narrow field of view has a higher angular resolution and can preserve image texture details. On the other hand, a wide field of view allows to track features over longer periods since the overlap between two successive frames is more substantial. We propose a semi-independent stereo system where each camera performs individually temporal multi-view optimization but their initial parameters are still jointly optimized in an iterative framework. Furthermore, the concept of lead and follow camera is introduced to adaptively propagate information between the cameras. We evaluate the method qualitatively on two indoor datasets, and quantitatively on a synthetic dataset to allow the comparison across different fields of view.

Access to Document (DOI)

Line-based SLAM considering prior distribution of distance and angle of line features in an urban environment

Uehara K., Saito H., Hara K.

Communications in Computer and Information Science （Communications in Computer and Information Science） 983 105 - 127 2019年

ISSN 9783030122089

© Springer Nature Switzerland AG 2019. In this paper, we propose a line-based SLAM from an image sequence captured by a camera mounted on a vehicle in consideration with the prior distribution of line features that detected in an urban environments. Since such scenes captured by the vehicle in urban envirounments can be expected to include a lot of line segments detected from road markings and buildings, we employ line segments as features for our SLAM. We use additional prior regarding the line segments so that we can improve the accuracy of the SLAM. We assume that the angle of the vector of the line segments to the vehicle’s direction of travel conform to four-component Gaussian mixture distribution. We define a new cost function considering the prior distribution and optimize the relative camera pose, position, and the 3D line segments by bundle adjustment. The prior distribution is also extended into 2D, the distance and angle of the line segments. In addition, we make digital maps from the detected line segments. Our method increases the accuracy of localization and corrects tilted lines in the digital maps. We apply our method to both the single-camera system and the multi-camera system for demonstrate the accuracy improvement by the prior distribution of distance and angle of line features.

Access to Document (DOI)

Learning an optimisable semantic segmentation map with image conditioned variational autoencoder

Zhuang P., Sekikawa Y., Hara K., Saito H.

ISSN 9783030306441

© Springer Nature Switzerland AG 2019. Recent semantic segmentation systems have achieved significant improvement by performing pixel-wise training with hierarchical features using deep convolutional neural network models. While the learning process usually requires pixel-level annotated images, it is difficult to get desirable amounts of fine-labeled data and thus the training set size is more likely to be limited, often in thousands. This means that top methods for a dataset can be fine-tuned for a specific situation, making the generalization ability unclear. In real-world applications like self-driving systems, ambiguous region or lack of context information can cause errors in the predicted results. Resolving such ambiguities is crucial for subsequent operations to be performed safely. We are inspired by work from CodeSLAM where optimizable pixel-wise depth representation is learned. We modify the regression method to work on the pixel-wise classification problem. By training a variational auto-encoder network conditioned with a color image, the computed latent space works as a low-dimensional representation of semantic segmentation, which can be efficiently optimized. As a consequence, our model can correct the error or ambiguity of the prediction during the inference phase given useful scene information. We show how this approach works by giving partial scene truth and perform optimization on the latent variable.

Access to Document (DOI)

Image-based center of mass estimation of the human body via 3D shape and kinematic structure

Kaichi T., Mori S., Saito H., Takahashi K., Mikami D., Isogawa M., Kusachi Y.

Sports Engineering （Sports Engineering） 22 （ 3-4 ） 2019年

ISSN 13697072

© 2019, International Sports Engineering Association. This paper presents a method to estimate a time-sequential trajectory of the center of mass (CoM) of an athlete from a multi-view set of cameras. Collecting the CoM typically requires large-scale measuring systems or attaching sensors to the athletes. To mitigate such hardware limitations, the present study takes a multi-view video-based approach. The proposed method reconstructs subjects’ voxels from a set of multi-view frames and weights each voxel with body part-dependent weights to calculate a CoM. Our results, using real data measured in a studio, showed that the proposed method can estimate CoM within 20 mm concerning center of pressure measures.

Access to Document (DOI)

Efficient object-oriented semantic mapping with object detector

Nakajima Y., Saito H.

IEEE Access （IEEE Access） 7 3206 - 3213 2019年

© 2018 IEEE. Incrementally, building a 3D map in which object instances are semantically annotated has a wide range of applications, including scene understanding, human-robot interactions, and simultaneous localization and mapping extensions. Although researchers are developing efficient and accurate systems, these methods still face a critical issue: Real-time processing, because the task requires a series of heavy processing components, e.g., camera pose estimation, 3D map reconstruction, and especially recognition. In this paper, we propose a novel object-oriented semantic mapping approach aiming at overcoming such issues by introducing highly accurate object-oriented semantic scene reconstruction in real time. For high efficiency, the proposed method employs a fast and scalable object detection algorithm for exploiting semantic information from the incoming frames. These outputs are integrated into geometric regions of the 3D map, which are carried by the geometric-based incremental segmentation method. The strategy of assigning class probabilities to each segmented region, not each element (e.g., surfels and voxels), notably reduces the computational cost, as well as the memory footprint. In addition to efficiency, by geometrically segmenting the 3D map first, clear boundaries between objects appear. We complementarily improve the geometric-based segmentation results beyond the geometric only to the semantic-aware representation. We validate the proposed method's accuracy and computational efficiency through experiments in a common office scene.

Access to Document (DOI)

Fast and Accurate Semantic Mapping through Geometric-based Incremental Segmentation

Nakajima Y., Tateno K., Tombari F., Saito H.

IEEE International Conference on Intelligent Robots and Systems （IEEE International Conference on Intelligent Robots and Systems） 385 - 392 2018年12月

ISSN 9781538680940

© 2018 IEEE. We propose an efficient and scalable method for incrementally building a dense, semantically annotated 3D map in real-time. The proposed method assigns class probabilities to each region, not each element (e.g., surfel and voxel), of the 3D map which is built up through a robust SLAM framework and incrementally segmented with a geometric-based segmentation method. Differently from all other approaches, our method has a capability of running at over 30Hz while performing all processing components, including SLAM, segmentation, 2D recognition, and updating class probabilities of each segmentation label at every incoming frame, thanks to the high efficiency that characterizes the computationally intensive stages of our framework. By utilizing a specifically designed CNN to improve the frame-wise segmentation result, we can also achieve high accuracy. We validate our method on the NYUv2 dataset by comparing with the state of the art in terms of accuracy and computational efficiency, and by means of an analysis in terms of time and space complexity.

Access to Document (DOI)

Estimation of center of mass for sports scene using weighted visual hull

Kaichi T., Mori S., Saito H., Takahashi K., Mikami D., Isogawa M., Kimata H.

IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops （IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops） 2018-June 1890 - 1896 2018年12月

ISSN 9781538661000

© 2018 IEEE. This paper presents a method to estimate the 3D position of a center of mass (CoM) of a human body from a set of multi-view images. As a well-known fact, in sports, collections of CoM are important for analyzing the athletes' performance. Most conventional studies in CoM estimation require installing a measuring system (e.g., a force plate or optical motion capture system) or attaching sensors to the athlete. While such systems reliably estimate CoM, casual settings are preferable for simplifying preparations. To address this issue, the proposed method takes a vision-based approach that does not require specialized hardware and wearable devices. Our method calculates subject's CoM using voxels with body parts dependent weighting. This individual voxel reconstruction and voxel-wise weighting reflects the differences in each body shape, and are expected to contribute to higher performance in analysis. The results using real data demonstrated the performance of the proposed method were compared to force plate data, and provided a 3D CoM visualization in a dynamic scene.

Access to Document (DOI)

FusionMLS: Highly dynamic 3D reconstruction with consumer-grade RGB-D cameras

Meerits S., Thomas D., Nozick V., Saito H.

Computational Visual Media （Computational Visual Media） 4 （ 4 ） 287 - 303 2018年12月

ISSN 20960433

© 2018, The Author(s). Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory, making it memory-efficient and scalable to large scenes. Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to state-of-the-art approaches.

Access to Document (DOI)

Estimation of runners’ number of steps, stride length and speed transition from video of a 100-meter race

Yagi K., Hasegawa K., Sugiura Y., Saito H.

MMSports 2018 - Proceedings of the 1st International Workshop on Multimedia Content Analysis in Sports, Co-located with MM 2018 （MMSports 2018 - Proceedings of the 1st International Workshop on Multimedia Content Analysis in Sports, Co-located with MM 2018） 87 - 95 2018年10月

ISSN 9781450359818

© 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. The purpose of this study is sensing movements of 100-m runners from video that is publicly available, for example, Internet broadcasts. Normally, information that can be obtained from a video is limited to the number of steps and average stride length. However, our proposed method makes it possible to measure not only this information, but also time-scale information like every stride length and speed transition from the same input. Our proposed method can be divided into three steps. First, we generate a panoramic image of the 100-m track. By doing this, we can estimate where the runners are running in a frame at the 100-meter scale. Second, we detect whether the runner steps in the frame. For this process, we utilize the detected track lines and leg joint positions of runners. Finally, we project every steps to the overview image of the 100-m track to estimate the stride length at the 100-m scale. In the experiment part, we apply our method to various race videos. We evaluate the accuracy of our method via comparison with the data measured using typical methods. In addition, we evaluate the accuracy of estimation of the number of steps and show visualized runners’ steps and speed transitions.

Access to Document (DOI)

1 st ACM international workshop on multimedia content analysis in sports

Lienhart R., Moeslund T., Saito H.

MM 2018 - Proceedings of the 2018 ACM Multimedia Conference （MM 2018 - Proceedings of the 2018 ACM Multimedia Conference） 2125 - 2126 2018年10月

ISSN 9781450356657

© 2018 Association for Computing Machinery. The first ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'18) is held in Seoul, South Korea on October 26th, 2018 and is co-located with the ACM International Conference on Multimedia 2018 (ACM Multimedia 2018). The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining and content analysis of multimedia/multimodal data in sports. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation, understanding, statistical analysis and evaluation, and sensor fusion. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this first workshop of a serious workshops on multimedia content analysis in sports.

Access to Document (DOI)

Constant velocity 3D convolution

Sekikawa Y., Ishikawa K., Hara K., Yoshida Y., Suzuki K., Sato I., Saito H.

Proceedings - 2018 International Conference on 3D Vision, 3DV 2018 （Proceedings - 2018 International Conference on 3D Vision, 3DV 2018） 343 - 351 2018年10月

ISSN 9781538684252

© 2018 IEEE. We propose a novel three-dimensional (3D)-convolution method, cv3dconv, for detecting spatiotemporal features from videos. It reduces the number of sum-of-products of 3D convolution by thousands of times by assuming the constant moving velocity of the camera. We observed that a specific class of video sequences, such as those captured by an in-vehicle camera, can be well approximated with piece-wise linear movements of 2D features in the temporal dimension. Our principal finding is that the 3D kernel, represented by the constant-velocity, can be decomposed into a convolution of a 2D kernel representing the shapes and a 3D kernel representing the velocity. We derived the efficient recursive algorithm for this class of 3D convolution which is exceptionally suited for sparse data, and this parameterized decomposed representation imposes a structured regularization along the temporal direction. We experimentally verified the validity of our approximation using a controlled dataset, and we also showed the effectiveness of cv3dconv for the visual odometry estimation task using real event camera data captured in urban road scene.

Access to Document (DOI)

3D PixMix: Image Inpainting in 3D Environments

Mori S., Herling J., Broll W., Kawai N., Saito H., Schmalstieg D., Kalkofen D.

Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018 （Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018） 1 - 2 2018年07月

ISSN 9781538675922

© 2018 IEEE. State of the art methods for diminished reality (DR) propagate pixel information from keyframes to later frames to achieve real-time image inpainting in 3D spaces. However, this approach assumes a planar scene and produces artifacts, if the scene geometry is not sufficiently planar. In this paper, we present 3D PixMix, a new real-time inpainting method that addresses non-planar scenes by considering both color and depth information in the inpainting process. We define cost functions for both the color and the geometric appearance in the inpainting scheme and use an RGB-D sensor for depth fusion using a SLAM. Comparison results against the conventional PixMix show that 3D PixMix obtains the equivalent or even better quality in 3D scenes with additional depth information.

Access to Document (DOI)

Superimposing thermal-infrared data on 3D structure reconstructed by RGB visual odometry

Yamaguchi M., Truong T.P., Mori S., Nozick V., Saito H., Yachida S., Sato H.

IEICE Transactions on Information and Systems （IEICE Transactions on Information and Systems） E101D （ 5 ） 1296 - 1307 2018年05月

ISSN 09168532

© 2018 The Institute of Electronics, Information and Communication Engineers. In this paper, we propose a method to generate a threedimensional (3D) thermal map and RGB + thermal (RGB-T) images of a scene from thermal-infrared and RGB images. The scene images are acquired by moving both a RGB camera and an thermal-infrared camera mounted on a stereo rig. Before capturing the scene with those cameras, we estimate their respective intrinsic parameters and their relative pose. Then, we reconstruct the 3D structures of the scene by using Direct Sparse Odometry (DSO) using the RGB images. In order to superimpose thermal information onto each point generated from DSO, we propose a method for estimating the scale of the point cloud corresponding to the extrinsic parameters between both cameras by matching depth images recovered from the RGB camera and the thermal-infrared camera based on mutual information. We also generate RGB-T images using the 3D structure of the scene and Delaunay triangulation. We do not rely on depth cameras and, therefore, our technique is not limited to scenes within the measurement range of the depth cameras. To demonstrate this technique, we generate 3D thermal maps and RGB-T images for both indoor and outdoor scenes.

Access to Document (DOI)

Line-based SLAM using non-overlapping cameras in an urban environment

Kawasaki A., Hara K., Saito H.

IEICE Transactions on Information and Systems （IEICE Transactions on Information and Systems） E101D （ 5 ） 1232 - 1242 2018年05月

ISSN 09168532

© 2018 The Institute of Electronics, Information and Communication Engineers. We propose a method of line-based Simultaneous Localization and Mapping (SLAM) using non-overlapping multiple cameras for vehicles running in an urban environment. It uses corresponding line segments between images taken by different frames and different cameras. The contribution is a novel line segment matching algorithm by warping processing based on urban structures. This idea significantly improves the accuracy of line segment matching when viewing direction are very different, so that a number of correspondences between front-view and rear-view cameras can be found and the accuracy of SLAM can be improved. Additionally, to enhance the accuracy of SLAM we apply a geometrical constraint of urban area for initial estimation of 3D mapping of line segments and optimization by bundle adjustment. We can further improve the accuracy of SLAM by combining points and lines. The position error is stable within 1.5m for the entire image dataset evaluated in this paper. The estimation accuracy of our method is as high as that of ground truth captured by RTK-GPS. Our high accuracy SLAM algorithm can be apply for generating a road map represented by line segments. According to an evaluation of our generating map, true positive rate around the vehicle exceeding 70% is achieved.

Access to Document (DOI)

Simultaneous object segmentation and recognition by merging CNN outputs from uniformly distributed multiple viewpoints

Nakajima Y., Saito H.

IEICE Transactions on Information and Systems （IEICE Transactions on Information and Systems） E101D （ 5 ） 1308 - 1316 2018年05月

ISSN 09168532

© 2018 The Institute of Electronics, Information and Communication Engineers. We propose a novel object recognition system that is able to (i) work in real-time while reconstructing segmented 3D maps and simultaneously recognize objects in a scene, (ii) manage various kinds of objects, including those with smooth surfaces and those with a large number of categories, utilizing a CNN for feature extraction, and (iii) maintain high accuracy no matter how the camera moves by distributing the viewpoints for each object uniformly and aggregating recognition results from each distributed viewpoint as the same weight. Through experiments, the advantages of our system with respect to current state-of-the-art object recognition approaches are demonstrated on the UW RGB-D Dataset and Scenes and on our own scenes prepared to verify the effectiveness of the Viewpoint-Class-based approach.

Access to Document (DOI)

Synchronizing 3D point cloud from 3D scene flow estimation with 3D Lidar and RGB camera

Usami H., Saito H., Kawai J., Itani N.

IS and T International Symposium on Electronic Imaging Science and Technology （IS and T International Symposium on Electronic Imaging Science and Technology） 1 - 6 2018年01月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

© 2018 Society for Imaging Science and Technology. We present a method for synchronizing three-dimensional (3D) point cloud from 3D scene with estimation using a 3D Lidar and an RGB camera. These 3D points sensed by the 3D Lidar are not captured at the same time, which makes it difficult to measure the correct shape of the object in a dynamic scene. In our method, we generate synchronized 3D points at arbitrary times using linear interpolation in four-dimensional space, time and space. For interpolating the 3D point, we obtain corresponding 3D point matching with the pixel value captured by the RGB camera in a continuous frame. The experimental results demonstrate the effectiveness of the presented method by depicting a synchronized 3D point cloud that is correctly shaped.

Access to Document (DOI)

Vehicle pose estimation from drive recorder images by monocular SLAM and matching with rendered 3D point cloud of surrounding environment

Kurobe A., Kinoshita H., Saito H.

IS and T International Symposium on Electronic Imaging Science and Technology （IS and T International Symposium on Electronic Imaging Science and Technology） 2371 - 2376 2018年

© 2018, Society for Imaging Science and Technology. Vehicle pose estimation is a vital technology for reconstructing the circumstaces of traffic accidents. We propose a novel method for reconstructing the trajectory of vehicles from drive recorder images and a point cloud around the road. First, we apply ORB-SLAM to image sequence of the drive recorder for obtaining the vehicle pose trajectory; however this is based on relative coordinates and a relative scale. For estimating the absolute coordinates and scale of the trajectory, which cannot be obtained from a monocular SLAM like ORB-SLAM, we match the feature points detected in the image sequence with the three-dimensional (3D) point cloud of surrounding environment. For finding 3D points matching the feature points, we generate candidate images by the rendering 3D point cloud of the surrounding environment using the position initially estimated by the Global Positioning System (GPS). Next, we match to obtain the 3D two-dimensional (2D) generated images and drive recorder image to get 3D-2D point correspondences between the 3D point cloud and the drive recorder images; thus, we can convert the relative estimation of the camera pose by ORB-SLAM to the coordinates of the 3D point cloud of the surrounding environment. In the evaluation experiments, we confirmed the effectiveness of our method by comparing the vehicle poses estimated by our method, with those of RTKGPS, which exhibits high measurement precision.

Access to Document (DOI)

Synthesis of wide-FOV RGB-D images by registration and upsampling of 3D Lidar with omnidirectional RGB camera

Usami H., Miyata S., Saito H.

23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016 （23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016） 4 2340 - 2342 2018年

ISSN 9781510845510

Multi-view surface inspection using a rotating table

Kaichi T., Mori S., Saito H., Sugano J., Adachi H.

IS and T International Symposium on Electronic Imaging Science and Technology （IS and T International Symposium on Electronic Imaging Science and Technology） 2021 - 2026 2018年

© 2018, Society for Imaging Science and Technology. In this paper, we introduce a method to visually inspect in- dustrial parts by using multi-view images of an industrial part on a rotating table. During the visual inspection of the parts' sur- faces, the relationship between the camera pose, the light vectors, and the normal vectors of a part's surface is the key factor in de- tecting abnormalities. We can change the relationship between the three factors by rotating the part; then, the abnormalities are visible in certain positional relationships. We, therefore, track the points on a part's surfaces in an image sequence and discrimi- nate abnormal points (such as points on scratches or dents) from normal points based on pixel value transitions while rotating the part. The experimental results, which were based on real data of industrial parts, showed that the proposed method could detect the abnormalities on the surfaces.

Access to Document (DOI)

Constant velocity 3d convolution

Sekikawa Y., Ishikawa K., Saito H.

IEEE Access （IEEE Access） 6 76490 - 76501 2018年

© 2018 IEEE. We propose a novel 3-D convolution method, cv3dconv, for extracting spatiotemporal features from videos. It reduces the number of sum-of-products operations in 3-D convolution by thousands of times by assuming the constant moving velocity of the features. We observed that a specific class of video sequences, such as video captured by an in-vehicle camera, can be well approximated with piece-wise linear movements of 2-D features in a temporal dimension. Our principal finding is that a 3-D kernel, represented by constant velocity, can be decomposed into a convolution of a 2-D-shaped kernel and a 3-D-velocity kernel, which is parameterized using only two parameters. We derived an efficient recursive algorithm for this class of 3-D convolution, which is exceptionally suited for sparse spatiotemporal data, and this parameterized decomposed representation imposes a structured regularization along a temporal direction. We experimentally verified the validity of our approximation using a controlled dataset, and we also showed the effectiveness of the cv3dconv by adopting it for deep neural networks (DNNs) in visual odometry estimation task using publicly available event-based camera dataset captured in urban road scene. Our DNN architecture improves the estimation accuracy for about 30% compared with the existing states-of-the-arts architecture designed for event data.

Access to Document (DOI)

An overview of augmented visualization: Observing the real world as desired

Mori S., Saito H.

APSIPA Transactions on Signal and Information Processing （APSIPA Transactions on Signal and Information Processing） 7 2018年

Copyright © The Authors, 2018 This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.. Over 20 years have passed since a free-viewpoint video technology has been proposed with which a user's viewpoint can be freely set up in a reconstructed three-dimensional space of a target scene photographed by multi-view cameras. This technology allows us to capture and reproduce the real world as recorded. Once we capture the world in a digital form, we can modify it as augmented reality (i.e., placing virtual objects in the digitized real world). Unlike this concept, the augmented world allows us to see through real objects by synthesizing the backgrounds that cannot be observed in our raw perspective directly. The key idea is to generate the background image using multi-view cameras, observing the backgrounds at different positions and seamlessly overlaying the recovered image in our digitized perspective. In this paper, we review such desired view-generation techniques from the perspective of free-view point image generation and discuss challenges and open problems through a case study of our implementations.

Access to Document (DOI)

A simple 3D scanning system of human foot using a smartphone with a depth camera

Kobayashi T., Ienaga N., Sugiura Y., Saito H., Miyata N., Tada M.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering （Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering） 84 （ 12 ） 996 - 1002 2018年

ISSN 09120289

© 2018 Japan Society for Precision Engineering. All rights reserved. This paper presents a method to measure foot shape from a 3D point cloud as input captured from multiple directions using a smartphone depth camera. Such a 3D point cloud could potentially include noise or omit parts of the foot due to occlusion. To deal with this occlusion problem, we propose to use a dataset of 3D foot shapes collected by a precise 3D shape scanner of foot shapes. According to the dataset of 3D foot shapes, we can generate a deformable model by performing a principal component analysis (PCA) on the dataset. Then we minimize the error of the shape represented by the deformable model and the 3D point cloud acquired by the smartphone camera, to recover a complete 3D shape of the entire foot with high accuracy. We test this method by comparing the 3D shape produced by our proposed method to the 3D shape precisely measured by the 3D scanner. Our proposed method can scan the foot shape with an error of about 1,13mm.

Access to Document (DOI)

A refocus-interface for diminished reality work area visualization

Maezawa M., Mori S., Saito H.

IS and T International Symposium on Electronic Imaging Science and Technology （IS and T International Symposium on Electronic Imaging Science and Technology） 1121 - 1124 2018年

© 2018, Society for Imaging Science and Technology. In this paper, we present a refocus interface to set the parameters used for diminished reality (DR)-based work area visualization and a multiview camera-based rendering scheme. The refocus interface allows the user to determine two planes - one for setting a virtual window, through which the user can observe the background occluded by an object, and the other for a background plane, which is used for the subsequent background rendering. The background is rendered considering the geometric and appearance relationships of the multiview cameras observing the scene. Our preliminary results demonstrate that our DR system can visualize the hidden background.

Access to Document (DOI)

Synthesis of top view image and the detection of obstacles using multiple cameras for monitoring rea around a truck

Uehara K., Saito H., Yamamoto K., Sato H.

23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016 （23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016） 4 2304 - 2307 2018年

ISSN 9781510845510

Augmented visualization: Observing as desired

Mori S., Saito H.

Proceedings - 9th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2017 （Proceedings - 9th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2017） 2018-February 351 - 356 2017年12月

研究論文（国際会議プロシーディングス）, 査読有り, ISSN 9781538615423

© 2017 IEEE. Over 20 years have passed since a free-viewpoint video technology has been proposed in which a user's viewpoint can be freely set up reconstructed 3D space of a target scene photographed by multi-view cameras. This technology allows us to capture and reproduce the real world as it recorded. Once we capture the world in a digital form, we can modify the world as augmented reality does (i.e., placing virtual objects in the digitized real world). As oppose to this concept, the augmented world also allows us to see through real objects by synthesizing the backgrounds that cannot be observed in our raw perspective directly. The key idea is to generate the background image using multi-view cameras observing the backgrounds at different positions, and seamlessly overlaying the recovered image in our digitized perspective. In this article, our studies regarding such desired view generation techniques are reviewed, and its future directions and open problems are discussed.

Access to Document (DOI)

回転台を用いて撮像した多視点画像列による工業部品の外観検査

鶏内朋也, 森尚平, 斎藤英雄, 菅野純一, 足立秀之

精密工学会誌 83 （ 12 ） 1184 - 1191 2017年12月

研究論文（学術雑誌）, 査読有り

Visual inspection is a vital step for maintaining the quality of industrial parts. In visual inspection, the relation between the three factors, camera pose, lights direction and normal vector of a part is very important to detect abnormalities on industrial part. We propose a visual inspection method for industrial parts using an image sequence captured while rotating an industrial part using a rotating table. By rotating a part, we can easily change the relation between three factors. We track the points on part's surfaces in an image sequence. We discriminate abnormal surface points such as scratches from the normal ones based on pixel value transitions in an image sequence. For accurately tracking the points on the part surface, we propose a novel camera calibration method of the rotating table and the camera with a telecentric lens and a pose estimation method for the rotating part. We use angular invariant feature vectors for improving robustness against types and shapes of industrial parts to be inspected. We presented experimental results using real data of industrial parts and verified that the proposed method could detect scratches on surfaces.

Access to Document (DOI)

都市環境における線分特徴の方向性分布を利用したSLAM

上原啓, 斎藤英雄, 原　孝介

画像電子学会誌 46 （ 4 ） 522 - 532 2017年12月

研究論文（学術雑誌）, 共著, 査読有り

本論文では，線分特徴の事前分布を考慮した自動運転車のためのSLAM (Simultaneous Localization and Mapping) を提案する．提案手法は，道路標示や建物など，線分を検出しやすい物体が存在する都市環境を走行する車両に搭載されたカメラにより撮影される動画像を入力とすることを前提とし，この動画像から検出される線分特徴の方向に関する制約を加えることによりSLAM の精度を向上させるものである．線分特徴の方向に関する制約を加えるため，本手法では道路標示中の線分が車両進行方向に対して平行，もしくは垂直であるものが多いことに着目し，それら2 つの線分間の角度分布が混合ガウス分布に則ると仮定し，バンドル調整による最適化に導入する．これにより，カメラの位置姿勢と線分特徴の位置推定の精度を高めることが期待できる．本手法の有効性を確かめるため，カメラを1 つ用いたシングルカメラシステムと，2 つ用いたマルチカメラシステムの両方に提案手法を適用し精度検証を行った．その結果マルチカメラによる従来手法の精度と，シングルカメラによる本手法の精度が同等となった．また提案手法により生成した路面地図は，線分の方向に関する制約を導入したことにより補正され，精度向上を確認した．

Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras

Meerits S., Nozick V., Saito H.

Journal of Real-Time Image Processing （Journal of Real-Time Image Processing） 16 （ 6 ） 1 - 13 2017年11月

研究論文（学術雑誌）, ISSN 1861-8200

© 2017, The Author(s). We present a novel 3D reconstruction system that can generate a stable triangle mesh using data from multiple RGB-D sensors in real time for dynamic scenes. The first part of the system uses moving least squares (MLS) point set surfaces to smooth and filter point clouds acquired from RGB-D sensors. The second part of the system generates triangle meshes from point clouds. The whole pipeline is executed on the GPU and is tailored to scale linearly with the size of the input data. Our contributions include changes to the MLS method for improving meshing, a fast triangle mesh generation method and GPU implementations of all parts of the pipeline.

Access to Document (DOI)

Semantic Object Selection and Detection for Diminished Reality Based on SLAM with Viewpoint Class

Nakajima Yoshikatsu, Mori Shohei, Saito Hideo

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 338 - 343 2017年10月

研究論文（国際会議プロシーディングス）, 査読有り

We propose a novel diminished reality method which is able to (i) automatically recognize the region to be diminished, (ii) work with a single RGB-D sensor, and (iii) work without pre-processing to generate a 3D model of the target scene by utilizing SLAM, segmentation, and recognition framework. Especially, regarding the recognition of the area to be diminished, our method is able to maintain high accuracy no matter how the camera moves by distributing the viewpoints for each object uniformly and aggregating recognition results from each distributed viewpoint as the same weight. These advantages are demonstrated on the UW RGB-D Dataset and Scenes.

Access to Document (DOI)

An Instant See-Through Vision System Using a Wide Field-of-View Camera and a 3D-Lidar

Oishi Kei, Mori Shohei, Saito Hideo

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 344 - 347 2017年10月

研究論文（国際会議プロシーディングス）

Diminished reality (DR) enables us to see through real objects occluding some areas in our field of view. This interactive display has various applications, such as see-through vision to visualize invisible areas, work area visualization in surgery and landscape simulation. In this paper, we propose two underlying problems in see-through vision, in which hidden areas are observed in real time. First, see-through vision methods require a common area to calibrate every camera in the environment. However, the field of view is limited and many approaches rely on a time-consuming calibration, sensors, or fiducial markers. Second, see-through vision applications assume that the background is planar to ease image alignment. We therefore present a place-and-play see-through vision system using a wide field-of-view RGB-D camera. We validated the accuracy and the robustness of our system and showed results in various environments to show the applicability.

Access to Document (DOI)

Diminished Reality for Privacy Protection by Hiding Pedestrians in Motion Image Sequences Using Structure from Motion

Yagi Kentaro, Hasegawa Kunihiro, Saito Hideo

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 334 - 337 2017年10月

研究論文（国際会議プロシーディングス）, 査読有り

We present a method for generating images in which people are hidden from image sequences taken with a hand-held camera. Our method is basically used for privacy protection of people whose images are unintentionally captured in image sequences. We hide people from images by reconstructing a 3D model of background and projecting it to 2D images. By detecting the area in which people are present beforehand, we can reconstruct a 3D model of the background without people. In the experiment, We compare our method with some conventional approaches for diminished reality.

Access to Document (DOI)

魚眼力メラと3D-LIDARを用いた自由視点画像によるSee-Through Vision

大石圭, 森尚平, 斎藤英雄

映像情報メディア学会誌 71 （ 11 ） J276 - J279 2017年10月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1342-6907

現実世界から実物体を消し去る隠消現実感 (Diminished Reality; DR) 技術は，見通しの悪い交差点におけるSee-Through Visionなど，さまざま応用例が考えられている．しかし，そこにはいくつかの課題が存在した．第一に隠背景を観測するカメラと，体験者のカメラの直接的なカメラの位置関係を取得することが困難であること．第二に，屋外において隠背景が平面で近似できる場面でしか応用できなかったことである．本稿では，魚眼カメラと3D-LIDARを組み合わせた，高画角のRGB-Dカメラを用いて，両者のカメラの共通の視野を獲得することにより上記の問題解決を図る．また，屋外実験を通して，提案手法によるSee-Through Visionの実現可能性について議論する．

Access to Document (DOI)

組込み型カメラによる把持動作時の人体手形状3D復元

柏木直諒, 杉浦裕太, 宮田なつき, 多田充徳, 杉本麻樹, 斎藤英雄

映像情報メディア学会誌 71 （ 11 ） J283 - J286 2017年10月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1342-6907

本研究では，物体に対する人間の把持姿勢を3D復元するシステムを提案する．ペットボトルやビンといった筒状の物体の蓋の上面に魚眼レンズを装着したカメラを付与し，肌色抽出やセグメンテーションといった画像処理を加えて指を認識する．取得した指位置をディジタルハンドモデルと組合せることで手全体の位置姿勢を復元する．本システムは，日用品の利用状況の計測に基づく製品改良や，スポーツのトレーニングに活用できる．

Access to Document (DOI)

Registration of RGB and thermal point clouds generated by structure from motion

Trong Phuc Truong, Masahiro Yamaguchi, Shohei Mori, Vincent Nozick, and Hideo Saito

2017 IEEE International Conference on Computer Vision Workshops (ICCVW) 419 - 427 2017年10月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Thermal imaging has become a valuable tool in various
fields for remote sensing and can provide relevant information
to perform object recognition or classification.
In this paper, we present an automated method to obtain a
3D model fusing data from a visible and a thermal camera.
The RGB and thermal point clouds are generated independently
by structure from motion. The registration process
includes a normalization of the point cloud scale, a global
registration based on calibration data and the output of the
structure from motion, and a fine registration employing a
variant of the Iterative Closest Point optimization. Experimental
results demonstrate the accuracy and robustness of
the overall process.

Access to Document (DOI)

A Work Area Visualization by Multi-View Camera-Based Diminished Reality

Shohei Mori, Momoko Maezawa and Hideo Saito

Multimodal Technologies Interaction 1 （ 3 ） 2017年09月

研究論文（学術雑誌）, 共著

Hand-held tools are indispensable for efficient manual working in fields ranging from
woodworking to surgery. In this paper, we present a diminished reality (DR) method to visualize areas
occluded by hands and tools in various hand-working scenarios. We propose a redesigned existing
arbitrary viewpoint image generation method for DR applications as a core DR background rendering
to recover views without undesirable objects. We conducted quantitative and qualitative experiments
using real data to validate the performance of the proposed method. The experimental results showed
that our method runs in real-time (40.1 fps) and surpasses conventional methods (including image
inpainting-based and geometry-based approaches) in terms of image similarity measures.

Access to Document (DOI)

Structure Based Matching between Aerial and Map Images using Brightness- and Rotation-Invariant Curve Features

Yoshikatsu Nakajima, Hideo Saito

Proceedings of the Irish Machine Vision and Image Processing Conference (IMVIP2017) 233 - 236 2017年08月

研究論文（国際会議プロシーディングス）, 査読有り

Since conventional feature descriptor algorithms depend on brightness values, those algorithms cannot
obtain point correspondences between aerial and map images where brightness values do not correspond but
geometric structures do correspond. The research in this paper focused on obtaining point correspondences
between those images. This study proposes a novel method by which curve features, edge shape, and
road extraction by a CNN, independent on brightness value, produce robustness in image matching. Its
effectiveness was confirmed through experiments on Massachusetts Roads Dataset.

輝度値変化及び回転に頑健な曲線特徴量

中島由勝, 斎藤英雄

電子情報通信学会論文誌D J100-D （ 12 ） 978 - ９８１ 2017年08月

研究論文（学術雑誌）, 査読有り

Access to Document (DOI)

Ball 3D Trajectory Reconstruction without Preliminary Temporal and Geometrical Camera Calibration

Miyata Shogo, Saito Hideo, Takahashi Kosuke, Mikami Dan, Isogawa Mariko, Kimata Hideaki

2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2017-July 164 - 169 2017年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

This paper proposes a method for reconstructing 3D ball trajectories by using multiple temporally and geometrically uncalibrated cameras. To use cameras to measure the trajectory of a fast-moving object, such as a ball thrown by a pitcher, the cameras must be temporally synchronized and their position and orientation should be calibrated. In some cases, these conditions cannot be met, e.g., one cannot geometrically calibrate cameras when one cannot step into a baseball stadium. The basic idea of the proposed method is to use a ball captured by multiple cameras as a corresponding point. The method first detects a ball. Then, it estimates temporal difference between cameras. After that, the ball positions are used as corresponding points for geometrically calibrating the cameras. Experiments using actual pitching videos verify the effectiveness of our method.

Access to Document (DOI)

Extrinsic Camera Calibration Without Visible Corresponding Points Using Omnidirectional Cameras

Miyata S., Saito H., Takahashi K., Mikami D., Isogawa M., Kojima A.

IEEE Transactions on Circuits and Systems for Video Technology （IEEE Transactions on Circuits and Systems for Video Technology） 28 （ 9 ） 2210 - 2219 2017年07月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1051-8215

© 1991-2012 IEEE. This paper proposes a novel algorithm that calibrates multiple cameras scattered across a broad area. The key idea of the proposed method is 'using the position of an omnidirectional camera as a reference point.' The common approach to calibrating multiple cameras assumes that the cameras capture at least some common points. This means calibration becomes quite difficult if there are no shared points in each camera's field of view (FOV). The proposed method uses the position of an omnidirectional camera to determine point correspondence. The position of an omnidirectional camera relative to the calibrated camera is estimated by the theory of epipolar geometry, even if the omnidirectional camera is placed outside the camera's FOV. This property makes our method applicable to multiple cameras scattered across a broad area. Qualitative and quantitative evaluations using synthesized and real data, e.g., a sports field, demonstrate the advantages of the proposed method.

Access to Document (DOI)

カメラ位置姿勢推定のための3次元形状の冗長な線分除去

家永直人, Siim Meerits, 森尚平, 斎藤英雄

映像情報メディア学会誌 71 （ 6 ） J189 - J191 2017年06月

研究論文（学術雑誌）, 共著, 査読有り

Access to Document (DOI)

A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects

Shohei Mori, Sei Ikeda and Hideo Saito

IPSJ Transactions on Computer Vision and Applications （Springer） 9 （ 17 ） 1 - 14 2017年06月

研究論文（学術雑誌）

In this paper, we review diminished reality (DR) studies that visually remove, hide, and see through real objects from
the real world. We systematically analyze and classify publications and present a technology map as a reference for
future research. We also discuss future directions, including multimodal diminished reality. We believe that this paper
will be useful mainly for students who are interested in DR, beginning DR researchers, and teachers who introduce DR
in their classes.

Access to Document (DOI)

Diminished hand: A Diminished Reality-Based Work Area Visualization

Mori Shohei, Maezawa Momoko, Ienaga Naoto, Saito Hideo

2017 IEEE Virtual Reality (VR) 443 - 444 2017年04月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Live instructors perspective videos are useful to present intuitive visual instructions for trainees in medical and industrial settings. In such videos, the instructors hands often hide the work area. In this demo, we present a diminished hand for visualizing the work area hidden by hands by capturing the work area with multiple cameras. To achieve the diminished reality, we use a light field rendering technique, in which light rays avoid passing through penalty points set in the unstructured light fields reconstructed from the multiple viewpoint images.

Access to Document (DOI)

Speaker Identification Based on Integrated Face Direction in a Group Conversation

Naoto Ienaga, Yuko Ozasa, and Hideo Saito

2017 IEEE Winter Applications of Computer Vision Workshops (WACVW) （IEEE） 53 - 57 2017年03月

研究論文（国際会議プロシーディングス）, 査読有り

Access to Document (DOI)

Measuring grasp posture using an embedded camera

Naoaki Kashiwagi, Yuta Sugiura, Natsuki Miyata, Mitsunori Tada, Maki Sugimoto, and Hideo Saito

2017 IEEE Winter Applications of Computer Vision Workshops (WACVW) 42 - 47 2017年03月

研究論文（国際会議プロシーディングス）, 共著

Access to Document (DOI)

Line-based SLAM Considering Directional Distribution of Line Features in an Urban Environment

Kei Uehara, Hideo Saito and Kosuke Hara

Proceedings of the 12th International Conference on Computer Vision Theory and Applications (VISAPP2017) 6 255 - 264 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

In this paper, we propose a line-based SLAM from an image sequence captured by a vehicle in consideration
with the directional distribution of line features that detected in an urban environments. The proposed SLAM
is based on line segments detected from objects in an urban environment, for example, road markings and
buildings, that are too conspicuous to be detected. We use additional constraints regarding the line segments
so that we can improve the accuracy of the SLAM. We assume that the angle of the vector of the line segments
to the vehicle’s direction of travel conform to four-component Gaussian mixture distribution. We define a
new cost function considering the distribution and optimize the relative camera pose, position, and the 3D
line segments by bundle adjustment. In addition, we make digital maps from the detected line segments. Our
method increases the accuracy of localization and revises tilted lines in the digital maps. We implement our
method to both the single-camera system and the multi-camera system. The accuracy of SLAM, which uses
a single-camera system with our constraint, works just as well as a method that uses a multi-camera system
without our constraint.

Primitive Shape Recognition via Superquadric Representation using Large Margin Nearest Neighbor Classifier

Ryo Hachiuma, Yuko Ozasa and Hideo Saito

Proceedings of the 12th International Conference on Computer Vision Theory and Applications (VISAPP2017) 5 325 - 332 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

It is known that humans recognize objects using combinations and positional relations of primitive shapes.
The first step of such recognition is to recognize 3D primitive shapes. In this paper, we propose a method for
primitive shape recognition using superquadric parameters with a metric learning method, large margin nearest
neighbor (LMNN). Superquadrics can represent various types of primitive shapes using a single equation with
few parameters. These parameters are used as the feature vector of classification. The real objects of primitive
shapes are used in our experiment, and the results show the effectiveness of using LMNN for recognition based
on superquadrics. Compared to the previous methods, which used k-nearest neighbors (76.5%) and Support
Vector Machines (73.5%), our LMNN method has the best performance (79.5%).

Application of LSD-SLAM for Visualization Temperature in Wide-area Environment

Masahiro Yamaguchi, Hideo Saito and Shoji Yachida

Proceedings of the 12th International Conference on Computer Vision Theory and Applications (VISAPP2017) 4 216 - 223 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

In this paper, we propose a method to generate a three-dimensional (3D) thermal map by overlaying thermal
images onto a 3D surface reconstructed by a monocular RGB camera. In this method, we capture the target
scene moving both an RGB camera and a thermal camera, which are mounted on the same zig. From the
RGB image sequence, we reconstruct 3D structures of the scene by using Large-Scale Direct Monocular
Simultaneous Localization and Mapping (LSD-SLAM), on which temperature distribution captured by the
thermal camera is overlaid, thus generate a 3D thermal map. The geometrical relationship between those
cameras is calibrated beforehand by using a calibration board that can be detected by both cameras. Since we
do not use depth cameras such as Kinect, the depth of the target scene is not limited by the measurement range
of the depth camera; any depth range can be captured. To demonstrating this technique, we show synthesized
3D thermal maps for both indoor and outdoor scenes.

Eating and Drinking Recognition via Integrated Information of Head Directions and Joint Positions in a Group

Naoto Ienaga, Yuko Ozasa and Hideo Saito

Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods (ICPRAM 2017) （Science and Technology Publications） 527 - 533 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Recent years have seen the introduction of service robots as waiters or waitresses in restaurants and cafes.
In such venues, it is common for customers to visit in groups as well as for them to engage in conversation
while eating and drinking. It is important for cyber serving staff to understand whether they are eating and
drinking, or not, in order to wait on tables at appropriate times. In this paper, we present a method by which
the robots can recognize eating and drinking actions performed by individuals in a group. Our approach uses
the positions of joints in the human body as a feature and long short-term memory to achieve a recognition
task on time-series data. We also used head directions in our method, as we assumed that it is effective for
recognition in a group. The information garnered from head directions and joint positions is integrated via
logistic regression and employed in recognition. The results show that this yielded the highest accuracy and
effectiveness of the robots’ tasks.

First Deployment of Diminished Reality for Anatomy Education

Ienaga Naoto, Bork Felix, Meerits Siim, Mori Shohei, Fallavollita Pascal, Navab Nassir, Saito Hideo

The second International Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR2016), 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) （IEEE） 294 - 296 2017年01月

研究論文（国際会議プロシーディングス）, 査読有り

Understanding the anatomy of the human body is vital for everyone working in the medical domain. Augmented reality (AR) systems for anatomy teaching, which display virtual information directly on top of a users' body, have proven to facilitate mental mapping compared to traditional teaching paradigms. In this paper, we explore the potential of diminished reality (DR) in the context of anatomy education. As a first necessary step to achieving a DR anatomy education system, parts of the human body have to be extracted and diminished from the video stream. Our system diminishes either the arm or head of the user by projecting a background image recovered using RGB-D cameras. Such a system, if combined with an accurate overlay of virtual counterparts, could potentially improve the learning effect by attracting the users' attention to the virtual information and improve visual perception by avoiding the well-known floating effect of AR.

Access to Document (DOI)

Detour Light Field Rendering for Diminished Reality Using Unstructured Multiple Views

Mori Shohei, Maezawa Momoko, Ienaga Naoto, Saito Hideo

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Instructor's perspective videos are useful for presenting intuitive visual instructions for trainees in medical and industrial settings. In such videos, the instructor's arms often obstruct the trainee's view of the work area. In this article, we present a diminished reality method for visualizing the work area hidden by an instructor's arms by capturing the work area with multiple cameras. To achieve such diminished reality, we propose detour light field rendering (DLFR), in which light rays avoid passing through penalty points set in the unstructured light fields reconstructed from multiple viewpoint images. In DLFR, the camera blending field used in an existing freeviewpoint image generation method known as unstructured lumigraph is re-designed based on our use cases. In this re-design, lesser weights are given to light rays as they pass close to given penalty points. Experimental results demonstrate that using DLFR, the appearance of an undesirable object can be removed from an image in real time.

Access to Document (DOI)

Robust camera pose estimation by viewpoint classification using deep learning

Yoshikatsu Nakajima and Hideo Saito

Computational Visual Media （Springer） 3 （ 2 ） 189 - 198 2016年12月

研究論文（学術雑誌）, 共著, 査読有り

Camera pose estimation with respect to target scenes is an important technology for superimposing virtual information in augmented reality (AR). However, it is difficult to estimate the camera pose for all possible view angles because feature descriptors such as SIFT are not completely invariant from every perspective. We propose a novel method of robust camera pose estimation using multiple feature descriptor databases generated for each partitioned viewpoint, in which the feature descriptor of each keypoint is almost invariant. Our method estimates the viewpoint class for each input image using deep learning based on a set of training images prepared for each viewpoint class. We give two ways to prepare these images for deep learning and generating databases. In the first method, images are generated using a projection matrix to ensure robust learning in a range of environments with changing backgrounds. The second method uses real images to learn a given environment around a planar pattern. Our evaluation results confirm that our approach increases the number of correct matches and the accuracy of camera pose estimation compared to the conventional method.

Access to Document (DOI)

Reconstruction of 3D models consisting of line segments

Ienaga Naoto, Saito Hideo

ACCV 2016 International Workshops, Revised Selected Papers, Part II, Lecture Notes in Computer Science book series (LNCS, volume 10117) （Springer International Publishing） 100 - 113 2016年11月

研究論文（国際会議プロシーディングス）, 査読有り

Reconstruction of three-dimensional (3D) models from images is currently one of the most important areas of research in computer vision. In this paper, we propose a method to recover 3D models using the minimum number of line segments. By using structure-from-motion, the proposed method first recovers a 3D model of line segments detected from an input image sequence. We then detect overlapping 3D line segments that redundantly represent a single line structure so that the number of 3D line segments representing the target scene can be reduced without losing the detailed geometry of the structure. We apply matching and depth information to remove redundant line segments from the model while keeping the necessary segments. In experiments, we confirm that the proposed method can greatly reduce the number of line segments. We also demonstrate that the accuracy and computational time for camera pose estimation can be significantly improved with the 3D line segment model recovered by the proposed method. Moreover, we have applied the proposed method to see through occluded areas.

Access to Document (DOI)

Synthesis of a stroboscopic image from a hand-held camera sequence for a sports analysis

Kunihiro Hasegawa and Hideo Saito

Computational Visual Media （Springer） 2 （ 3 ） 277 - 289 2016年09月

研究論文（学術雑誌）, 共著, 査読有り

This paper presents a method for synthesizing a stroboscopic image of a moving sports player from a hand-held camera sequence. This method has three steps: synthesis of background image, synthesis of stroboscopic image, and removal of player’s shadow. In synthesis of background image step, all input frames masked a bounding box of the player are stitched together to generate a background image. The player is extracted by an HOG-based people detector. In synthesis of stroboscopic image step, the background image, the input frame, and a mask of the player synthesize a stroboscopic image. In removal of shadow step, we remove the player’s shadow which negatively affects an analysis by using mean-shift. In our previous work, synthesis of background image has been timeconsuming. In this paper, by using the bounding box of the player detected by HOG and by subtracting the images for synthesizing a mask, computational speed and accuracy can be improved. These have contributed greatly to the improvement from the previous method. These are main improvements and novelty points from our previous method. In experiments, we confirmed the effectiveness of the proposed method, measured the player’s speed and stride length, and made a footprint image. The image sequence was captured under a simple condition that no other people were in the background and the person controlling the video camera was standing still, such like a motion parallax was not occurred. In addition, we applied the synthesis method to various scenes to confirm its versatility.

Access to Document (DOI)

Recognition and pose estimation of primitive shapes from depth images for spatial augmented reality

Ryo Hachiuma and Hideo Saito

Proceedings of IEEE 2nd Workshop on Everyday Virtual Reality (WEVR2016) 2016年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Access to Document (DOI)

Marker-less augmented reality framework using on-site 3D line-segment-basedmodel generation

Nakayama Yusuke, Saito Hideo, Shimizu Masayoshi, Yamaguchi Nobuyasu

Journal of Imaging Science and Technology 60 （ 2 ） 20401-1 - 20401-24 2016年03月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1062-3701

The authors propose a line-segment-based marker-less augmented reality (AR) framework that involves an on-site model-generation method and on-line camera tracking. In most conventional model-based marker-less AR frameworks, correspondences between the 3D model and the 2D frame for camera-pose estimation are obtained by feature-point matching. However, 3D models of the target scene are not always available, and feature points are not detected from texture-less objects. The authors' framework is based on a model-generation method with an RGB-D camera and model-based tracking using line segments, which can be detected even with only a few feature points. The camera pose of the input images can be estimated from the 2D-3D line-segment correspondences given by a line-segment feature descriptor. The experimental results show that the proposed framework can achieve AR when other point-based frameworks cannot. The authors also argue that their framework can generate a model and estimate camera pose more accurately than their previous study.

Access to Document (DOI)

Chromaticity-matched Superimposition of Foreground Objects in Different Environments

Yohei Ogura, Yoichi Nakasone, Shao Ming and Hideo Saito

Proceedings of the 22nd Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV2016) 141 - 145 2016年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

The illumination chromaticity is one of the important
factor for appearance of objects in images. When we
superimpose some objects on another scene to generate composite
images, the resulting image may give us strangeness if the
illumination chromaticity of foreground scene and background
scene are different. In this paper, we propose a composite
image synthesis method which can achieve illumination chromaticity
consistency. Our goal is to generate a illumination
chromaticity-matched composite image from two images which
are taken under different illumination environment. We estimate
chromaticity of two scenes based on segmented color regions.
Subsequently illumination chromaticity matching is performed
based on estimated chromaticity of two scenes. After the chromaticity
matching process, we extract the target person by using
the grab cut segmentation. The result image is obtained by
superimposing the target person or object onto background
scene. We demonstrate that our method can perform some
different illumination chromaticity environment.

Camera Pose Estimation Based on Keypoints Matching with Pre-Captured Set of Real Images

Kei Obata and Hideo Saito

Proceedings of the 22nd Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV2016) 76 - 80 2016年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

In this paper, we propose a method for camera pose
estimation based on keypoints matching with images capturing
the target scene from multiple viewpoints. For estimating camera
pose from an image, 3D structure of the target scene is needed.
Then a set of correspondences between 2D position in the image
and 3D positions in the target scene provides the camera pose by
solving Perspective-n-Point problem. Image keypoints are used
as these positions, but finding corresponding 3D positions of the
3D structure is not easy problem even if a 3D textured model
of the target scene is given. This is because the appearance of
each 3D position is not always the same. In order to solve this
problem, we adopt real images as a source of keypoints contained
in the database. Real image represents accurate appearance for
each 3D position, so keypoint matching with the input images
can be more accurate. We constructed database based on the
keypoint using the local image feature derived from real images
and evaluated the accuracy of camera pose estimation. The result
showed that the database constructed by our method is effective
to accurate camera tracking.

Stroboscopic Image Synthesis of Sports Player from Hand-Held Camera Sequence

Hasegawa Kunihiro, Saito Hideo

Proceedings of 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) 726 - 733 2016年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

A method for synthesizing the stroboscopic image of a moving sports player from a hand-held camera sequence is proposed. In this method, a player is extracted from each frame of an input video sequence using a HOG-based people detector. After removing the bounding box of the extracted player from each frame, all frames are stitched together to generate a background image without the player. The player's area is then overlaid onto the stitched background image and a stroboscopic image of the player is synthesized. By using the bounding box of the player detected by HOG and by subtracting the images, computational speed and accuracy can be improved compared with the conventional method that segments the player's region. In addition to the stroboscopic image synthesis, we also remove the player's shadow to improve the appearance of the resultant stroboscopic image. Experimental results demonstrate the effectiveness of the proposed method.

Access to Document (DOI)

勾配画像処理に基づく三次元点群からの高精度地図の自動生成

原孝介, 斎藤英雄

自動車技術会論文集 47 （ 1 ） 183 - 188 2016年01月

研究論文（学術雑誌）, 共著, 査読有り

Relighting for an arbitrary shape object under unknown illumination environment

Ogura Yohei, Saito Hideo

Proceedings of 11th International Symposium, ISVC 2015,Lecture Notes in Computer Science Volume 9475 （Springer） 433 - 442 2015年12月

研究論文（国際会議プロシーディングス）, 共著

Relighting techniques can achieve the photometric consistency in synthesizing a composite images. Relighting generally need the object’s shape and illumination environment. Recent research demonstrates possibilities of relighting for an unknown shape object or the relighting the object under unknown illumination environment. However, achieving both tasks are still challenging issue. In this paper, we propose a relighting method for an unknown shape object captured under unknown illumination environment by using an RGB-D camera. The relighted object can be rendered from pixel intensity, surface albedo and shape of the object and illumination environment. The pixel intensity and the shape of the object can simultaneously be obtained from an RGB-D camera. Then surface albedo and illumination environment are iteratively estimated from the pixel intensity and shape of the object.We demonstrate that our method can perform relighting for a dynamic shape object captured under unknown illumination using an RGB-D camera.

Access to Document (DOI)

アルベドと法線ベクトルの緊密度に着目した影と鏡面反射除去のための組み合わせ照度差ステレオ法

家永　直人, 斎藤英雄, 清水雅芳，手塚耕一，岩村康正

精密工学会誌（公益社団法人精密工学会） 81 （ 12 ） 1154 - 1161 2015年12月

研究論文（学術雑誌）, 共著, 査読有り

We propose a method of ego-motion estimation for a self-driving car on which we installed cameras with non-overlapping views. By finding corresponding points between the multi-camera images, we aim to enhance the accuracy of the ego-motion estimation. However since the viewing directions are very different from one camera to the other, a conventional algorithm such as SURF cannot detect a sufficient number of correspondences. Additionally in the case where cameras have low frame rate and the vehicle has high speed, the scene changes might be too big to find correspondences between the same camera images. We propose a novel matching algorithm by warping feature patches detected in different cameras based on urban 3D structure. We assume that detected features exist on the surface of buildings or roads and the patch around the feature is planar. Based on this assumption, we can warp the patches so that the feature descriptors are similar for the corresponding feature points. We apply Bundle Adjustment to the found correspondences to optimize the odometry. The result shows higher estimation accuracy when compared to other matching methods.

Access to Document (DOI)

都市の3次元構造を利用した非重複カメラ間の特徴点マッチングによる車両運動の推定

川崎敦史, 斎藤英雄, 原孝介

精密工学会誌（公益社団法人精密工学会） 81 （ 12 ） 1146 - 1153 2015年12月

研究論文（学術雑誌）, 共著, 査読有り

Access to Document (DOI)

Vehicle localization based on the detection of line segments from multi-camera images

Hara Kosuke, Saito Hideo

Journal of Robotics and Mechatronics 27 （ 6 ） 617 - 626 2015年12月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 0915-3942

For realizing autonomous vehicle driving and advanced safety systems, it is necessary to achieve accurate vehicle localization in cities. This paper proposes a method of accurately estimating vehicle position by matching a map and line segment features detected from images captured by a camera. Features such as white road lines, yellow road lines, road signs, and curb stones, which could be used as clues for vehicle localization, were expressed as line segment features on a two-dimensional road plane in an integrated manner. The detected line segments were subjected to bird’s-eye view transformation to transform them to the vehicle coordinate system so that they could be used for vehicle localization regardless of the camera configuration. Moreover, an extended Kalman filter was applied after a detailed study of the line observation errors for realizing real-time estimation. Vehicle localization was tested under city driving conditions, and the vehicle position was identified with sub-meter accuracy.

Access to Document (DOI)

Geolocation for printed maps using line segment-based SIFT-like feature matching

Minster Gautier, Moreau Guillaume, Saito Hideo

Proceedings of 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, Challenges and Applications of Urban Augmented Reality 88 - 93 2015年10月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

This paper presents a method for the geolocation of printed maps. It enables the registration of unprepared maps with a Geographical Information System (GIS) database, and can for example be used as a first step to augment an unknown map. We define and match local road pattern descriptors, which are similar to SIFT descriptors [6], but adapted to the case of simple textureless line segments. Using a processing pipeline commonly encountered in the feature-point based matching of texture images - composed of offline description and indexing, followed by an online description, matching and robust transformation estimation - we show that local descriptors can successfully register unprepared maps using only geographic features and no texture information. Our method is scale and rotation invariant, and circumvents the two hurdles that are the level-of-detail, and the changing colormaps and textures, allowing the processing of large classes of printed maps.

Access to Document (DOI)

Diminished reality for hiding a pedestrian using hand-held camera

Hasegawa Kunihiro, Saito Hideo

Proceedings of 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, International Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR2015) 47 - 52 2015年09月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

This paper proposes a diminished reality method for hiding a pedestrian from a hand-held camera sequence. In the proposed method, the pedestrian is extracted from each frame of input video sequence using HOG based people detector. The detected area is excluded from each frame by masking, and then two frames are stitched for generating a stitched background image. By using the stitched background image, the pedestrian's mask for hiding is made. It is used for overlaying the background pixel to the pedestrian in the original frame. The presented experimental result is demonstrate the effectiveness of the proposed method.

Access to Document (DOI)

Motion estimation for non-overlapping cameras by improvement of feature points matching based on urban 3D structure

Kawasaki Atsushi, Saito Hideo, Hara Kosuke

Proceedings of 2015 IEEE International Conference on Image Processing (ICIP) 1230 - 1234 2015年09月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

We propose a method of ego-motion estimation for a self-driving vehicle using multiple cameras. By finding corresponding points between the multi-camera images, we aim to enhance the accuracy of the ego-motion estimation. However since the viewing directions are very different from one camera to the other, a conventional algorithm such as SURF cannot detect a sufficient number of correspondences. We propose a novel matching algorithm by warping feature patches detected in different cameras based on urban 3D structure. We assume that detected features exist on the surface of buildings or roads and the patch around the feature is planar. Based on this assumption, we can warp the patches so that the feature descriptors are similar for the corresponding feature points. We apply Bundle Adjustment to the found correspondences to optimizes the odometry. The result shows higher estimation accuracy when compared to other matching method.

Access to Document (DOI)

Real-time diminished reality for dynamic scenes

Meerits Siim, Saito Hideo

研究論文（国際会議プロシーディングス）, 査読有り

Diminished reality (DR) research aims to develop methods to visually hide objects from real scenes. One or more cameras capture a scene where some objects, called the occluding objects, are 'diminished' by an DR system. To do this in real time we need to use multiple cameras. We introduce a novel multi-camera DR system utilizing an RGB-D camera to hide arbitrary trackable objects from a scene. In our case, the scene background does not have to be planar for the system to work and we can handle scene changes in real time. Additonally, we introduce color correction technique to merge image content originating from different cameras and also demonstrate method to repair images with partially missing content.

Access to Document (DOI)

Combination photometric stereo using compactness of albedo and surface normal in the presence of shadows and specular reflection

Ienaga Naoto, Saito Hideo, Tezuka Kouichi, Iwamura Yasumasa, Shimizu Masayoshi

Proceedings of the 16th International Conference on Computer Analysis of Images and Patterns, CAIP 2015, LNCS 9256 （Springer） 445 - 455 2015年09月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

We present a novel combination photometric stereo which can estimate surface normals precisely even for images including shadows and specular reflection. We can use photometric stereo if there are more than three input images. Therefore we can employ photometric stereo with nC3 combinations for n input images. We make 3D distribution of albedos and surface normals estimated from pixel intensities of nC3 pixel combinations. In the distribution, we define a novel value “compactness” to distinguish pixels which are included in neither shadows nor specular reflection from pixels which are included in shadows or specular reflection. Through experimental results, we demonstrate that the proposed method can estimate surface normals in the presence of shadows and specular reflection. Moreover the proposed method is superior to previous works in better accuracy.

Access to Document (DOI)

RGB-Dスーパーピクセル領域分割された局所領域内の平面性を利用した長距離デプス画像の高解像度化

松本一紀，斎藤英雄

電子情報通信学会論文誌 D J98-D （ 9 ） 1226 - 1229 2015年09月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1881-0225

Reconstructing the 3D Trajectory of a Ball with Unsynchronized Cameras

Tamaki Sho, Saito Hideo

International journal of computer science and sport 14 （ 1 ） 51 - 68 2015年09月

研究論文（学術雑誌）, 査読有り, ISSN 1684-4769

The 3D trajectory of the ball is one of the most important performance indicators
in ball games. However, it often cannot be measured because of the requirements:
synchronized cameras and control points within the playing field. Thus, this paper
proposes a method that reconstructs the 3D trajectory of a ball with
unsynchronized cameras. The proposed method consists of ball detection, camera
calibration, and trajectory reconstruction. At first, ball candidates are detected on
the basis of their appearance. Balls are then extracted from the candidates on the
basis of their motion. After the extraction, ball trajectories are reconstructed in
image spaces. The corresponding points in two views are estimated on the basis
of the ball trajectory in image spaces and the temporal offset between cameras,
which is supposed to be recorded with a few erroneous frames. The matrix
including the geometrical relationship between cameras, namely the fundamental
matrix, is then computed from the corresponding points. The estimated one is
inaccurate due to the error of the temporal offset. The key feature of this method
is to optimize the temporal offset and fundamental matrix simultaneously. After
the optimization, the geometrical relationship between cameras is computed from
the matrix. Balls are extracted again by using the temporal offsets and the
geometrical relationship between cameras. A ball trajectory is finally
reconstructed as connected trajectories that are separated at collisions. It is
experimentally demonstrated that the proposed method accurately calibrates
cameras and successfully reconstructs the 3D trajectory of the ball.

Visualization of Dynamic Hidden Areas by Real-Time 3D Structure Acquisition Using RGB-D Camera

Siim Meerits and Hideo Saito

Proceedings of International Display Manufacturing Conference (IDMC) & 3D Systems and Applications (3DSA) 2015 AP4-008 2015年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

An omnidirectional vision system for bus safety surveillance

Hung Dao Huu, Saito Hideo, Yamamoto Keiichi, Sato Hiromitsu

Proceedings of 3rd Workshop on Activity Monitoring by Multiple Distributed Sensing (AMMDS), 12th edition of the IEEE AVSS conference 2015年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Sudden brake seems to be unavoidable in driving practice to prevent from accidence. The sudden brake may often cause falling risks to on-board passengers in public transport and even threaten the lives of the senior. Thus, this paper presents an omnidirectional vision system to assist bus drivers in ensuring the safety for passengers. To this end, we configure an auxiliary camera in the boarding gate to recognize the age of passengers and an omnidirectional camera on the ceiling of the bus to detect and track the passengers. Their trajectories and positions are mapped into a bus layout, together with their age information transferred from the auxiliary camera. This bus layout is shown on a small display in front of bus drivers. Instead of looking to the mirror to guess and gather information of passengers, the driver can collect such necessary information for safe driving. We set up experiments on a real bus and demonstrate the system by preliminary results.

Access to Document (DOI)

Mutliple Object Tracking for End-to-End Transmission System of Free-Viewpoint Video

Shogo Miyata, Hideo Saito

Proceedings of International Display Manufacturing Conference (IDMC) & 3D Systems and Applications (3DSA) 2015 3DSA S5-5 2015年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Polymorphic cataloguing and viewing system for using digital archives

Ishikawa Hiroyo, Saito Hideo, Miyashita Yamato, Kaneko Kunitake

Proceedings of 2014 International Conference on Virtual Systems & Multimedia (VSMM) （IEEE） 176 - 179 2015年06月

研究論文（国際会議プロシーディングス）, 共著

We propose a digital content viewing system which visualizes the polymorphic relationships among digital content interactively and uses it as a visual interface. The relationships are expressed by associating of two content and grouping and shown by directed graphs. On the graphs, nodes mean the content files or the groups, and edges mean correspondences or group members. The graphs are archived as relation catalogues separately from content files. We visualize the relations of content on a directed graph. To visualize the graphs, we use a method of locating nodes by horizontal directions and height. Nodes linked with a target node will be located on a horizontal circle and their height is decided by their relation catalogue. After deciding the node location, nodes and links are drawn using 3D graphics and animation. The visualization result called 'polymorphic view' is used as a visual interface for selecting content. Selecting content on the visualized graph will expand new links and new content nodes additionally. We can show another content by tracing graph links. It becomes easy to understand the relations between digital content, because we can change the direction and size of the directed graph. We developed a content viewer system (MoSaIC-II) using the polymorphic view as a visual interface and we can see the usability of the polymorphic view.

Access to Document (DOI)

3D Line Segment Based Model Generation by RGB-D Camera for Camera Pose Estimation

Yusuke Nakayama, Hideo Saito, Masayoshi Shimizu, Nobuyasu Yamaguchi

Computer Vision - ACCV 2014 Workshops Lecture Notes in Computer Science 9010 459 - 472 2015年04月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

In this paper, we propose a novel method for generating 3D line segment based model from an image sequence taken with a RGB-D camera. Constructing 3D geometrical representation by 3D model is essential for model based camera pose estimation that can be performed by corresponding 2D features in images with 3D features of the captured scene. While point features are mostly used as such features for conventional camera pose estimation, we aim to use line segment features for improving the performance of the camera pose estimation. In this method, using RGB images and depth images of two continuous frames, 2D line segments from the current frame and 3D line segments from the previous frame are corresponded. The 2D-3D line segment correspondences provide camera pose of the current frame. All of 2D line segments are finally back-projected to the world coordinate based on the estimated camera pose for generating 3D line segment based model of the target scene. In experiments, we confirmed that the proposed method can successfully generate line segment based models, while 3D models based on the point features often fail to successfully represent the target scene.

Access to Document (DOI)

ボールマークと軌跡に基づく卓球ボールの3次元回転計測

玉城　将，斎藤英雄，吉田　和人

電子情報通信学会論文誌 D J98-D （ 4 ） 630 - 650 2015年04月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1881-0225

卓球においてボールの回転は戦術的に重要な要素と考えられている．しかし，従来の回転計測法は，表面全体にマークを追加したボールを高速度カメラを用いて撮影するものであり，撮影されるデータ量が膨大になる，あるいは試合と同じ条件下の回転による攻防を分析できない等の問題がある．ボールの運動モデルに基づき軌跡から回転を推定する方法も提案されているが，正確な回転計測は難しい．そこで本研究では，高速度カメラを用いることを前提とせず，且つ試合で用いられる卓球ボールを対象とできる回転計測法を提案する．提案手法ではまずボールのマークを検出し，検出されたマーク領域の重心から回転軸を算出する．高速度カメラを用いないことから，回転軸周りの回転方向及び隣接フレーム間の回転角速度は一意には求まらないが，回転角速度の全ての候補値に対応する軌跡を運動モデルに基づきシミュレートし，実測された軌跡に最も近い軌跡を探索することで最も真値に近い回転角速度を選択する．回転する卓球ボールを対象とした実画像実験により，本手法が実用的な精度で回転軸及び回転角速度を計測できることを示す．

Access to Document (DOI)

カメラ位置姿勢推定のためのRGB-Dカメラによる3次元線分モデル生成法

中山祐介, 斎藤英雄, 清水雅芳, 山口伸康

映像情報メディア学会誌 69 （ 4 ） J148 - J159 2015年03月

研究論文（学術雑誌）, 査読有り, ISSN 1342-6907

本論文では，カメラ位置姿勢推定のための3次元線分モデルを，対象シーンをRGB-Dカメラで撮影するだけで簡易に生成する手法を提案する．従来の特徴点マッチングによるモデル生成は，特徴点が検出されない場合は失敗するという問題点があった．そのような場面でも線分は安定して検出されるため，3次元線分モデルの生成法が求められている．本提案手法では線分特徴記述子を用いた線分マッチングにより，簡易に，そして3次元情報が必要となるその場で3次元線分モデルを生成する．まず，入力されたRGB画像と距離画像から前フレームの3次元線分と現フレームの2次元線分の対応付けを行う．この2次元3次元線分対応から現フレームのカメラ位置姿勢を推定し，フレーム中の2次元線分を3次元空間上へ逆投影して3次元線分モデルとする．実験により，提案手法が特徴点マッチングではモデルが生成できないシーンでも正確なモデルを生成できることを確認した．

Access to Document (DOI)

環境３Ｄモデルによるカメラトラッキングを用いた温度分布のAR表示システム

中川航, 松本一紀, ドゥソルビエフランソワ, 杉本麻樹, 斎藤英雄, 仙田修司, 柴田剛志, 池谷彰彦

映像情報メディア学会誌 69 （ 4 ） J160 - J168 2015年03月

研究論文（学術雑誌）, 共著, ISSN 1342-6907

近年、温度カメラは、機械設備・建築物の検査や，人間や動物の体温の検査等，多くの目的に利用されるようになってきた．本論文では，RGB-Dカメラで取得した3次元環境モデルを利用して、このモデル上に温度分布をレンダリング表示することにより，対象環境の温度分布を自由な視点から観察可能なシステムを提案する．本システムでは，３Ｄ環境モデルを利用したカメラ位置姿勢のトラッキング手法に基づき，事前に取得しておいた温度分布に対して，現在の温度変化を表示することも可能である．提案システムの有効性を検証するために，幾つかのシーンに対して本システムを利用して表示した温度分布や温度変化分布を示す．

Access to Document (DOI)

Visualization of Temperature Change Using RGB-D Camera and Thermal Camera

Wataru Nakagawa, Kazuki Matsumoto, Francois de Sorbier, Maki Sugimoto, Hideo Saito, Shuji Senda, Takashi Shibata, Akihiko Iketani

Computer Vision - ECCV 2014 Workshops Lecture Notes in Computer Science Volume （Springer International Publishing） 8925 386 - 400 2015年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り, ISSN 0302-9743

In this paper, we present a system for visualizing temperature changes in a scene using an RGB-D camera coupled with a thermal camera. This system has applications in the context of maintenance of power equipments. We propose a two-stage approach made of with an offline and an online phases. During the first stage, after the calibration, we generate a 3D reconstruction of the scene with the color and the thermal data. We then apply the Viewpoint Generative Learning (VGL) method on the colored 3D model for creating a database of descriptors obtained from features robust to strong viewpoint changes. During the second online phase we compare the descriptors extracted from the current view against the ones in the database for estimating the pose of the camera. In this situation, we can display the current thermal data and compare it with the data saved during the offline phase.

Access to Document (DOI)

Marker-less AR system based on line segment feature, The Engineering Reality of Virtual Reality 2015

Yusuke Nakayama, Hideo Saito, Masayoshi Shimizu, and Nobuyasu Yamaguchi

Proc. SPIE 9392, The Engineering Reality of Virtual Reality 2015 93920I 2015年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

We propose a method of marker-less AR which uses line segment feature. Estimating camera poses is important
part of AR system. In most of conventional marker-less AR system, feature point matching between a model and
its image is required for the camera pose estimation. However, a sufficient number of correspondence points is not
always detected to estimate accurate camera poses. To solve this problem, we propose the use of line segments
feature that can possibly be detected even when only a few feature points can be detected. In this paper, we
propose a marker-less AR system that uses line segment features for camera pose estimation. In this system,
we propose a novel descriptor of the line segment feature for achieving fast computation of the camera pose
estimation. We also construct a database which contains a k-d tree of line feature and 3D line segment position
for nding 2D-3D line segment correspondences from input image and the database, so that we can estimate the
camera pose and perform AR. We demonstrated that our proposed method can estimate the camera pose and
provide robust marker-less AR in the situation where point matching method fails.

Access to Document (DOI)

AR visualization of thermal 3D model by hand-held cameras

Kazuki Matsumoto, Wataru Nakagawa, Hideo Saito, Maki Sugimoto, Takashi Shibata, and Shoji Yachida

Proceedings of the Int. Conf. on Computer Vision Theory and Applications (VISAPP2015) 480 - 487 2015年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Illumination estimation and relighting using an RGB-D camera

Yohei Ogura, Takuya Ikeda, Francois de Sorbier, and Hideo Saito

Proceedings of the Int. Conf. on Computer Vision Theory and Applications (VISAPP2015) 305 - 312 2015年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

n this paper, we propose a relighting system combined with a
n illumination estimation method using RGB-
D camera. Relighting techniques can achieve the photometri
c registration of composite images. They often
need illumination environments of the scene which include a
target object and the background scene. Some
relighting methods obtain the illumination environments b
eforehand. In this case, they cannot be used under
the unknown dynamic illumination environment. Some on-lin
e illumination estimation methods need light
probes which can be invade the scene geometry. In our method,
the illumination environment is estimated
from pixel intensity, normal map and surface reflectance bas
ed on inverse rendering in on-line processing.
The normal map of the arbitrary object which is used in the ill
umination estimation part and the relighting part
is calculated from the denoised depth image on each frame. Re
lighting is achieved by calculating the ratio for
the estimated Illumination environment of the each scene. T
hus our implementation can be used for dynamic
illumination or a dynamic object.

Extraction of blood vessels in retinal images using resampling high-order background estimation

Paripurana Sukritta, Chiracharit Werapon, Chamnongthai Kosin, Saito Hideo

IEICE Transactions on Information and Systems E98D （ 3 ） 692 - 703 2015年03月

研究論文（学術雑誌）, 共著, ISSN 0916-8532

In retinal blood vessel extraction through background removal, the vessels in a fundus image which appear in a higher illumination variance area are often missing after the background is removed. This is because the intensity values of the vessel and the background are nearly the same. Thus, the estimated background should be robust to changes of the illumination intensity. This paper proposes retinal blood vessel extraction using background estimation. The estimated background is calculated by using a weight surface fitting method with a high degree polynomial. Bright pixels are defined as unwanted data and are set as zero in a weight matrix. To fit a retinal surface with a higher degree polynomial, fundus images are reduced in size by different scaling parameters in order to reduce the processing time and complexity in calculation. The estimated background is then removed from the original image. The candidate vessel pixels are extracted from the image by using the local threshold values. To identify the true vessel region, the candidate vessel pixels are dilated from the candidate. After that, the active contour without edge method is applied. The experimental results show that the efficiency of the proposed method is higher than the conventional low-pass filter and the conventional surface fitting method. Moreover, rescaling an image down using the scaling parameter at 0.25 before background estimation provides as good a result as a non-rescaled image does. The correlation value between the non-rescaled image and the rescaled image is 0.99. The results of the proposed method in the sensitivity, the specificity, the accuracy, the area under the receiver operating characteristic (ROC) curve (AUC) and the processing time per image are 0.7994, 0.9717, 0.9543, 0.9676 and 1.8320 seconds for the DRIVE database respectively.

Access to Document (DOI)

RGB-Dカメラから取得した3次元点群の平面フィッティングを用いた実時間ノイズ低減と補間

松本一紀，フランソワドゥソルビエ，斎藤英雄

映像情報メディア学会誌 69 （ 3 ） J113 - J120 2015年02月

研究論文（学術雑誌）, 査読有り, ISSN 1342-6907

近年登場した距離センサーを搭載したデバイスの中でもその価格と汎用性ゆえに最も多く利用されているのがMicrosoftのKinectに代表されるRGB-Dカメラである．しかし，RGB-Dカメラから得られる３次元点群は多くのノイズを含んでおり，精密な距離情報の取得が求められるアプリケーションでの利用は依然として困難である．そこで本論文ではRGB-Dカメラから得られた３次元点群の観測ノイズの低減と補間を実時間で行う手法を提案する．本手法はRGB-Dカメラから得られた３次元点群からSuperpixel Segmentaiotnとラベリングを併用して平面で構成される小領域群を検出し，領域毎に平面方程式を推定し３次元点群をフィッティングさせることにより，もとの距離画像に含まれるノイズの低減と補間を行うものである．本手法では検出された領域内の点群の法線分布を算出し，その領域が平面の場合平面フィッティングを適用し曲面の場合Joint Bilateral Filterを適用することで，曲面分布した点群を保持しながら平面分布した点群のみを平面フィッティングすることができる．また，領域分割と平面フィッティングはすべて局所領域ごとに並列化できる手法を用いているため，本手法の実装はすべてGPUで行い実時間での処理が可能である．従来の手法と比べて，本手法はRGB-Dカメラから取得した点群に対して，曲面領域を保持したまま平面領域を平滑化することでノイズ低減と補間を行うことができる．

Access to Document (DOI)

手持ちカメラを用いた走者ストロボ画像の自動生成

長谷川邦洋，斎藤英雄

精密工学会誌 81 （ 2 ） 156 - 163 2015年02月

研究論文（学術雑誌）, 共著, 査読有り

This paper discusses the method for the automatically generating stroboscopic image from a video image using a handheld camera comprehensively. The stroboscopic image can be used in order to measure stride length and speed of the runner based on the authors' previously proposed method, in which it is shown that measuring stride length and speed of the runner from the stroboscopic image is theoretically possible. The proposed automatic stroboscopic image generating method has two steps. The first step is generating a stitched background image from input video image. In the second step, runner images extracted from arbitrary frames are overlapped onto the stitched background image. Thus, this method can generate a stroboscopic image including the runners in arbitrary frames without overwriting them in some frames. These processes can be executed automatically, and these are noteworthy features. In the experiments under various environment to show the effectiveness of this method, it was confirmed that could generate a stroboscopic images well in respectively environment. It is also demonstrated that stride length and speed of the runner captured into the generated stroboscopic image can be measured with satisfactory accuracy by manually selected landing points in the stroboscopic image.

Access to Document (DOI)

Plane fitting and depth variance based upsampling for noisy depth map from 3D-ToF cameras in real-time

Matsumoto Kazuki, De Sorbier Francois, Saito Hideo

Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM-2015) （SCITEPRESS (Science and Technology Publications, Lda.)） 150 - 157 2015年01月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Recent advances of ToF depth sensor devices enables us to easily retrieve scene depth data with high frame rates. However, the resolution of the depth map captured from these devices is much lower than that of color images and the depth data suffers from the optical noise effects. In this paper, we propose an efficient algorithm that upsamples depth map captured by ToF depth cameras and reduces noise. The upsampling is carried out by applying plane based interpolation to the groups of points similar to planar structures and depth variance based joint bilateral upsampling to curved or bumpy surface points. For dividing the depth map into piecewise planar areas, we apply superpixel segmentation and graph component labeling. In order to distinguish planar areas and curved areas, we evaluate the reliability of detected plane structures. Compared with other state-of-the-art algorithms, our method is observed to produce an upsampled depth map that is smoothed and closer to the ground truth depth map both visually and numerically. Since the algorithm is parallelizable, it can work in real-time by utilizing highly parallel processing capabilities of modern commodity GPUs.

Access to Document (DOI)

Real Time Detection and Tracking of Printed Maps Based on Road Structure

斎藤英雄

ITE Transactions on Media Technology and Applications 3 （ 1 ） 85 - 94 2015年01月

研究論文（学術雑誌）, 共著, 査読有り

This paper presents a method to detect and track a paper map and overlay virtual data over it. First, we take a picture of the map and identify the road colors using our road area extraction method. Using the color information we can then extract in real time the roads from a video and use the position of the intersections to identify the map from a database without any further user input. The main issue is to get reliable intersection positions in real time even though the map can contain a lot of overlapping data. Finally, using the location of the intersections we can track and augment the map with virtual information. Our method enables the real time matching, 3D tracking and augmentation of a paper map based on a database that can contain several thousands of intersections coordinates.

Access to Document (DOI)

REAL-TIME ENHANCEMENT OF RGB-D POINT CLOUDS USING PIECEWISE PLANE FITTING

Kazuki Matsumoto, Francois de Sorbier and Hideo Saito

Proceedings of the 2014 5th European Workshop on Visual Information Processing (EUVIP2014) （IEEE） 2014年12月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Access to Document (DOI)

非同期カメラを用いたボールの3次元軌跡復元法

玉城将，斎藤英雄

精密工学会誌（精密工学会） 80 （ 12 ） 1157 - 1165 2014年12月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 0912-0289

Ball trajectory is an important performance indicator in ball game. However, Measuring 3D trajectories is often unusable because synchronized cameras and control points inside the playing field are generally required. A method that reconstructs 3D trajectory of ball with unsynchronized cameras is proposed in this paper. Proposed method consists of ball detection, camera calibration, and trajectory reconstruction. At first, ball candidates are detected based on the appearance. For the camera calibration, highly probable balls are extracted from the candidates based on the motion. Imaged ball trajectories are then reconstructed. Next, fundamental matrix is computed from the corresponding points, which are estimated with the ball trajectories, and roughly estimated temporal offset, which is estimated manually. The estimated fundamental matrix is assumed to be inaccurate due to the inaccuracy of the temporal offset. The key feature of this method is to accurately estimate temporal offset and fundamental matrix by optimizing them iteratively based on the error of fundamental matrix. After the calibration, balls are extracted robustly with geometrical and temporal relationship between cameras. A ball trajectory is finally reconstructed as connected trajectories which separated at collisions. It is experimentally demonstrated that the proposed method can calibrate cameras precisely and reconstruct accurate ball trajectories.

Access to Document (DOI)

RGB-Dカメラを用いた実時間リライティング映像生成システムの開発

池田拓也, 小倉洋平, ドゥソルビエフランソワ, 斎藤英雄

映像情報メディア学会誌（映像情報メディア学会） 68 （ 12 ） J558 - J568 2014年11月

研究論文（学術雑誌）, 査読有り, ISSN 1342-6907

リライティングとは，実画像から，それを撮影した照明環境を別の正面環境で撮影したかのような画像を合成する処理のことであり，その実現には，対象物体の3次元形状（法線分布），反射特性，光源環境が必要である．本論文では，RGB-Dカメラを用いて実時間リライティングを実現可能なシステムを提案する．本システムでは，正確な物体領域抽出のためにノイズを含む距離画像の修正を行う．領域抽出を行った後，対象物体の法線分布を推定する．光源環境は，観測された鏡面球から取得し，任意の環境に対してリライティングを行う．また，実時間処理を実現するために，GPUを用いた並列処理を行う．我々の実装では，約10fpsの速度でリライティングが可能である．提案システムの有効性を確認するために行った実験では，対象物体に対して事前に撮影した動的光源環境へリライティングを行い，提案システムの性能を評価した．

Access to Document (DOI)

Auto-Generation of Runner's Stroboscopic Image and Measuring Landing Points Using a Handheld Camera

K. Hasegawa and H. Saito

Proc. 2014 Irish Machine Vision and Image Processing (IMVIP2014) 169 - 174 2014年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Specular 3D Object Tracking by View Generative Learning

Yukiko Shinozuka, Francois de Sorbier and Hideo Saito

Proc. 2014 Irish Machine Vision and Image Processing (IMVIP2014) 9 - 14 2014年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Combination Method: Photometric Stereo with Shadows

A. Dulac, S. Martedi, H. Saito, K. Tezuka and M. Shimizu

Proc. 2014 Irish Machine Vision and Image Processing (IMVIP2014) 137 - 142 2014年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Human Trajectory Tracking Using a Single Omnidirectional Camera

A. Kawasaki, D. Hung and H. Saito

Proc. 2014 Irish Machine Vision and Image Processing (IMVIP2014) 157 - 162 2014年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Accurate Camera Pose Estimation for KinectFusion Based on Line Segment Matching by LEHF

Yusuke Nakayama, Toshihiro Honda, Hideo Saito, Masayoshi Shimizu,and Nobuyasu Yamaguchi Accurate Camera Pose Estimation for KinectFusion

Proceedings of the 2014 22nd International Conference on Pattern Recognition （IEEE Computer Society Washington, DC, USA） 2149 - 2154 2014年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り, ISSN 978-1-4799-5209-0

Access to Document (DOI)

Camera Pose Estimation for Mixed and Diminished Reality in FTV

Hideo Saito, Toshihiro Honda, Yusuke Nakayama, Francois de Sorbier

3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), 2014 （IEEE） 2014年07月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Access to Document (DOI)

RGB-Dカメラによる環境の3次元計測に基づく実時間隠消現実感

本田俊博，斎藤英雄

日本バーチャルリアリティ学会論文誌（日本バーチャルリアリティ学会） 19 （ 2 ） 105 - 116 2014年06月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1344011X

Dynamic learning, retrieval, and tracking to augment hundreds of photographs

Julien Pilet, Hideo Saito

Virtual Reality （Springer London） 18 （ 2 ） 89 - 100 2014年06月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1434-9957

In this paper, we present a method that combines the speed and accuracy of tracking with the scalability of image retrieval. At the heart of our approach is a bi-layer clustering process that allows our system to index and retrieve objects based on tracks of features, thereby effectively summarizing the information available on multiple video frames. Dynamic learning of new viewpoints as the camera moves naturally yields the kind of robustness and reliability expected from an augmented reality engine. As a result, our system is able to track in real-time multiple objects, recognized with low delay from a database of more than 300 entries. We released the source code of our system in a package called Polyora.

Access to Document (DOI)

Rapid Feedback of Table Tennis Ball Trajectory with Unsynchronized Cameras

Sho Tamaki, Hideo Saito

Proc. International Association of Computer Science in Sports 2014 Conference 39 - 45 2014年06月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Virtual rope slider

Tatsuya Kodera, Naoto Tani, Jun Morita, Naoya Maeda, Kazuna Tsuboi, Motoko Kanegae, Yukiko Shinozuka, Sho Shimamura, Kadoki Kubo, Yusuke Nakayama, Jaejun Lee, Maxime Pruneau, Hideo Saito, Maki Sugimoto

Proceedings of the 2014 Virtual Reality International Conference （ Article No. 36 ） 2014年04月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Access to Document (DOI)

Virtual Slicer: Interactive visualizer for tomographic medical images based on position and orientation of handheld device

Sho Shimamura, Motoko Kanegae, Yuji Uema, Masahiko Inami, Tetsu Hayashida, Hideo Saito, and Maki Sugimoto

Proceedings of the 2014 Virtual Reality International Conference （ Article No. 12 ） 2014年04月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

This paper introduces an interface that helps understand the
correspondence between the patient and medical images.
Surgeons determine the extent of resection by using tomographic
images such as MRI (Magnetic Resonance Imaging) data
.
However, understanding the relationship between the patient and
tomographic images is di
ffi
cult. This study aims to visualize the
correspond
en
ce more intuitive
ly
.
In this paper, we propose an
interactive visualizer for medical images based on the relative
position and orientation of the handheld device and the patient.
We conducted an experiment to verify the performances of the
proposed method and several other methods. In the experiment,
t
he
proposed method showed the minimum error.

Access to Document (DOI)

Sharing 3D object with multiple clients via networks using vision-based 3D object tracking

Yukiko Shinozuka and Hideo Saito

Proceedings of the 2014 Virtual Reality International Conference （ Article No. 34 ） 2014年04月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

This paper proposes a new system for 3D object sharing with
multiple clients using networks. Our system aims to support group
communication across devices over the same 3D object and puts
the annotations with augmented reality automatically. In order to
keep putting the annotations during the interaction, we apply the
vision-based 3D object tracking algorithm which estimates 3D
position and pose of the object just by capturing the object with a
single RGB camera. By employing the tracking algorithm robust
to viewpoint changes and occlusion, we develop the 3D object
sharing system with a desktop as a server and a laptop and a smart
phone as clients. We conducted the experiments to show the
tracking algorithm is tolerant of interaction and can be applied to
various objects with texture.

Access to Document (DOI)

An AR edutainment system supporting bone anatomy learning

斎藤英雄

IEEE Virtual Reality (VR2014) 113 - 114 2014年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

We present a medical Augmented Reality (AR) edutainment system for bone anatomy learning. This learning environment, called AR bone puzzle, is a metaphor for bone anatomy learning with AR visualization and intuitive interaction. AR bone puzzle uses its user's body as a puzzle frame and computer generated virtual bones as puzzle pieces. Users learn bone anatomy by assembling the virtual bone pieces on their body. Key features of this system are 3D AR visualization and intuitive gesture based user interaction.

Access to Document (DOI)

Stereoscopic augmented reality with pseudo-realistic global illumination effects

斎藤英雄

Proc. SPIE. 9011, Stereoscopic Displays and Applications XXV 90111W 2014年03月

研究論文（国際会議プロシーディングス）, 査読有り

Access to Document (DOI)

Joint upsampling and noise reduction for real-time depth map enhancement

Kazuki Matsumoto, Chiyoung Song, Francois de Sorbier, Hideo Saito

Proc. SPIE 9011, Stereoscopic Displays and Applications XXV 901120 2014年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

An efficient system that upsamples depth map captured by Microsoft Kinect while jointly reducing the effect of noise is presented. The upsampling is carried by detecting and exploiting the piecewise locally planar structures of the downsampled depth map, based on corresponding high-resolution RGB image. The amount of noise is reduced by accumulating the downsampled data simultaneously. By benefiting from massively parallel computing capability of modern commodity GPUs, the system is able to maintain high frame rate. Our system is observed to produce the upsampled depth map that is very close to the original depth map both visually and mathematically.

Access to Document (DOI)

A calibration method for performance analysis in table tennis

Sho Tamaki, Hideo Saito

Proc. 20th Korea-Japan Joint Workshop on Frontiers of Computer Vision 330 - 334 2014年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

3D Surface Reconstruction of a BGA Connector with Large Specular Highlightsby Photometric Stereo

Maxime Vallee, Hideo Saito, Junichi Sugano, Yoshinobu Takizawa

Proc. 20th Korea-Japan Joint Workshop on Frontiers of Computer Vision 2014年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

疑似正面画像を用いた視点によらない車名判別

篠塚祐紀子, 皆川卓也, 斎藤英雄

電気学会論文誌Ｃ（電子・情報・システム部門誌） 134 （ 2 ） 233 - 241 2014年02月

研究論文（学術雑誌）, 査読有り

This paper proposes a novel view independent vehicle make and model recognition method (VMMR). Our system identifies the make and model from the variety of viewpoints while the conventional methods for VMMR work only for the fixed frontal or rear images. In addition, it needs only the 2D images not CAD data for database. To solve the alignment issue, our method uses SIFT, that has scale and rotation invariance. For the view independent recognition, it creates less distorted frontal view images by view morphing or homography matrix calculated by the position of the license plate and extracts the keypoints from them. Our method enables to recognize up to 60-degree angle with high accuracy due to the less distorted virtual frontal images.

Access to Document (DOI)

View Independent Recognition of Vehicle Make and Model from View Morphed Frontal Image

Yukiko Shinozuka, Takuya Minagawa, Hideo Saito

2014 Int. Workshop on Advanced Image Technology (IWAIT 2014) 451 - 455 2014年01月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

This paper proposes a novel view independent
vehicle make and model recognition method(VMMR). Our system
identifies the make and model from the variety of viewpoints while
the conventional methods for VMMR work only for the fixed
frontal or rear images. In addition, it needs only the 2D images
not CAD data for database. To solve the alignment issue, our
method uses SIFT, that has scale and rotation invariance. For the
view independent recognition, it creates the more realistic and less
distorted frontal view images by view morphing and extracts the
keypoints from them. Our method enables to recognize up to 40-
degree angle with high accuracy due to the less distorted morphed
images. Our method can be extended to the other product model
recognition.

Augmented reality system using a smartphone based on getting a 3D environment model in real-time with an RGB-D camera

Toshihiro Honda, Francois de Sorbier, and Hideo Saito

2014 Int. Workshop on Advanced Image Technology (IWAIT 2014) 355 - 359 2014年01月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

n this paper, we propose a system to achieve
Augmented Reality (AR) on a smartphone. In this system, we
assume a fixed RGB-D camera connected to a server is installed in
the environment, and perform AR based on the 3D environment
shape got by the RGB-D camera in real-time. On the smartphone
side, it sends the image captured by its camera, and receives
the output image processed to perform AR by the server. On
the server side, it sends back the image that virtual objects
are superimposed on. At this time, by constructing the 3D
environment model in real-time using the RGB-D camera, it is
possible to calculate the smartphone pose and the interaction
between the virtual objects and the environment properly. It is
also possible to change the lighting virtually. Our experiment
revealed it is possible to superimpose virtual objects on the
smartphone in approximately 8fps.

Real time relighting with dynamic light environment using an RGB-D camera

Takuya Ikeda, Francois de Sorbier, and Hideo Saito

2014 Int. Workshop on Advanced Image Technology (IWAIT 2014) 420 - 425 2014年01月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

elighting techniques often require knowledge
about the shape and the reflectance property of a target object
and the light environment. The quality of the relighting is highly
dependent on the normals accuracy of the object since they
are used for the computation of the illumination. We propose
a new relighting approach for arbitrary shaped objects with
dynamic light environment using an RGB-D camera such as
the Microsoft’s Kinect. The generated depth map is useful to
estimate the normals of the object, but it’s inaccurate because of
the noise such as discrete depth values or missing data. We focus
on the depth map modification to segment the accurate object
region, and normal estimation for relighting. Our implementation
achieves a relighting at 10fps with GPU. Therefore, our relighting
system can apply for a moving object and a dynamic light
environment in real time.

Speed up in Computation of HMAX Features for Object Localization

斎藤英雄

ITE Transactions on Media Technology and Applications 2 （ 1 ） 60 - 73 2014年01月

研究論文（学術雑誌）, 査読有り

While HMAX features have been proved to have excellent performance in image categorization tasks, the computational cost of recognition is expensive. If we aim to apply the HMAX features to object localization tasks, in which the categorization tasks are repeatedly performed by sliding windows, their processing time increases enormously. In this paper, we propose a method for speed up in computation of object localization based on HMAX features. We found that the HMAX features cause specific redundancies in the sliding window approach. The speed up is achieved by eliminating the redundancies in our method. The results from experiments using the University of Illinois-Urbana-Champaign (UIUC) car dataset and the face detection dataset benchmark (FDDB) indicate that this modification improved processing speeds significantly with insignificant reductions in precision.

Access to Document (DOI)

Plane approximation-based approach for 3d reconstruction of ball trajectory for performance analysis in table tennis

Tamaki Sho, Saito Hideo

Advances in Computer Vision and Pattern Recognition 71 47 - 66 2014年

研究論文（学術雑誌）, 共著, ISSN 2191-6586

A plane approximation-based approach for 3D reconstruction of ball trajectory is introduced, which was developed as a solution to the problem with conventional analysis in table tennis. There are methods of reconstructing 2D ball trajectories or 3D trajectories of balls heavier than those in table tennis. However, these methods cannot be adopted to reconstruct the 3D trajectories of table tennis balls, because there are problems that are attributed to the dimensions of the trajectories and weight of the balls. The method proposed in this chapter could reconstruct the 3D trajectories of a table tennis ball. The key feature of the method is that it approximates that a ball is traveling on tilted planes. This approximation makes reconstruction robust against failure to measure 3D ball positions. Two systems were developed based on the new method. A system using two RGB cameras experimentally demonstrated that it could provide accurate information for match analysis. A system using an RGB-D camera was experimentally demonstrated that the system could provide accurate information for service analysis.

Access to Document (DOI)

A Tracking Method for 2D Canvas in MR-Based Interactive Painting System

斎藤英雄

International Conference on Signal-Image Technology & Internet-Based Systems (SITIS2013) （IEEE） 790 - 794 2013年12月

研究論文（国際会議プロシーディングス）, 査読有り

We have proposed a mixed reality based painting system. In this paper, we tackle a problem in our previous painting system that fully relies on magnetic sensor that is attached on the canvas, the users needed to detach and attach a sensor on the canvas during painting when they want to switch the canvas. Instead of that, in this paper, we aim to automatically detect the shape of the canvas for registration purpose. Using the shape or region detection method such as MSER (maximally stable extremal regions), we detect and track the shape on the canvas on the captured camera image. We then compute the camera pose for virtually overlay the painting result. Using the brush device, we can draw and paint freely on the tracked canvases. We show that using visual based tracking method, we can generate the equivalent result compared to the result of using the sensor.

Access to Document (DOI)

Arbitrarily Shaped Objects Relighting Using an RGB-D Camera

斎藤英雄

2nd IAPR Asian Conference on Pattern Recognition (ACPR), 2013 631 - 636 2013年11月

研究論文（国際会議プロシーディングス）, 査読有り

Access to Document (DOI)

Detecting Fall Incidents of the Elderly Based on Human-Ground Contact Areas

斎藤英雄

2nd IAPR Asian Conference on Pattern Recognition (ACPR), 2013 516 - 521 2013年11月

研究論文（国際会議プロシーディングス）, 査読有り

Access to Document (DOI)

パフォーマンス分析への応用を目的とした卓球ボールの3次元軌跡復元システム

玉城将、斎藤英雄

精密工学会誌 79 （ 11 ） 1102 - 1109 2013年11月

研究論文（学術雑誌）, 共著, 査読有り

Virtual Slicer: Development of Interactive Visualizer for Tomographic Medical Images Based on Position and Orientation of Handheld Device

斎藤英雄

International Conference on Cyberworlds (CW), 2013 （IEEE） 383 - 383 2013年10月

研究論文（国際会議プロシーディングス）, 査読有り

Access to Document (DOI)

Region-based tracking using sequences of relevance measures

斎藤英雄

IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2013 （IEEE） 1 - 6 2013年10月

研究論文（国際会議プロシーディングス）, 査読有り

We present the preliminary results of our proposal: a region-based detection and tracking method of arbitrary shapes. The method is designed to be robust against orientation and scale changes and also occlusions. In this work, we study the effectiveness of sequence of shape descriptors for matching purpose. We detect and track surfaces by matching the sequences of descriptor so called relevance measures with their correspondences in the database. First, we extract stable shapes as the detection target using Maximally Stable Extreme Region (MSER) method. The keypoints on the stable shapes are then extracted by simplifying the outline of the stable regions. The relevance measures that are composed by three keypoints are then computed and the sequences of them are composed as descriptors. During runtime, the sequences of relevance measures are extracted from the captured image and are matched with those in the database. When a particular region is matched with one in the database, the orientation of the region is then estimated and virtual annotations can be superimposed. We apply this approach in an interactive task support system that helps users for creating paper craft objects.

Access to Document (DOI)

An Automatic Broadcast System for a Weather Report Radio Program

斎藤英雄

IEEE Transactions on Broadcasting （IEEE） 59 （ 3 ） 548 - 555 2013年09月

研究論文（学術雑誌）, 査読有り, ISSN 0018-9316

Access to Document (DOI)

From smartphone to virtual window

斎藤英雄

IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013 （IEEE） 1 - 6 2013年07月

研究論文（国際会議プロシーディングス）, 査読有り

We propose a vehicle make and model recognition method for the smart phone, and implemented it onto them. Our method identifies the make and model from the variety of viewpoints while the conventional methods for VMMR work only for the frontal or rear view images. This method enables the users to take the pictures from a certain range of angles. Our method uses SIFT, that has scale and rotation invariance to solve the alignment issue. It creates the pseudo frontal view images by homography matrix and extracts the keypoints. Homography matrix is calculated with the position of the license plate. Our experimental result shows our method enables to recognize up to 60-degree angle.

Access to Document (DOI)

Vehicle make and model recognition by keypoint matching of pseudo frontal view

斎藤英雄

IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013 （IEEE） 1 - 6 2013年07月

研究論文（国際会議プロシーディングス）, 査読有り

Access to Document (DOI)

Reconstruction of 3D Trajectories for Performance Analysis in Table Tennis

斎藤英雄

IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2013 （IEEE） 1019 - 1026 2013年06月

研究論文（国際会議プロシーディングス）, 査読有り

A method of reconstructing the 3D trajectories of a table tennis ball is introduced, which was developed to solve the problem with conventional analysis in table tennis. There are several methods of reconstructing 2D ball trajectories or 3D trajectories of balls heavier than those in table tennis. However, these methods cannot be adopted to reconstruct the 3D trajectories of table tennis balls, because there are problems that are attributed to the dimensions of the trajectories and weight of the balls. The method proposed in this paper could reconstruct the 3D trajectories of a table tennis ball. The key feature of the method is that it approximates that a ball is traveling on tilted planes. This approximation makes reconstruction robust against failure to measure 3D ball positions. A system using two RGB cameras was developed based on the new method. The system experimentally demonstrated that it could provide accurate information for match analysis. A system using an RGB-D camera was then developed to optimize usability for practitioners. We experimentally demonstrated that the system could provide accurate information for service analysis.

Access to Document (DOI)

Displayed Object Recognition for Smartphone Interaction

斎藤英雄

13th IAPR International Conference on Machine Vision Applications (MVA 2013) 105 - 108 2013年05月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

A Training Support System for Table Tennis Service using Kinect

斎藤英雄

The 13th ITTF Sports Science Congress 2013年05月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

RGB-Dカメラから得られる部分物体形状と影に基づく光源推定

池田拓也, 小山田雄仁, 杉本麻樹, 斎藤英雄

映像情報メディア学会誌（社団法人映像情報メディア学会） 67 （ 4 ） J124-J133 2013年03月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 13426907

Feature-based Alignment Method for Projecting Virtual Content on a Movable Paper Map

斎藤英雄

IEEJ Trans. on Electronics, Information, and Systems （電気学会） 133 （ 3 ） 672-679 2013年03月

研究論文（学術雑誌）, 共著, 査読有り

Stable Keypoint Recognition using Viewpoint Generative Learning

斎藤英雄

the International Conference on Computer Vision Theory and Applications(VISAPP2013) （The Institute for Systems and Technologies of Information, Control and Communication (INSTICC)） 2 310 - 315 2013年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

We propose a stable keypoint recognition method that is robust to viewpoint changes. Conventional local
features such as SIFT, SURF, etc., have scale and rotation invariance but often fail in matching points when
the camera pose significantly changes. In order to solve this problem, we adopt viewpoint generative learning.
By generating various patterns as seen from different viewpoints and collecting local invariant features, our
system can learn feature descriptors under various camera poses for each keypoint before actual matching. Ex-
perimental results comparing usual local feature matching or patch classification method show both robustness
and fastness of learning.

A Mobile AR System for Sports Spectators using Multiple Viewpoint Cameras

斎藤英雄

共著, 査読有り

In this paper, we aimto develop an AR system which supports spectators who are watching a sports game using
smartphones in a spectators’ stand. The final goal of this system is that a spectator can watch information of
players through a smartphone and share experiences with other spectators. For this goal, we propose a system
which consists of smartphones and fixed cameras. Fixed cameras are set to cover the whole sports field and
used to analyze players. Smartphones held by spectators are used to estimate positions where they are looking
on the sports field. We built an AR system which makes annotation of players’ information onto a smartphone
image. And we evaluated the accuracy and the processing time of our system and revealed its practicality.

Non-rigid Surface Tracking for Virtual Fitting System

斎藤英雄

研究論文（国際会議プロシーディングス）, 共著, 査読有り

In this paper, we describe a method for overlaying a texture onto a T-shirt, for improving current virtual fitting
system. In such systems, users can try on clothes virtually. In order to realize such a system, a depth camera
has been used. These depth cameras can capture 3D data in real time and have been used by some industrial
virtual cloth fitting systems. However, these systems roughly, or just do not, consider the shape of the clothes
that user is wearing. So the appearance of these virtual fitting systems looks unnaturally. For a better fitting,
we need to estimate 3D shape of cloth surface, and overlay a texture of the cloth that the user wants to see
onto the surface. There are some methods that register a 3D deformable mesh onto captured depth data of a
target surface. Although those registration methods are very accurate, most of them require large amount of
processing time or either manually-set markers or special rectangles. The main contribution of our method is
to overlay a texture onto a texture of T-shirt in real-time without modifying the surface.

The Estimation of Heights and Occupied Areas of Humans from Two Orthogonal Views for Fall Detection

斎藤英雄

IEEJ Trans. on Electronics, Information, and Systems （電気学会） 133 （ 1 ） 117-127 2013年01月

研究論文（学術雑誌）, 共著, 査読有り

Violin pedagogy for finger and bow placement using augmented reality

斎藤英雄

Asia-Pacific Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC 2012) （IEEE） 1 - 5 2012年12月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Beginners need a long time before being able to
play correctly violin. Learning the bowing technique appears to
be a difficult task and retains most of the attention of beginners.
Besides this point, the finger placement is also an important part
of the learning but often under estimated. One difficulty is that
the fingerboard of the violin does not have frets. In this on-going
work, we present a marker-less augmented reality system that
advises the novice players about their fingering and bowing. We
display in real-time the virtual frets by tracking the violin with a
depth camera. We also capture and recognize the note currently
played to direct the placement of the bow on the strings.

レーザプラズマ式3次元ディスプレイデバイスのための物体の表面特徴量を利用したリソースアウェア・レンダリング

中谷文香、藤代一成、石川尋代、斎藤英雄

日本バーチャルリアリティ学会論文誌（日本バーチャルリアリティ学会） 17 （ 4 ） 419-428 2012年12月

研究論文（学術雑誌）, 共著, 査読有り

デプスカメラを用いたリアルタイム領域抽出による隠消現実感映像生成

清水直樹, 橋本昂宗, 植松裕子, 斎藤英雄

映像情報メディア学会誌（社団法人映像情報メディア学会） 66 （ 12 ） J549-J552 2012年11月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 13426907

Toward an augmented reality system for violin learning support

斎藤英雄

Advances in Depth Image Analysis and Applications Lecture Notes in Computer Science （Springer Berlin Heidelberg） 7854 136 - 145 2012年11月

研究論文（国際会議プロシーディングス）, 共著, 査読有り, ISSN 0302-9743

Violin is one of the most beautiful but also one of the most difficult musical instruments for a beginner. This paper presents an on-going work about a new augmented reality system for training how to play violin. We propose to help the players by virtually guiding the movement of the bow and the correct position of their fingers for pressing the strings. Our system also recognizes the musical note played and the correctness of its pitch. The main benefit of our system is that it does not require any specific marker since our real-time solu-tion is based on a depth camera.

Access to Document (DOI)

Camera Pose Estimation of a Smartphone at a Field without Interest Points

斎藤英雄

Computer Vision - ACCV 2012 Workshops Lecture Notes in Computer Science （Springer Berlin Heidelberg） 7729 545 - 555 2012年11月

研究論文（国際会議プロシーディングス）, 共著, 査読有り, ISSN 0302-9743

Access to Document (DOI)

Colour Descriptors for Tracking in Spatial Augmented Reality

斎藤英雄

Computer Vision - ACCV 2012 Workshops Lecture Notes in Computer Science （Springer Berlin Heidelberg） 7729 387 - 399 2012年11月

研究論文（国際会議プロシーディングス）, 共著, 査読有り, ISSN 0302-9743

Augmented Reality is an emerging research field, that aims
for the composition of real and virtual imagery, by means of a camera
and display device. Spatial augmented reality employs data projectors
to augment the real world. In this setting, traditional tracking methods
fall short due to the interference caused by the projector. Recent works
assume a calibration process to model the projector and assume continuity
in movement of the object being tracked. In this paper we present
a tracking-by-detection system that does not require such a procedure
and makes use of natural features represented by SIFT descriptors. We
evaluate a set of photometric invariants that have previously been shown
to improve the performance of object recognition, added to the descriptor
to reduce the influence of the projector. We evaluate the descriptors
based on precision-recall under projector distortion and the total system
based on its tracking performance. Results show tracking is significantly
more precise using one of the invariants.

Access to Document (DOI)

Illumination Estimation from Shadow and Incomplete Object Shape Using an RGB-D Camera

斎藤英雄

International Conference on Pattern Recognition (ICPR2012) （IEEE） 165 - 169 2012年11月

研究論文（国際会議プロシーディングス）, 共著, 査読有り, ISSN 1051-4651

Optical consistency between the real world and the
virtual objects is one of the important issues in Augmented
Reality (AR). This paper proposes a method
to estimate illuminations from an object shadow and
incomplete object shape information captured by an
RGB-D camera. The environmental illumination can
be estimated without any prior knowledge of the object
shape. The radiance of each light source is computed by
solving linear equations derived from color and depth
images. Since a priori knowledge of object shape is
not necessary, we can flexibly use/combine any objects.
Thus, we can use/combine multiple objects in the target
scene to increase the accuracy and flexibility. Experimental
results show the characteristics of our proposed
method.

Calibration-Free Projector-Camera System for Spatial Augmented Reality on Planar Surfaces

斎藤英雄

International Conference on Pattern Recognition (ICPR2012) （IEEE） 85 - 88 2012年11月

研究論文（国際会議プロシーディングス）, 共著, 査読有り, ISSN 1051-4651

Spatial augmented reality extends augmented real-
ity by projecting virtual data directly on a target sur-
face, but requires to calibrate a projector-camera sys-
tem. This paper introduces a free-calibration projector-
camera system for spatial augmented reality with a pla-
nar surface. A pattern is projected on the target surface
that can be freely moved. The main difficulty is to make
the projected image fitting the target surface. This is of-
ten achieved by calibrating the projector according to
the camera’s pose, but the calibration process is offline
and restricts the configuration of the setup. Our pro-
posed method allows to independently move the projec-
tor, the camera or the surface while the projected image
continuously fit the target surface. The experimental re-
sults demonstrate the efficiency of our calibration-free
spatial augmented reality approach when applied on a
moving planar surface.

可視化と画像解析第2回：CVにおける3次元復元技術とその応用

斎藤英雄

Ｍｅｄｉｃａｌ　Ｉｍａｇｉｎｇ　Ｔｅｃｈｎｏｌｏｇｙ（日本医用画像工学会） 30 （ 4 ） 209-214, 2012年10月

研究論文（学術雑誌）, 単著

複数のスマートフォンカメラの協調利用による実時間隠消現実感

本田俊博、斎藤英雄

日本バーチャルリアリティ学会論文誌（日本バーチャルリアリティ学会） 17 （ 3 ） 181-190 2012年09月

研究論文（学術雑誌）, 共著, 査読有り

線分記述子（LEHF）を用いた実時間SLAM

廣瀬圭佑、斎藤英雄

日本バーチャルリアリティ学会論文誌（日本バーチャルリアリティ学会） 17 （ 3 ） 201-208 2012年09月

研究論文（学術雑誌）, 共著, 査読有り

視点生成型学習による頑健な平面位置姿勢推定

吉田拓洋, 斎藤英雄, 清水雅芳, 田口哲典

日本バーチャルリアリティ学会論文誌（日本バーチャルリアリティ学会） 17 （ 3 ） 191-200 2012年09月

研究論文（学術雑誌）, 共著, 査読有り

Fast Line Description for Line-based SLAM

斎藤英雄

Proceedings of the British Machine Vision Conference （BMVA Press） 83.1 - 83.11 2012年09月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

Simultaneous localization and mapping (SLAM) is a technique to simultaneously
perform mapping of environments and localization of a camera in real-time. Most existing
monocular vision based SLAM techniques use point features as landmarks. However,
images of artificial environments with little texture often contain many line segments,
whereas few point features can be localized in such a scene. We propose here a realtime
line-based SLAM system, and a novel method for describing the features of line
segments (LEHF:Line-based Eight-directional Histogram Feature) in order to establish
correct 2D and 3D line correspondences (2D-3D correspondences). LEHF is a fast and
efficient way of describing features of line segments, which are detected by the line segment
detector (LSD) method. The line-based orthogonal iteration (LBOI) method and
the RANSAC algorithm are applied for the camera pose estimation. We conducted an
experiment in order to test our SLAM system in a desktop environment and to perform
augmented reality (AR). Moreover our SLAM system was evaluated by synthetic data.

Access to Document (DOI)

Automatic Road Area Extraction from Printed Maps Based on Linear Feature Detection

斎藤英雄

IEICE TRANSACTIONS on Information and Systems （電子情報通信学会） E95-D （ 7 ） 1758-1765 2012年07月

研究論文（学術雑誌）, 共著, 査読有り

Real-Time People Counting in Crowded Areas by using Local Empirical Templates and Density Ratios

斎藤英雄

IEICE TRANSACTIONS on Information and Systems （電子情報通信学会） E95-D （ 7 ） 1791 - 1803 2012年07月

研究論文（学術雑誌）, 共著, 査読有り

全方位カメラを用いたテーブルトップ型ARシステム

林　智紀、斎藤　英雄、内山　英昭、ジュリアン　ピレ

画像ラボ 7-15 2012年05月

研究論文（学術雑誌）, 共著

FOOD REGION DETECTION USING BAG-OF-FEATURES REPRESENTATION AND COLOR FEATURE

斎藤英雄

International Conference on Computer Vision Theory and Applications 2012 (VISAPP2012) （SciTePress） 1 709 - 713 2012年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA

斎藤英雄

International Conference on Computer Vision Theory and Applications 2012 (VISAPP2012) （SciTePress） 2 66 - 71 2012年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

We present a method for overlaying a texture onto a non-rigid surface using a commodity depth camera. The
depth cameras are able to capture 3-D data of a surface in real-time, and have several advantages compared
with methods using only standard color cameras. However, it is not easy to register a 3-D deformable mesh to a
point cloud of the non-rigid surface while keeping its geometrical topology. In order to solve this problem, our
method starts by learning many representative meshes to generate surface deformation models. Then, while
capturing 3-D data, we register a feasible 3-D mesh to the target surface and overlay a template texture onto
the registered mesh. Even if the depth data are noisy or sparse, the learning-based method provides us with a
smooth surface mesh. In addition, our method can be applied to real-time applications. In our experiments,
we show some augmented reality results of texture overlay onto a non-textured T-shirt.

Foldable Augmented Maps

斎藤英雄

IEICE TRANSACTIONS on Information and Systems （電子情報通信学会） E95-D （ 1 ） 256-266 2012年01月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 1745-1361

Camera tracking by online learning of keypoint arrangements using LLAH in augmented reality applications

斎藤英雄

Virtual Reality （Springer） 15 （ 2-3 ） 109-117 2011年10月

研究論文（学術雑誌）, 共著, 査読有り

A novel Inpainting-based Layered Depth Video for 3DTV

斎藤英雄

IEEE TRANSACTIONS ON BROADCASTING （IEEE） 57 （ 2 ） 533-541 2011年06月

研究論文（学術雑誌）, 共著, 査読有り

高速カメラ映像を用いた硬式野球ボールの回転速度推定システム

井上卓也，植松裕子,斎藤英雄

電気学会論文誌. D, 産業応用部門誌（社団法人電気学会） 131 （ 4 ） 608-615 2011年04月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 09136339

多視点カメラ撮影による野球のシースルー映像生成

橋本昂宗, 植松裕子斎藤英雄

映像情報メディア学会誌（社団法人映像情報メディア学会） 65 （ 4 ） 505-513 2011年04月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 13426907

個人識別に向けた顔表情変化の時系列解析

田中秀典, 斎藤英雄

電気学会論文誌. C, 電子・情報・システム部門誌（社団法人電気学会） 130 （ 11 ） 2047-2057 2010年11月

研究論文（学術雑誌）, 共著, 査読有り, ISSN 03854221

Real-time video-based rendering from uncalibrated cameras using plane-sweep algorithm

Songkran Jarusirisawad, Vincent Nozick, Hideo Saito

Journal of Visual Communication and Image Representation （Elsevier） 21 577 - 585 2010年01月

研究論文（学術雑誌）, 査読有り

In this paper, we present a new online video-based rendering (VBR) method that creates new views of a
scene from uncalibrated cameras. Our method does not require information about the cameras intrinsic
parameters. For obtaining a geometrical relation among the cameras, we use projective grid space (PGS)
which is 3D space defined by epipolar geometry between two basis cameras. The other cameras are registered
to the same 3D space by trifocal tensors between these basis cameras. We simultaneously reconstruct
and render novel view using our proposed plane-sweep algorithm in PGS. To achieve real-time
performance, we implemented the proposed algorithm in graphics processing unit (GPU). We succeed
to create novel view images in real-time from uncalibrated cameras and the results show the efficiency
of our proposed method.

Access to Document (DOI)

光源マーカを用いた可視光通信に基づく写真測量システム

内山英昭、吉野昌樹、春山真一郎、斎藤英雄、中川正雄、掛橋孝夫、永元直樹

画像電子学会誌、Vol. 38、No.5、pp. 703-711、2009 2009年

査読有り

3DTV View Generation Using Uncalibrated Pure Rotating and Zooming Cameras

Songkran Jarusirisawad and Hideo Saito

the Special Issue on Advances in Three Dimensional Television and Video of the Journal Signal Processing: Image Communication 24 17-30 2009年

研究論文（学術雑誌）, 共著, 査読有り

Multiple Planes Based Registration Using 3D Projective Space for Augmented Reality

''Yuko Uematsu, Hideo Saito''

''Journal of Image and Vision Computing, Vol.27, pp.1484-1496, 2009.'' 2009年

共著, 査読有り

Model-Based Hand Tracking by Chamfer Distance and Adaptive Color Learning Using Particle Filter

Chutisant Kerdvibulvech and Hideo Saito

''EURASIP Journal on Image and Video Processing, Volume 2009 (2009), Article ID 724947, 10 pages'' 2009年

共著, 査読有り

Calibration of a Structured Light System Using Planar Objects

''Koichiro Yamauchi, Hideo Saito, and Yukio Sato''

''In Peng-Yeng Yin, editor. Pattern Recognition pp. 1-14, IN-TECH, Vienna, Austria, October 2009'' 2009年

共著, 査読有り

フリーハンド撮影された多視点スナップショットからの顔形状復元

鈴木誠司，斎藤英雄，持丸正明

''映像情報メディア学会誌，Vol.63,No.4, pp.506-515,2009'' 2009年

査読有り

レーザプラズマ式３Dディスプレイにおける点列を用いた物体表現

石川尋代，斎藤英雄

''映像情報メディア学会誌，Vol.63,No.5, pp.665-672,2009'' 2009年

査読有り

画像クエリによる情報検索システム構築のための階層型オブジェクトカテゴリ認識手法

皆川卓也，斎藤英雄

''電気学会論文誌C，Volume 129-C, Number 5, pp.947-955, 2009'' 2009年

査読有り

基準平面を用いたスリット光投影式レンジファインダの校正

山内航一郎，斎藤英雄，佐藤幸男

''電気学会論文誌C，Volume 129-C, Number 6, pp.1162-1168, 2009'' 2009年

査読有り

単眼カメラを用いた車両の実時間蛇行検出手法

手島知昭，斎藤英雄，小澤慎治(愛知工科大学)，山本恵一 (三菱ふそう)，伊原　徹(三菱ふそう)

''電気学会論文誌C，Volume 129-C, Number 9, pp.1714-1723, 2009'' 2009年

査読有り

時間・空間的にまばらな多視点画像を用いた個人識別のための動的な見え方学習法

田中秀典、北原格、斎藤英雄、村瀬洋、小暮潔、萩田紀博

画像電子学会誌、Vol. 38、No.5、pp. 623-633、2009 2009年

査読有り

レスキュー用アレイアンテナ搭載レーダシステム

''竹内敏夫，斎藤英雄，大矢晃久,青木義満，松野文俊，秋山いわき''

''計測自動制御学会論文集, Vol.44, No.12, pp.962-969, 2008'' 2008年

査読有り

Calibration of a Structured Light System by Observing Planar Object

''Koichiro Yamauchi, Hideo Saito and Yukio Sato''

''International Conference on Pattern Recognition, ICPR08, pp.1-4, December, 2008'' 2008年

共著, 査読有り

PCA-based 3D Shape Reconstruction of Human Foot Using Multiple Viewpoint Cameras

''Edmee Amstutz, Tomoaki Teshima, Makoto Kimura, Masaaki Mochimaru, Hideo Saito''

''International Journal of Automation and Computing, Volume 5, Issue 3, pp. 217-225, July 2008.'' 2008年

共著, 査読有り

On-line Free-Viewpoint Video: From Single to Multiple View Rendering

''Vincent Nozick, Hideo Saito''

''International Journal of Automation and Computing, Volume 5, Issue 3, pp. 257-265, July 2008.'' 2008年

研究論文（研究会，シンポジウム資料等）, 共著, 査読有り

Visual Enhancement for Sports Entertainment by Vision-Based Augmented Reality

Yuko Uematsu and Hideo Saito

''Advances in Human-Computer Interaction, Volume 2008 (2008), Article ID 145363, 14 pages.'' 2008年

共著, 査読有り

AR Supporting System for Pool Games Using a Camera Mounted Handheld Display

Hideaki Uchiyama and Hideo Saito

''Advances in Human-Computer Interaction, Volume 2008 (2008), Article ID 357270, 13 pages'' 2008年

共著, 査読有り

Guitarist Fingertip Tracking by Integrating a Bayesian Classifier into Particle Filters

Chutisant Kerdvibulvech and Hideo Saito

''Advances in Human-Computer Interaction, Volume 2008 (2008) , Article ID 384749, 10 pages.'' 2008年

共著, 査読有り

ギター演奏支援のための構造特徴追跡を利用したマーカレスAR表示

元川洋一，斎藤英雄

''日本バーチャルリアリティ学会論文，Vol. 13, No. 2 pp.267-278, 2008'' 2008年

査読有り

複数のハンディカメラの協調利用による遮蔽物体除去映像のオンライン生成

榎本暁人，斎藤英雄

''映像情報メディア学会誌，Vol.62,No.6, pp.901-908,2008'' 2008年

査読有り

鼻孔のテンプレート探索とパーティクルフィルタによるドライバの頭部姿勢追跡

久保田整(ズズキ），武士正美(ズズキ），斎藤英雄

''電気学会論文誌C, Volume 128-C Number 9, pp.1447-1454, 2008'' 2008年

査読有り

プロジェクタスクリーン上に生じる焦点ボケ解消のための投影画像の事前補正法

小山田雄仁，斎藤英雄

''日本バーチャルリアリティ学会論文誌, vol.12, No.4, pp.479-486, 2007'' 2007年

査読有り

Online Multiple View Computation for Autostereoscopic Display

''Vincent Nozick, Hideo Saito''

''Proc. 2007 Pacific Rim Symposium on Image Video and Technology (PSIVT2007), Lecture Note in Computer Science 4872, pp. 399-412, 2007.'' 2007年

共著, 査読有り

Free viewpoint video synthesis and presentation from multiple sporting videos

''Naho Inamoto, Hideo Saito''

''Electronics and Communications in Japan (Part III: Fundamental Electronic Science), Volume 90, Issue 2, pp. 40-49, February 2007.'' 2007年

研究論文（研究会，シンポジウム資料等）, 共著, 査読有り

Real-Time Guitar Chord Recognition System Using Stereo Cameras for Supporting Guitarists

''Chutisant Kerdvibulvech, Hideo Saito''

''ECTI TRANSACTIONS ON ELECTRICAL ENG., ELECTRONICS, AND COMMUNICATIONS VOL.5, NO.2, pp.147-157, August 2007.'' 2007年

共著, 査読有り

Free Viewpoint Video Synthesis based on Natural Features Using Uncalibrated Moving Cameras

''Songkran Jarusirisawad, Hideo Saito''

''ECTI TRANSACTIONS ON ELECTRICAL ENG., ELECTRONICS, AND COMMUNICATIONS VOL.5, NO.2, pp.181-190, August 2007.'' 2007年

共著, 査読有り

Virtual Viewpoint Replay for a Soccer Match by View Interpolation From Multiple Cameras

''Naho Inamoto, Hideo Saito''

''IEEE Trans. Multimedia, Vol.9, Issue 6, pp. 1155-1166, October 2007.'' 2007年

共著, 査読有り

拡張現実表示技術を用いたギターの演奏支援システム

元川洋一，斎藤英雄

''映像情報メディア学会誌，Vol.61,No.6, pp.789 −796, 2007'' 2007年

査読有り

ハンディカメラ入力からのシーン解析に基づくビリヤード戦略発想支援用AR表示システム

内山英昭，斎藤英雄

''日本バーチャルリアリティ学会論文誌，VOL.12，NO.2，pp.159−170, 2007'' 2007年

査読有り

任意位置姿勢に配置された複数マーカの統合によるオンラインARシステム

植松裕子，斎藤英雄

''映像情報メディア学会誌，Vol.60, No.6, 2006.'' 2006年

査読有り

Calibration Free Virtual Display System Using Video Projector onto Real Object Surface

''Shinichiro HIROOKA, Hideo SAITO''

''IEICE TRANSACTIONS on Information and Systems, Volume E89-D, No.1,pp.88-97, 2006'' 2006年

共著, 査読有り

Improving Quality of Free-Viewpoint Image by Mesh Based 3D Shape Deformation

S.Yaguchi，H.Saito

''Journal of WSCG, Vol. 14, No. 1-3, pp. 57-64, 2006'' 2006年

共著, 査読有り

Human Foot Reconstruction from Multiple Camera Images with Foot Shape Database

''Jiahui WANG, Hideo SAITO, Makoto KIMURA, Masaaki MOCHIMARU, Takeo KANADE''

''IEICE TRANSACTIONS on Information and Systems, Volume E89-D, No.5, pp.1732-1742, 2006'' 2006年

共著, 査読有り

平面領域抽出に基づく距離画像の位置合わせ法

渡辺大樹，斎藤英雄

''映像情報メディア学会誌，Vol.60, No.4, pp.576-579, 2006.'' 2006年

査読有り

未校正多視点カメラからの自由視点画像生成のための擬似正射影グリッド空間

磯大輔，斎藤英雄

''画像電子学会誌，Vol.34, No.5, pp.512-521, 2005.'' 2005年

査読有り

未校正カメラ及びプロジェクタによるディジタルコンテンツの実物展示システム

広岡慎一郎，斎藤英雄

''画像電子学会誌，Vol.34, No.1, pp.45-52, 2005.'' 2005年

査読有り

視点間の射影幾何的関係を利用した多視点動画像のデータ圧縮

秋山修，斎藤英雄

''映像情報メディア学会誌，Vol.59, No.1, pp.136-145. 2005.'' 2005年

査読有り

胴体に装着したステレオカメラを用いたゲーム操作用3次元ハンドコントローラ

保利栄作・斎藤英雄

''知能と情報（日本知能情報ファジィ学会誌） Vol.17 , No.2, pp.58-66, 2005年4月'' 2005年

査読有り

3次元動的輪郭モデルによる皮膚内部組織3次元画像からの境界面検出

王　加輝，斎藤英雄，小沢慎治，桑原智裕，山下豊信，高橋元次

''電気学会論文誌C(電子・情報・システム部門誌) 2005年 Vol. 125-C No.5, pp.756-764,2005.'' 2005年

査読有り

プロジェクタと2台のカメラを用いた任意位置姿勢平面への手書きパターンの入力・表示システム

伴　仁志，斎藤英雄

''日本バーチャルリアリティ学会論文誌，投影型インタフェース特集, Vol.10 No.2, pp.173-182, 2005.'' 2005年

査読有り

多視点スポーツ映像からの自由視点映像合成と提示

稲本奈穂，斎藤英雄

''電子情報通信学会論文誌D-II, Vol.J88-D-II, No.8, pp.1693-1701, 2005.'' 2005年

査読有り

Arbitrary Viewpoint Video Synthesis from Multiple Uncalibrated Cameras

S.Yaguchi，H.Saito

''IEEE Trans. on Systems, Man and Cybernetics, PartB, vol. 34, no.1, pp.430-439, Feb. 2004.'' 2004年

共著, 査読有り

Arbitrary viewpoint image synthesis based on light field construction from uncalibrated image sequence

''Kenkichi Kobayashi, Hideo Saito''

''Electronics and Communications in Japan (Part III: Fundamental Electronic Science), Volume 87, Issue 9, pp.11-21, September 2004.'' 2004年

共著, 査読有り

視点位置の内挿に基づく３次元サッカー映像の自由視点観賞システム

稲本奈穂，斎藤英雄

''映像情報メディア学会誌，Vol.58, No.4, pp.529-539, 2004.'' 2004年

査読有り

多視点画像からの薄板状柔軟物体の形状と姿勢の推定

久保田　整，小野勝一，武士正美，斎藤英雄

''電気学会論文誌C(電子・情報・システム部門誌), Vol. 124-C, No.3, pp.659-665, 2004.'' 2004年

査読有り

単眼カメラを用いたインストゥルメンタルパネル操作用ハンドシェープスイッチの構築

五十嵐弘也，斎藤英雄，小沢慎治，山本恵一，佐藤広充

''電気学会論文誌C(電子・情報・システム部門誌), Vol. 124-C, pp.1820-1826, 2004.'' 2004年

査読有り

因子分解法のための射影復元を利用した計測行列の補間法

上島重治，斎藤英雄

''画像電子学会誌，Vol.33, No.4-B,pp.576-585，2004.'' 2004年

査読有り

HMDを用いた形状可変物体表面へのテクスチャ重畳表示システム

江森基倫，斎藤英雄

''日本バーチャルリアリティ学会論文誌，Vol.9 No.3, pp.309-317, 2004.'' 2004年

査読有り

多視点テニス画像からのプレイヤー視点映像提示システム

木村健二，斎藤英雄

''映像情報メディア学会誌，Vol.58, No.10, pp.1503-1510, 2004.'' 2004年

査読有り

全方位カメラ画像からの光線空間構築による任意視点画像生成

等々力寛，斎藤英雄

''映像情報メディア学会誌，Vol.58, No.10, pp.489-1496, 2004.'' 2004年

査読有り

Appearance-Based Virtual View Generation From Multicamera Videos Captured in the 3-D Room

''H.Saito, S.Baba, T.Kanade''

IEEE Trans. on Multimedia vol.5 （ no.3 ） 303-316 2003年09月

共著, 査読有り

Stereo Matching between Three Images by Iterative Refinement in PVS

''M.Kimura, H.Saito, T.Kanade''

IEICE Transactions on Information and Systems Vol.E86-D （ No.1 ） 89-100 2003年01月

共著, 査読有り

High-Resolution Image Synthesis from Video Sequence by Light Field

''Kenkichi Kobayashi, Hideo Saito''

Journal of Robotics and Mechatronics Vol.15 （ No.3 ） 254-262 2003年

共著, 査読有り

Virtual Object Overlay onto Uncalibrated Camera Image Sequence Enabling Tracking of Natural Features

''Shunsuke Harasaki, Hideo Saito''

Journal of Robotics and Mechatronics Vol.15 （ No.3 ） 263-270 2003年

共著, 査読有り

Surface Extraction of Dermo-epidermal Surface from 3-D Volumetric Images of Human Skin

''J.Wang, H.Saito, S.Ozawa, T.Yamashita, T.Kuwahara, M.Takahashi''

International Journal of Image and Graphics，Special Issue on　3D Vision and Medical Image Processing Vol. 3 （ No. 4 ） 589-608 2003年

共著, 査読有り

画面一体型のマルチカメラシステムによる手指の三次元位置の推定

高橋伸昌，斎藤英雄

''計測自動制御学会論文集, Vol.39, No.1, Jan.2003'' 2003年

共著, 査読有り

未校正画像列からの光線空間生成に基づく任意視点画像合成

小林賢吉，斎藤英雄

''電子情報通信学会論文誌D-II，VoL.J86-D-II, No.2, pp.272-281, Feb.2003'' 2003年

共著, 査読有り

未校正の移動カメラによる複数平面シーンの超解像度画像生成

清真一朗，斎藤英雄

''電子情報通信学会論文誌D-II，VoL.J86-D-II, No.5, pp.688-696, May 2003'' 2003年

共著, 査読有り

拡張現実感のための未校正カメラ画像への仮想物体の重ね合わせ

原崎俊介、斎藤英雄

'電子情報通信学会論文誌DII,Jan.2002' VOL.J85-D-II （ NO.1 ） 46-55 2002年01月

共著, 査読有り

距離情報とカラー情報の融合によるシルエット検出に基づく任意視点画像描画システム

''磯大輔, 斎藤英雄, 小沢慎治''

''画像電子学会誌，ビジュアルコンピューティング論文特集号，Vol.31, No.4, pp.525-533, Aug.2002.'' 2002年

共著, 査読有り

平面射影変換を利用した多視点動画像の圧縮

''秋山　修，小沢慎治, 斎藤英雄''

''情報技術レターズ, Sep.2002.'' 2002年

共著, 査読有り

射影幾何を利用した多視点画像からの中間視点映像生成

斎藤英雄，木村誠，矢口悟志，稲本奈穂

''情報処理学会コンピュータビジョンとイメージメディア研究会論文誌，Vol.43, No.SIG11, pp.21-32, Dec.2002'' 2002年

共著, 査読有り

サッカーシーンにおける自由視点映像生成のための視点内挿法

稲本奈穂，斎藤英雄

''日本バーチャルリアリティー学会論文誌TVRSJ, Vol.7, No.4, pp.513-520, Dec.2002'' 2002年

共著, 査読有り

拡張現実感のための未校正カメラ画像への仮想物体の重ね合わせ

原崎俊介，斎藤英雄

''電子情報通信学会論文誌D-II, VOL.J85-D-II, NO.1, pp.46-55, Jan.2002'' 2002年

共著, 査読有り

Virtual 3-D Interface System via Hand Motion Recognition from Two Cameras

''K.Abe, H.Saito, S.Ozawa''

''IEEE Transactions on Systems, Man, and Cybernetics, Part A'' Vol.32 （ No.4 ） 536-540 2002年

共著, 査読有り

3D Reconstruction Based on Epipolar Geometry

'Makoto Kimura, Hideo Saito'

IEICE Transactions on Information and Systems Vol.E84-D （ No.12 ） 1690-1697 2001年12月

共著, 査読有り

皮膚表面画像を用いた肌診断システム

竹前嘉修、斎藤英雄、小沢慎治

'計測自動制御学会論文集, 2001' Vol.37 （ No.11 ） 1097-1103 2001年11月

共著, 査読有り

Recording Multiple Videos in a Large-scale Space for Large-scale Virtualized Reality

'I. Kitahara, Y.Ohta, H.Saito, S.Akimichi, T.Ono, T.Kanade'

'The 8th International Display Workshops, Nagoya, Oct.2001' 2001年10月

研究論文（研究会，シンポジウム資料等）, 共著

サッカー放送における視点選択のための多視点画像の統合によるボール追跡

松本圭介、須藤智、斎藤英雄、小沢慎治

電気学会論文誌Ｃ Vol.121-C （ No.10 ） 1530-1539 2001年10月

共著, 査読有り

Three-dimensional shape reconstructing system from multiple view images using octree and silhouette

'D. Iso, H. Saito, S. Ozawa'