論文 - 斎藤　英雄

魚眼力メラと3D-LIDARを用いた自由視点画像によるSee-Through Vision

大石 圭, 森 尚平, 斎藤 英雄

映像情報メディア学会誌 71 （ 11 ） J276 - J279 2017年10月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  1342-6907

　概要を見る

現実世界から実物体を消し去る隠消現実感 (Diminished Reality; DR) 技術は，見通しの悪い交差点におけるSee-Through Visionなど，さまざま応用例が考えられている．しかし，そこにはいくつかの課題が存在した．第一に隠背景を観測するカメラと，体験者のカメラの直接的なカメラの位置関係を取得することが困難であること．第二に，屋外において隠背景が平面で近似できる場面でしか応用できなかったことである．本稿では，魚眼カメラと3D-LIDARを組み合わせた，高画角のRGB-Dカメラを用いて，両者のカメラの共通の視野を獲得することにより上記の問題解決を図る．また，屋外実験を通して，提案手法によるSee-Through Visionの実現可能性について議論する．

• Access to Document (DOI)

組込み型カメラによる把持動作時の人体手形状3D復元

柏木 直諒, 杉浦 裕太, 宮田 なつき, 多田 充徳, 杉本 麻樹, 斎藤 英雄

映像情報メディア学会誌 71 （ 11 ） J283 - J286 2017年10月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  1342-6907

　概要を見る

本研究では，物体に対する人間の把持姿勢を3D復元するシステムを提案する．ペットボトルやビンといった筒状の物体の蓋の上面に魚眼レンズを装着したカメラを付与し，肌色抽出やセグメンテーションといった画像処理を加えて指を認識する．取得した指位置をディジタルハンドモデルと組合せることで手全体の位置姿勢を復元する．本システムは，日用品の利用状況の計測に基づく製品改良や，スポーツのトレーニングに活用できる．

• Access to Document (DOI)

Registration of RGB and thermal point clouds generated by structure from motion

Trong Phuc Truong, Masahiro Yamaguchi, Shohei Mori, Vincent Nozick, and Hideo Saito

2017 IEEE International Conference on Computer Vision Workshops (ICCVW)    419 - 427 2017年10月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

Thermal imaging has become a valuable tool in various
fields for remote sensing and can provide relevant information
to perform object recognition or classification.
In this paper, we present an automated method to obtain a
3D model fusing data from a visible and a thermal camera.
The RGB and thermal point clouds are generated independently
by structure from motion. The registration process
includes a normalization of the point cloud scale, a global
registration based on calibration data and the output of the
structure from motion, and a fine registration employing a
variant of the Iterative Closest Point optimization. Experimental
results demonstrate the accuracy and robustness of
the overall process.

• Access to Document (DOI)

A Work Area Visualization by Multi-View Camera-Based Diminished Reality

Shohei Mori, Momoko Maezawa and Hideo Saito

Multimodal Technologies Interaction 1 （ 3 ）  2017年09月

研究論文（学術雑誌）, 共著

　概要を見る

Hand-held tools are indispensable for efficient manual working in fields ranging from
woodworking to surgery. In this paper, we present a diminished reality (DR) method to visualize areas
occluded by hands and tools in various hand-working scenarios. We propose a redesigned existing
arbitrary viewpoint image generation method for DR applications as a core DR background rendering
to recover views without undesirable objects. We conducted quantitative and qualitative experiments
using real data to validate the performance of the proposed method. The experimental results showed
that our method runs in real-time (40.1 fps) and surpasses conventional methods (including image
inpainting-based and geometry-based approaches) in terms of image similarity measures.

• Access to Document (DOI)

Structure Based Matching between Aerial and Map Images using Brightness- and Rotation-Invariant Curve Features

Yoshikatsu Nakajima, Hideo Saito

Proceedings of the Irish Machine Vision and Image Processing Conference (IMVIP2017)    233 - 236 2017年08月

研究論文（国際会議プロシーディングス）, 査読有り

　概要を見る

Since conventional feature descriptor algorithms depend on brightness values, those algorithms cannot
obtain point correspondences between aerial and map images where brightness values do not correspond but
geometric structures do correspond. The research in this paper focused on obtaining point correspondences
between those images. This study proposes a novel method by which curve features, edge shape, and
road extraction by a CNN, independent on brightness value, produce robustness in image matching. Its
effectiveness was confirmed through experiments on Massachusetts Roads Dataset.

輝度値変化及び回転に頑健な曲線特徴量

中島 由勝, 斎藤 英雄

電子情報通信学会論文誌D J100-D （ 12 ） 978 - ９８１ 2017年08月

研究論文（学術雑誌）, 査読有り

• Access to Document (DOI)

Ball 3D Trajectory Reconstruction without Preliminary Temporal and Geometrical Camera Calibration

Miyata Shogo, Saito Hideo, Takahashi Kosuke, Mikami Dan, Isogawa Mariko, Kimata Hideaki

2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2017-July   164 - 169 2017年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>This paper proposes a method for reconstructing 3D ball trajectories by using multiple temporally and geometrically uncalibrated cameras. To use cameras to measure the trajectory of a fast-moving object, such as a ball thrown by a pitcher, the cameras must be temporally synchronized and their position and orientation should be calibrated. In some cases, these conditions cannot be met, e.g., one cannot geometrically calibrate cameras when one cannot step into a baseball stadium. The basic idea of the proposed method is to use a ball captured by multiple cameras as a corresponding point. The method first detects a ball. Then, it estimates temporal difference between cameras. After that, the ball positions are used as corresponding points for geometrically calibrating the cameras. Experiments using actual pitching videos verify the effectiveness of our method.</p>

• Access to Document (DOI)

Extrinsic Camera Calibration Without Visible Corresponding Points Using Omnidirectional Cameras

Miyata S., Saito H., Takahashi K., Mikami D., Isogawa M., Kojima A.

IEEE Transactions on Circuits and Systems for Video Technology （IEEE Transactions on Circuits and Systems for Video Technology）  28 （ 9 ） 2210 - 2219 2017年07月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  1051-8215

　概要を見る

© 1991-2012 IEEE. This paper proposes a novel algorithm that calibrates multiple cameras scattered across a broad area. The key idea of the proposed method is 'using the position of an omnidirectional camera as a reference point.' The common approach to calibrating multiple cameras assumes that the cameras capture at least some common points. This means calibration becomes quite difficult if there are no shared points in each camera's field of view (FOV). The proposed method uses the position of an omnidirectional camera to determine point correspondence. The position of an omnidirectional camera relative to the calibrated camera is estimated by the theory of epipolar geometry, even if the omnidirectional camera is placed outside the camera's FOV. This property makes our method applicable to multiple cameras scattered across a broad area. Qualitative and quantitative evaluations using synthesized and real data, e.g., a sports field, demonstrate the advantages of the proposed method.

• Access to Document (DOI)

カメラ位置姿勢推定のための3次元形状の冗長な線分除去

家永 直人, Siim Meerits, 森 尚平, 斎藤 英雄

映像情報メディア学会誌 71 （ 6 ） J189 - J191 2017年06月

研究論文（学術雑誌）, 共著, 査読有り

　概要を見る

本論文では，線分で構成された3次元形状の冗長な線分を除去する手法を提案する．実験では従来手法と提案手法の3次元形状を用いてカメラ位置姿勢推定を行い，その精度を保ちつつ処理時間を短縮したことを示す．また応用例の一つとして透視映像生成を行った．

• Access to Document (DOI)

A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects

Shohei Mori, Sei Ikeda and Hideo Saito

IPSJ Transactions on Computer Vision and Applications （Springer）  9 （ 17 ） 1 - 14 2017年06月

研究論文（学術雑誌）

　概要を見る

In this paper, we review diminished reality (DR) studies that visually remove, hide, and see through real objects from
the real world. We systematically analyze and classify publications and present a technology map as a reference for
future research. We also discuss future directions, including multimodal diminished reality. We believe that this paper
will be useful mainly for students who are interested in DR, beginning DR researchers, and teachers who introduce DR
in their classes.

• Access to Document (DOI)

Diminished hand: A Diminished Reality-Based Work Area Visualization

Mori Shohei, Maezawa Momoko, Ienaga Naoto, Saito Hideo

2017 IEEE Virtual Reality (VR)    443 - 444 2017年04月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>Live instructors perspective videos are useful to present intuitive visual instructions for trainees in medical and industrial settings. In such videos, the instructors hands often hide the work area. In this demo, we present a diminished hand for visualizing the work area hidden by hands by capturing the work area with multiple cameras. To achieve the diminished reality, we use a light field rendering technique, in which light rays avoid passing through penalty points set in the unstructured light fields reconstructed from the multiple viewpoint images.</p>

• Access to Document (DOI)

Speaker Identification Based on Integrated Face Direction in a Group Conversation

Naoto Ienaga, Yuko Ozasa, and Hideo Saito

2017 IEEE Winter Applications of Computer Vision Workshops (WACVW) （IEEE）     53 - 57 2017年03月

研究論文（国際会議プロシーディングス）, 査読有り

　概要を見る

2017 IEEE Winter Applications of Computer Vision Workshops (WACVW)

• Access to Document (DOI)

Measuring grasp posture using an embedded camera

Naoaki Kashiwagi, Yuta Sugiura, Natsuki Miyata, Mitsunori Tada, Maki Sugimoto, and Hideo Saito

2017 IEEE Winter Applications of Computer Vision Workshops (WACVW)    42 - 47 2017年03月

研究論文（国際会議プロシーディングス）, 共著

• Access to Document (DOI)

Line-based SLAM Considering Directional Distribution of Line Features in an Urban Environment

Kei Uehara, Hideo Saito and Kosuke Hara

Proceedings of the 12th International Conference on Computer Vision Theory and Applications (VISAPP2017) 6   255 - 264 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

In this paper, we propose a line-based SLAM from an image sequence captured by a vehicle in consideration
with the directional distribution of line features that detected in an urban environments. The proposed SLAM
is based on line segments detected from objects in an urban environment, for example, road markings and
buildings, that are too conspicuous to be detected. We use additional constraints regarding the line segments
so that we can improve the accuracy of the SLAM. We assume that the angle of the vector of the line segments
to the vehicle’s direction of travel conform to four-component Gaussian mixture distribution. We define a
new cost function considering the distribution and optimize the relative camera pose, position, and the 3D
line segments by bundle adjustment. In addition, we make digital maps from the detected line segments. Our
method increases the accuracy of localization and revises tilted lines in the digital maps. We implement our
method to both the single-camera system and the multi-camera system. The accuracy of SLAM, which uses
a single-camera system with our constraint, works just as well as a method that uses a multi-camera system
without our constraint.

Primitive Shape Recognition via Superquadric Representation using Large Margin Nearest Neighbor Classifier

Ryo Hachiuma, Yuko Ozasa and Hideo Saito

Proceedings of the 12th International Conference on Computer Vision Theory and Applications (VISAPP2017) 5   325 - 332 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

It is known that humans recognize objects using combinations and positional relations of primitive shapes.
The first step of such recognition is to recognize 3D primitive shapes. In this paper, we propose a method for
primitive shape recognition using superquadric parameters with a metric learning method, large margin nearest
neighbor (LMNN). Superquadrics can represent various types of primitive shapes using a single equation with
few parameters. These parameters are used as the feature vector of classification. The real objects of primitive
shapes are used in our experiment, and the results show the effectiveness of using LMNN for recognition based
on superquadrics. Compared to the previous methods, which used k-nearest neighbors (76.5%) and Support
Vector Machines (73.5%), our LMNN method has the best performance (79.5%).

Application of LSD-SLAM for Visualization Temperature in Wide-area Environment

Masahiro Yamaguchi, Hideo Saito and Shoji Yachida

Proceedings of the 12th International Conference on Computer Vision Theory and Applications (VISAPP2017) 4   216 - 223 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

In this paper, we propose a method to generate a three-dimensional (3D) thermal map by overlaying thermal
images onto a 3D surface reconstructed by a monocular RGB camera. In this method, we capture the target
scene moving both an RGB camera and a thermal camera, which are mounted on the same zig. From the
RGB image sequence, we reconstruct 3D structures of the scene by using Large-Scale Direct Monocular
Simultaneous Localization and Mapping (LSD-SLAM), on which temperature distribution captured by the
thermal camera is overlaid, thus generate a 3D thermal map. The geometrical relationship between those
cameras is calibrated beforehand by using a calibration board that can be detected by both cameras. Since we
do not use depth cameras such as Kinect, the depth of the target scene is not limited by the measurement range
of the depth camera; any depth range can be captured. To demonstrating this technique, we show synthesized
3D thermal maps for both indoor and outdoor scenes.

Eating and Drinking Recognition via Integrated Information of Head Directions and Joint Positions in a Group

Naoto Ienaga, Yuko Ozasa and Hideo Saito

Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods (ICPRAM 2017) （Science and Technology Publications）     527 - 533 2017年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

Recent years have seen the introduction of service robots as waiters or waitresses in restaurants and cafes.
In such venues, it is common for customers to visit in groups as well as for them to engage in conversation
while eating and drinking. It is important for cyber serving staff to understand whether they are eating and
drinking, or not, in order to wait on tables at appropriate times. In this paper, we present a method by which
the robots can recognize eating and drinking actions performed by individuals in a group. Our approach uses
the positions of joints in the human body as a feature and long short-term memory to achieve a recognition
task on time-series data. We also used head directions in our method, as we assumed that it is effective for
recognition in a group. The information garnered from head directions and joint positions is integrated via
logistic regression and employed in recognition. The results show that this yielded the highest accuracy and
effectiveness of the robots’ tasks.

First Deployment of Diminished Reality for Anatomy Education

Ienaga Naoto, Bork Felix, Meerits Siim, Mori Shohei, Fallavollita Pascal, Navab Nassir, Saito Hideo

The second International Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR2016), 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) （IEEE）     294 - 296 2017年01月

研究論文（国際会議プロシーディングス）, 査読有り

　概要を見る

<p>Understanding the anatomy of the human body is vital for everyone working in the medical domain. Augmented reality (AR) systems for anatomy teaching, which display virtual information directly on top of a users' body, have proven to facilitate mental mapping compared to traditional teaching paradigms. In this paper, we explore the potential of diminished reality (DR) in the context of anatomy education. As a first necessary step to achieving a DR anatomy education system, parts of the human body have to be extracted and diminished from the video stream. Our system diminishes either the arm or head of the user by projecting a background image recovered using RGB-D cameras. Such a system, if combined with an accurate overlay of virtual counterparts, could potentially improve the learning effect by attracting the users' attention to the virtual information and improve visual perception by avoiding the well-known floating effect of AR.</p>

• Access to Document (DOI)

Detour Light Field Rendering for Diminished Reality Using Unstructured Multiple Views

Mori Shohei, Maezawa Momoko, Ienaga Naoto, Saito Hideo

The second International Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR2016), 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) （IEEE）     292 - 293 2017年01月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>Instructor's perspective videos are useful for presenting intuitive visual instructions for trainees in medical and industrial settings. In such videos, the instructor's arms often obstruct the trainee's view of the work area. In this article, we present a diminished reality method for visualizing the work area hidden by an instructor's arms by capturing the work area with multiple cameras. To achieve such diminished reality, we propose detour light field rendering (DLFR), in which light rays avoid passing through penalty points set in the unstructured light fields reconstructed from multiple viewpoint images. In DLFR, the camera blending field used in an existing freeviewpoint image generation method known as unstructured lumigraph is re-designed based on our use cases. In this re-design, lesser weights are given to light rays as they pass close to given penalty points. Experimental results demonstrate that using DLFR, the appearance of an undesirable object can be removed from an image in real time.</p>

• Access to Document (DOI)

Robust camera pose estimation by viewpoint classification using deep learning

Yoshikatsu Nakajima and Hideo Saito

Computational Visual Media （Springer）  3 （ 2 ） 189 - 198 2016年12月

研究論文（学術雑誌）, 共著, 査読有り

　概要を見る

Camera pose estimation with respect to target scenes is an important technology for superimposing virtual information in augmented reality (AR). However, it is difficult to estimate the camera pose for all possible view angles because feature descriptors such as SIFT are not completely invariant from every perspective. We propose a novel method of robust camera pose estimation using multiple feature descriptor databases generated for each partitioned viewpoint, in which the feature descriptor of each keypoint is almost invariant. Our method estimates the viewpoint class for each input image using deep learning based on a set of training images prepared for each viewpoint class. We give two ways to prepare these images for deep learning and generating databases. In the first method, images are generated using a projection matrix to ensure robust learning in a range of environments with changing backgrounds. The second method uses real images to learn a given environment around a planar pattern. Our evaluation results confirm that our approach increases the number of correct matches and the accuracy of camera pose estimation compared to the conventional method.

• Access to Document (DOI)