論文 - 斎藤　英雄

Reconstruction of 3D models consisting of line segments

Ienaga Naoto, Saito Hideo

ACCV 2016 International Workshops, Revised Selected Papers, Part II, Lecture Notes in Computer Science book series (LNCS, volume 10117) （Springer International Publishing）     100 - 113 2016年11月

研究論文（国際会議プロシーディングス）, 査読有り

　概要を見る

<p>Reconstruction of three-dimensional (3D) models from images is currently one of the most important areas of research in computer vision. In this paper, we propose a method to recover 3D models using the minimum number of line segments. By using structure-from-motion, the proposed method first recovers a 3D model of line segments detected from an input image sequence. We then detect overlapping 3D line segments that redundantly represent a single line structure so that the number of 3D line segments representing the target scene can be reduced without losing the detailed geometry of the structure. We apply matching and depth information to remove redundant line segments from the model while keeping the necessary segments. In experiments, we confirm that the proposed method can greatly reduce the number of line segments. We also demonstrate that the accuracy and computational time for camera pose estimation can be significantly improved with the 3D line segment model recovered by the proposed method. Moreover, we have applied the proposed method to see through occluded areas.</p>

• Access to Document (DOI)

Synthesis of a stroboscopic image from a hand-held camera sequence for a sports analysis

Kunihiro Hasegawa and Hideo Saito

Computational Visual Media （Springer）  2 （ 3 ） 277 - 289 2016年09月

研究論文（学術雑誌）, 共著, 査読有り

　概要を見る

This paper presents a method for synthesizing a stroboscopic image of a moving sports player from a hand-held camera sequence. This method has three steps: synthesis of background image, synthesis of stroboscopic image, and removal of player’s shadow. In synthesis of background image step, all input frames masked a bounding box of the player are stitched together to generate a background image. The player is extracted by an HOG-based people detector. In synthesis of stroboscopic image step, the background image, the input frame, and a mask of the player synthesize a stroboscopic image. In removal of shadow step, we remove the player’s shadow which negatively affects an analysis by using mean-shift. In our previous work, synthesis of background image has been timeconsuming. In this paper, by using the bounding box of the player detected by HOG and by subtracting the images for synthesizing a mask, computational speed and accuracy can be improved. These have contributed greatly to the improvement from the previous method. These are main improvements and novelty points from our previous method. In experiments, we confirmed the effectiveness of the proposed method, measured the player’s speed and stride length, and made a footprint image. The image sequence was captured under a simple condition that no other people were in the background and the person controlling the video camera was standing still, such like a motion parallax was not occurred. In addition, we applied the synthesis method to various scenes to confirm its versatility.

• Access to Document (DOI)

Recognition and pose estimation of primitive shapes from depth images for spatial augmented reality

Ryo Hachiuma and Hideo Saito

Proceedings of IEEE 2nd Workshop on Everyday Virtual Reality (WEVR2016) 2016年03月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

In this paper, we propose a method for recognition and pose estimation of primitive shapes from depth images for spatial augmented reality (SAR). To use SAR in our everyday life, technology to recognize and estimate the pose of projected objects in the room is necessary. However, it is not a simple task to recognize primitive shapes because of their low 3D feature values.

• Access to Document (DOI)

Marker-less augmented reality framework using on-site 3D line-segment-basedmodel generation

Nakayama Yusuke, Saito Hideo, Shimizu Masayoshi, Yamaguchi Nobuyasu

Journal of Imaging Science and Technology 60 （ 2 ） 20401-1 - 20401-24 2016年03月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  1062-3701

　概要を見る

<p>The authors propose a line-segment-based marker-less augmented reality (AR) framework that involves an on-site model-generation method and on-line camera tracking. In most conventional model-based marker-less AR frameworks, correspondences between the 3D model and the 2D frame for camera-pose estimation are obtained by feature-point matching. However, 3D models of the target scene are not always available, and feature points are not detected from texture-less objects. The authors' framework is based on a model-generation method with an RGB-D camera and model-based tracking using line segments, which can be detected even with only a few feature points. The camera pose of the input images can be estimated from the 2D-3D line-segment correspondences given by a line-segment feature descriptor. The experimental results show that the proposed framework can achieve AR when other point-based frameworks cannot. The authors also argue that their framework can generate a model and estimate camera pose more accurately than their previous study.</p>

• Access to Document (DOI)

Chromaticity-matched Superimposition of Foreground Objects in Different Environments

Yohei Ogura, Yoichi Nakasone, Shao Ming and Hideo Saito

Proceedings of the 22nd Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV2016)    141 - 145 2016年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

The illumination chromaticity is one of the important
factor for appearance of objects in images. When we
superimpose some objects on another scene to generate composite
images, the resulting image may give us strangeness if the
illumination chromaticity of foreground scene and background
scene are different. In this paper, we propose a composite
image synthesis method which can achieve illumination chromaticity
consistency. Our goal is to generate a illumination
chromaticity-matched composite image from two images which
are taken under different illumination environment. We estimate
chromaticity of two scenes based on segmented color regions.
Subsequently illumination chromaticity matching is performed
based on estimated chromaticity of two scenes. After the chromaticity
matching process, we extract the target person by using
the grab cut segmentation. The result image is obtained by
superimposing the target person or object onto background
scene. We demonstrate that our method can perform some
different illumination chromaticity environment.

Camera Pose Estimation Based on Keypoints Matching with Pre-Captured Set of Real Images

Kei Obata and Hideo Saito

Proceedings of the 22nd Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV2016)    76 - 80 2016年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

In this paper, we propose a method for camera pose
estimation based on keypoints matching with images capturing
the target scene from multiple viewpoints. For estimating camera
pose from an image, 3D structure of the target scene is needed.
Then a set of correspondences between 2D position in the image
and 3D positions in the target scene provides the camera pose by
solving Perspective-n-Point problem. Image keypoints are used
as these positions, but finding corresponding 3D positions of the
3D structure is not easy problem even if a 3D textured model
of the target scene is given. This is because the appearance of
each 3D position is not always the same. In order to solve this
problem, we adopt real images as a source of keypoints contained
in the database. Real image represents accurate appearance for
each 3D position, so keypoint matching with the input images
can be more accurate. We constructed database based on the
keypoint using the local image feature derived from real images
and evaluated the accuracy of camera pose estimation. The result
showed that the database constructed by our method is effective
to accurate camera tracking.

Stroboscopic Image Synthesis of Sports Player from Hand-Held Camera Sequence

Hasegawa Kunihiro, Saito Hideo

Proceedings of 2015 IEEE International Conference on Computer Vision Workshop (ICCVW)    726 - 733 2016年02月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>A method for synthesizing the stroboscopic image of a moving sports player from a hand-held camera sequence is proposed. In this method, a player is extracted from each frame of an input video sequence using a HOG-based people detector. After removing the bounding box of the extracted player from each frame, all frames are stitched together to generate a background image without the player. The player's area is then overlaid onto the stitched background image and a stroboscopic image of the player is synthesized. By using the bounding box of the player detected by HOG and by subtracting the images, computational speed and accuracy can be improved compared with the conventional method that segments the player's region. In addition to the stroboscopic image synthesis, we also remove the player's shadow to improve the appearance of the resultant stroboscopic image. Experimental results demonstrate the effectiveness of the proposed method.</p>

• Access to Document (DOI)

勾配画像処理に基づく三次元点群からの高精度地図の自動生成

原 孝介, 斎藤 英雄

自動車技術会論文集 47 （ 1 ） 183 - 188 2016年01月

研究論文（学術雑誌）, 共著, 査読有り

Relighting for an arbitrary shape object under unknown illumination environment

Ogura Yohei, Saito Hideo

Proceedings of 11th International Symposium, ISVC 2015,Lecture Notes in Computer Science Volume 9475 （Springer）     433 - 442 2015年12月

研究論文（国際会議プロシーディングス）, 共著

　概要を見る

<p>Relighting techniques can achieve the photometric consistency in synthesizing a composite images. Relighting generally need the object’s shape and illumination environment. Recent research demonstrates possibilities of relighting for an unknown shape object or the relighting the object under unknown illumination environment. However, achieving both tasks are still challenging issue. In this paper, we propose a relighting method for an unknown shape object captured under unknown illumination environment by using an RGB-D camera. The relighted object can be rendered from pixel intensity, surface albedo and shape of the object and illumination environment. The pixel intensity and the shape of the object can simultaneously be obtained from an RGB-D camera. Then surface albedo and illumination environment are iteratively estimated from the pixel intensity and shape of the object.We demonstrate that our method can perform relighting for a dynamic shape object captured under unknown illumination using an RGB-D camera.</p>

• Access to Document (DOI)

アルベドと法線ベクトルの緊密度に着目した影と鏡面反射除去のための組み合わせ照度差ステレオ法

家永　直人, 斎藤 英雄, 清水 雅芳，手塚 耕一，岩村 康正

精密工学会誌 （公益社団法人 精密工学会）  81 （ 12 ） 1154 - 1161 2015年12月

研究論文（学術雑誌）, 共著, 査読有り

　概要を見る

We propose a method of ego-motion estimation for a self-driving car on which we installed cameras with non-overlapping views. By finding corresponding points between the multi-camera images, we aim to enhance the accuracy of the ego-motion estimation. However since the viewing directions are very different from one camera to the other, a conventional algorithm such as SURF cannot detect a sufficient number of correspondences. Additionally in the case where cameras have low frame rate and the vehicle has high speed, the scene changes might be too big to find correspondences between the same camera images. We propose a novel matching algorithm by warping feature patches detected in different cameras based on urban 3D structure. We assume that detected features exist on the surface of buildings or roads and the patch around the feature is planar. Based on this assumption, we can warp the patches so that the feature descriptors are similar for the corresponding feature points. We apply Bundle Adjustment to the found correspondences to optimize the odometry. The result shows higher estimation accuracy when compared to other matching methods.

• Access to Document (DOI)

都市の3次元構造を利用した非重複カメラ間の特徴点マッチングによる車両運動の推定

川崎 敦史, 斎藤 英雄, 原 孝介

精密工学会誌 （公益社団法人 精密工学会）  81 （ 12 ） 1146 - 1153 2015年12月

研究論文（学術雑誌）, 共著, 査読有り

　概要を見る

We propose a method of ego-motion estimation for a self-driving car on which we installed cameras with non-overlapping views. By finding corresponding points between the multi-camera images, we aim to enhance the accuracy of the ego-motion estimation. However since the viewing directions are very different from one camera to the other, a conventional algorithm such as SURF cannot detect a sufficient number of correspondences. Additionally in the case where cameras have low frame rate and the vehicle has high speed, the scene changes might be too big to find correspondences between the same camera images. We propose a novel matching algorithm by warping feature patches detected in different cameras based on urban 3D structure. We assume that detected features exist on the surface of buildings or roads and the patch around the feature is planar. Based on this assumption, we can warp the patches so that the feature descriptors are similar for the corresponding feature points. We apply Bundle Adjustment to the found correspondences to optimize the odometry. The result shows higher estimation accuracy when compared to other matching methods.

• Access to Document (DOI)

Vehicle localization based on the detection of line segments from multi-camera images

Hara Kosuke, Saito Hideo

Journal of Robotics and Mechatronics 27 （ 6 ） 617 - 626 2015年12月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  0915-3942

　概要を見る

<p>For realizing autonomous vehicle driving and advanced safety systems, it is necessary to achieve accurate vehicle localization in cities. This paper proposes a method of accurately estimating vehicle position by matching a map and line segment features detected from images captured by a camera. Features such as white road lines, yellow road lines, road signs, and curb stones, which could be used as clues for vehicle localization, were expressed as line segment features on a two-dimensional road plane in an integrated manner. The detected line segments were subjected to bird’s-eye view transformation to transform them to the vehicle coordinate system so that they could be used for vehicle localization regardless of the camera configuration. Moreover, an extended Kalman filter was applied after a detailed study of the line observation errors for realizing real-time estimation. Vehicle localization was tested under city driving conditions, and the vehicle position was identified with sub-meter accuracy.</p>

• Access to Document (DOI)

Geolocation for printed maps using line segment-based SIFT-like feature matching

Minster Gautier, Moreau Guillaume, Saito Hideo

Proceedings of 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, Challenges and Applications of Urban Augmented Reality    88 - 93 2015年10月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>This paper presents a method for the geolocation of printed maps. It enables the registration of unprepared maps with a Geographical Information System (GIS) database, and can for example be used as a first step to augment an unknown map. We define and match local road pattern descriptors, which are similar to SIFT descriptors [6], but adapted to the case of simple textureless line segments. Using a processing pipeline commonly encountered in the feature-point based matching of texture images - composed of offline description and indexing, followed by an online description, matching and robust transformation estimation - we show that local descriptors can successfully register unprepared maps using only geographic features and no texture information. Our method is scale and rotation invariant, and circumvents the two hurdles that are the level-of-detail, and the changing colormaps and textures, allowing the processing of large classes of printed maps.</p>

• Access to Document (DOI)

Diminished reality for hiding a pedestrian using hand-held camera

Hasegawa Kunihiro, Saito Hideo

Proceedings of 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, International Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR2015)    47 - 52 2015年09月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>This paper proposes a diminished reality method for hiding a pedestrian from a hand-held camera sequence. In the proposed method, the pedestrian is extracted from each frame of input video sequence using HOG based people detector. The detected area is excluded from each frame by masking, and then two frames are stitched for generating a stitched background image. By using the stitched background image, the pedestrian's mask for hiding is made. It is used for overlaying the background pixel to the pedestrian in the original frame. The presented experimental result is demonstrate the effectiveness of the proposed method.</p>

• Access to Document (DOI)

Motion estimation for non-overlapping cameras by improvement of feature points matching based on urban 3D structure

Kawasaki Atsushi, Saito Hideo, Hara Kosuke

Proceedings of 2015 IEEE International Conference on Image Processing (ICIP)    1230 - 1234 2015年09月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>We propose a method of ego-motion estimation for a self-driving vehicle using multiple cameras. By finding corresponding points between the multi-camera images, we aim to enhance the accuracy of the ego-motion estimation. However since the viewing directions are very different from one camera to the other, a conventional algorithm such as SURF cannot detect a sufficient number of correspondences. We propose a novel matching algorithm by warping feature patches detected in different cameras based on urban 3D structure. We assume that detected features exist on the surface of buildings or roads and the patch around the feature is planar. Based on this assumption, we can warp the patches so that the feature descriptors are similar for the corresponding feature points. We apply Bundle Adjustment to the found correspondences to optimizes the odometry. The result shows higher estimation accuracy when compared to other matching method.</p>

• Access to Document (DOI)

Real-time diminished reality for dynamic scenes

Meerits Siim, Saito Hideo

Proceedings of 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, International Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR2015)    53 - 59 2015年09月

研究論文（国際会議プロシーディングス）, 査読有り

　概要を見る

<p>Diminished reality (DR) research aims to develop methods to visually hide objects from real scenes. One or more cameras capture a scene where some objects, called the occluding objects, are 'diminished' by an DR system. To do this in real time we need to use multiple cameras. We introduce a novel multi-camera DR system utilizing an RGB-D camera to hide arbitrary trackable objects from a scene. In our case, the scene background does not have to be planar for the system to work and we can handle scene changes in real time. Additonally, we introduce color correction technique to merge image content originating from different cameras and also demonstrate method to repair images with partially missing content.</p>

• Access to Document (DOI)

Combination photometric stereo using compactness of albedo and surface normal in the presence of shadows and specular reflection

Ienaga Naoto, Saito Hideo, Tezuka Kouichi, Iwamura Yasumasa, Shimizu Masayoshi

Proceedings of the 16th International Conference on Computer Analysis of Images and Patterns, CAIP 2015, LNCS 9256 （Springer）     445 - 455 2015年09月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

<p>We present a novel combination photometric stereo which can estimate surface normals precisely even for images including shadows and specular reflection. We can use photometric stereo if there are more than three input images. Therefore we can employ photometric stereo with nC3 combinations for n input images. We make 3D distribution of albedos and surface normals estimated from pixel intensities of nC3 pixel combinations. In the distribution, we define a novel value “compactness” to distinguish pixels which are included in neither shadows nor specular reflection from pixels which are included in shadows or specular reflection. Through experimental results, we demonstrate that the proposed method can estimate surface normals in the presence of shadows and specular reflection. Moreover the proposed method is superior to previous works in better accuracy.</p>

• Access to Document (DOI)

RGB-Dスーパーピクセル領域分割された局所領域内の平面性を利用した長距離デプス画像の高解像度化

松本一紀，斎藤英雄

電子情報通信学会論文誌 D J98-D （ 9 ） 1226 - 1229 2015年09月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  1881-0225

Reconstructing the 3D Trajectory of a Ball with Unsynchronized Cameras

Tamaki Sho, Saito Hideo

International journal of computer science and sport 14 （ 1 ） 51 - 68 2015年09月

研究論文（学術雑誌）, 査読有り,  ISSN  1684-4769

　概要を見る

The 3D trajectory of the ball is one of the most important performance indicators
in ball games. However, it often cannot be measured because of the requirements:
synchronized cameras and control points within the playing field. Thus, this paper
proposes a method that reconstructs the 3D trajectory of a ball with
unsynchronized cameras. The proposed method consists of ball detection, camera
calibration, and trajectory reconstruction. At first, ball candidates are detected on
the basis of their appearance. Balls are then extracted from the candidates on the
basis of their motion. After the extraction, ball trajectories are reconstructed in
image spaces. The corresponding points in two views are estimated on the basis
of the ball trajectory in image spaces and the temporal offset between cameras,
which is supposed to be recorded with a few erroneous frames. The matrix
including the geometrical relationship between cameras, namely the fundamental
matrix, is then computed from the corresponding points. The estimated one is
inaccurate due to the error of the temporal offset. The key feature of this method
is to optimize the temporal offset and fundamental matrix simultaneously. After
the optimization, the geometrical relationship between cameras is computed from
the matrix. Balls are extracted again by using the temporal offsets and the
geometrical relationship between cameras. A ball trajectory is finally
reconstructed as connected trajectories that are separated at collisions. It is
experimentally demonstrated that the proposed method accurately calibrates
cameras and successfully reconstructs the 3D trajectory of the ball.

Visualization of Dynamic Hidden Areas by Real-Time 3D Structure Acquisition Using RGB-D Camera

Siim Meerits and Hideo Saito

Proceedings of International Display Manufacturing Conference (IDMC) & 3D Systems and Applications (3DSA) 2015    AP4-008 2015年08月

研究論文（国際会議プロシーディングス）, 共著, 査読有り

　概要を見る

We present a system that can accurately visualize changing (dynamic) areas captured by an RGB-D camera at an alternative viewpoint. Our system does not make assumptions about the scene 3D structure and works in real time.