研究者詳細 - 斎藤　英雄

論文 - 斎藤　英雄

分割表示 356 件中 1 - 20 件目／全件表示 >>

InpaintFusion: Incremental RGB-D Inpainting for 3D Scenes

Mori S., Erat O., Broll W., Saito H., Schmalstieg D., Kalkofen D.

IEEE Transactions on Visualization and Computer Graphics （IEEE Transactions on Visualization and Computer Graphics） 26 （ 10 ） 2994 - 3007 2020年10月

ISSN 10772626

© 1995-2012 IEEE. State-of-the-art methods for diminished reality propagate pixel information from a keyframe to subsequent frames for real-time inpainting. However, these approaches produce artifacts, if the scene geometry is not sufficiently planar. In this article, we present InpaintFusion, a new real-time method that extends inpainting to non-planar scenes by considering both color and depth information in the inpainting process. We use an RGB-D sensor for simultaneous localization and mapping, in order to both track the camera and obtain a surfel map in addition to RGB images. We use the RGB-D information in a cost function for both the color and the geometric appearance to derive a global optimization for simultaneous inpainting of color and depth. The inpainted depth is merged in a global map by depth fusion. For the final rendering, we project the map model into image space, where we can use it for effects such as relighting and stereo rendering of otherwise hidden structures. We demonstrate the capabilities of our method by comparing it to inpainting results with methods using planar geometric proxies.

Access to Document (DOI)

Resolving position ambiguity of imu-based human pose with a single RGB camera

Kaichi T., Maruyama T., Tada M., Saito H.

Sensors (Switzerland) （Sensors (Switzerland)） 20 （ 19 ） 1 - 12 2020年10月

ISSN 14248220

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. Human motion capture (MoCap) plays a key role in healthcare and human–robot collaboration. Some researchers have combined orientation measurements from inertial measurement units (IMUs) and positional inference from cameras to reconstruct the 3D human motion. Their works utilize multiple cameras or depth sensors to localize the human in three dimensions. Such multiple cameras are not always available in our daily life, but just a single camera attached in a smart IP devices has recently been popular. Therefore, we present a 3D pose estimation approach from IMUs and a single camera. In order to resolve the depth ambiguity of the single camera configuration and localize the global position of the subject, we present a constraint which optimizes the foot-ground contact points. The timing and 3D positions of the ground contact are calculated from the acceleration of IMUs on foot and geometric transformation of foot position detected on image, respectively. Since the results of pose estimation is greatly affected by the failure of the detection, we design the image-based constraints to handle the outliers of positional estimates. We evaluated the performance of our approach on public 3D human pose dataset. The experiments demonstrated that the proposed constraints contributed to improve the accuracy of pose estimation in single and multiple camera setting.

Access to Document (DOI)

Pose estimation of primitive-shaped objects from a depth image using superquadric representation

Hachiuma R., Saito H.

Applied Sciences (Switzerland) （Applied Sciences (Switzerland)） 10 （ 16 ） 2020年08月

© 2020 by the authors. This paper presents a method for estimating the six Degrees of Freedom (6DoF) pose of texture-less primitive-shaped objects from depth images. As the conventional methods for object pose estimation require rich texture or geometric features to the target objects, these methods are not suitable for texture-less and geometrically simple shaped objects. In order to estimate the pose of the primitive-shaped object, the parameters that represent primitive shapes are estimated. However, these methods explicitly limit the number of types of primitive shapes that can be estimated. We employ superquadrics as a primitive shape representation that can represent various types of primitive shapes with only a few parameters. In order to estimate the superquadric parameters of primitive-shaped objects, the point cloud of the object must be segmented from a depth image. It is known that the parameter estimation is sensitive to outliers, which are caused by the miss-segmentation of the depth image. Therefore, we propose a novel estimation method for superquadric parameters that are robust to outliers. In the experiment, we constructed a dataset in which the person grasps and moves the primitive-shaped objects. The experimental results show that our estimation method outperformed three conventional methods and the baseline method.

Access to Document (DOI)

CorsNet: 3D Point Cloud Registration by Deep Neural Network

Kurobe A., Sekikawa Y., Ishikawa K., Saito H.

IEEE Robotics and Automation Letters （IEEE Robotics and Automation Letters） 5 （ 3 ） 3960 - 3966 2020年07月

© 2016 IEEE. Point cloud registration is a key problem for robotics and computer vision communities. This represents estimating a rigid transform which aligns one point cloud to another. Iterative closest point (ICP) is a well-known classical method for this problem, yet it generally achieves high alignment only when the source and template point cloud are mostly pre-aligned. If each point cloud is far away or contains a repeating structure, the registration often fails because of being fallen into a local minimum. Recently, inspired by PointNet, several deep learning-based methods have been developed. PointNetLK is a representative approach, which directly optimizes the distance of aggregated features using gradient method by Jacobian. In this paper, we propose a point cloud registration system based on deep learning: CorsNet. Since CorsNet concatenates the local features with the global features and regresses correspondences between point clouds, not directly pose or aggregated features, more useful information is integrated than the conventional approaches. For comparison, we also developed a novel deep learning approach (DirectNet) that directly regresses the pose between point clouds. Through our experiments, we show that CorsNet achieves higher accuracy than not only the classic ICP method, but also the recently proposed learning-based proposal PointNetLK and DirectNet, including on seen and unseen categories.

Access to Document (DOI)

Pose Estimation of Stacked Rectangular Objects from Depth Images

Matsuno D., Hachiuma R., Saito H., Sugano J., Adachi H.

IEEE International Symposium on Industrial Electronics （IEEE International Symposium on Industrial Electronics） 2020-June 1409 - 1414 2020年06月

ISSN 9781728156354

© 2020 IEEE. This paper addresses the task of six degrees of freedom (6-DoF) pose estimation of stacked rectangular objects from depth images. Object pose estimation is one of the key challenges for visual processing systems since it plays a vital role in many situations such as warehouse/factory automation, robotic manipulation, and augmented reality. Many recent approaches to object pose estimation use RGB information for detecting and estimating the pose of objects. However, in warehouse/factory automation, objects are often small, occluded, cluttered, and texture-less which makes it difficult to utilize RGB features for detection and pose estimation. In order to overcome this restriction, we use only the depth information (without RGB information) and its geometric features to segment each object and to estimate the 6-DoF (position and orientation) in a stacked scene. We segment the rectangular objects in each scene from the depth and surface normal discontinuities (geometric segmentation). From the geometrically segmented image, four object corner points can be estimated using the convex hull detection and the eight corner points, which are required for the 6-DoF pose estimation, can be calculated. To improve the accuracy of orientation estimation, we estimate four orientation candidates and select the best among them. Experimental results using two evaluation methods show that our method outperformed the baseline method.

Access to Document (DOI)

Detection of cervical myelopathy with Leap Motion Sensor by random forests

Watanabe M., Sugiura Y., Saito H., Koyama T., Fujita K.

LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies （LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies） 214 - 216 2020年03月

ISSN 9781728170633

© 2020 IEEE. This paper presents a method for automatic detection of cervical myelopathy by using Leap Motion Sensor for sensing hand motion to apply a method called the 10-s grip and release test. Leap Motion data is analyzed for computing feature of hand motion, which is used for automatic detection of patient or healthy people by random forests algorithm for machine learning. For evaluating automatic detection performance, we collect real data of hand motion 54 of patient and healthy people, which is used for validating accuracy of detection.

Access to Document (DOI)

Gait Measurement at Home Using A Single RGB Camera

Yagi K., Sugiura Y., Hasegawa K., Saito H.

Gait and Posture （Gait and Posture） 76 136 - 140 2020年02月

ISSN 09666362

© 2019 Elsevier B.V. Background: An individual's gait is a key factor for consideration in evaluating their overall health. Several medical studies have demonstrated the correlation between gait and incidence rate of diseases, mortality, and risk of fall. However, gait is only occasionally evaluated during medical visits, which may delay the detection of health problems. Methods: In this paper, we propose a gait measurement system that is suitable for use at home. Our method requires only a single RGB camera, whereas other visionary sensor-based methods require depth sensors or multiple RGB cameras. In addition, the setup for the measurement is easy. What the user has to do is only putting a single camera in a room and choosing four location known points on the floor. Our method can measure step positions and step timings, and therefore, other important parameters such as stride length, step width, walking speed, and cadence may also be captured. The individual's gait is captured by the camera, and therefore the user is not required to wear any devices. Results: In the experiment described herein, we demonstrate our method's accuracy by comparing it with the motion capture system. The results indicate that our method can measure walking speed with an error of 3.62 cm/s from the side view, and which is too small a change to be clinically meaningful.

Access to Document (DOI)

Rgb-d image inpainting using generative adversarial network with a late fusion approach

Fujii R., Hachiuma R., Saito H.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) （Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)） 12242 LNCS 440 - 451 2020年

ISSN 9783030584641

© Springer Nature Switzerland AG 2020. Diminished reality is a technology that aims to remove objects from video images and fills in the missing region with plausible pixels. Most conventional methods utilize the different cameras that capture the same scene from different viewpoints to allow regions to be removed and restored. In this paper, we propose an RGB-D image inpainting method using generative adversarial network, which does not require multiple cameras. Recently, an RGB image inpainting method has achieved outstanding results by employing a generative adversarial network. However, RGB inpainting methods aim to restore only the texture of the missing region and, therefore, does not recover geometric information (i.e, 3D structure of the scene). We expand conventional image inpainting method to RGB-D image inpainting to jointly restore the texture and geometry of missing regions from a pair of RGB and depth images. Inspired by other tasks that use RGB and depth images (e.g., semantic segmentation and object detection), we propose late fusion approach that exploits the advantage of RGB and depth information each other. The experimental results verify the effectiveness of our proposed method.

Access to Document (DOI)

In-Plane Rotation-Aware Monocular Depth Estimation Using SLAM

Saito Y., Hachiuma R., Yamaguchi M., Saito H.

Communications in Computer and Information Science （Communications in Computer and Information Science） 1212 CCIS 305 - 317 2020年

ISSN 9789811548178

© 2020, Springer Nature Singapore Pte Ltd. Estimating accurate depth from an RGB image in any environment is challenging task in computer vision. Recent learning based method using deep Convolutional Neural Networks (CNNs) have driven plausible appearance, but these conventional methods are not good at estimating scenes that have a pure rotation of camera, such as in-plane rolling. This movement imposes perturbations on learning-based methods because gravity direction is considered to be strong prior to CNN depth estimation (i.e., the top region of an image has a relatively large depth, whereas bottom region tends to have a small depth). To overcome this crucial weakness in depth estimation with CNN, we propose a simple but effective refining method that incorporates in-plane roll alignment using camera poses of monocular Simultaneous Localization and Mapping (SLAM). For the experiment, we used public datasets and also created our own dataset composed of mostly in-plane roll camera movements. Evaluation results on these datasets show the effectiveness of our approach.

Access to Document (DOI)

Global-map-registered local visual odometry using on-the-fly pose graph updates

Yamaguchi M., Mori S., Saito H., Yachida S., Shibata T.

ISSN 9783030584641

© Springer Nature Switzerland AG 2020. Real-time camera pose estimation is one of the indispensable technologies for Augmented Reality (AR). While a large body of work in Visual Odometry (VO) has been proposed for AR, practical challenges such as scale ambiguities and accumulative errors still remain especially when we apply VO to large-scale scenes due to limited hardware and resources. We propose a camera pose registration method, where a local VO is consecutively optimized with respect to a large-scale scene map on the fly. This framework enables the scale estimation between a VO map and a scene map and reduces accumulative errors by finding corresponding locations in the map to the current frame and by on-the-fly pose graph optimization. The results using public datasets demonstrated that our approach reduces the accumulative errors of naïve VO.

Access to Document (DOI)

DetectFusion: Detecting and segmenting both known and unknown dynamic objects in real-time slam

Hachiuma R., Pirchheim C., Schmalstieg D., Saito H.

30th British Machine Vision Conference 2019, BMVC 2019 （30th British Machine Vision Conference 2019, BMVC 2019） 2020年

© 2019. The copyright of this document resides with its authors. We present DetectFusion, an RGB-D SLAM system that runs in real time and can robustly handle semantically known and unknown objects that can move dynamically in the scene. Our system detects, segments and assigns semantic class labels to known objects in the scene, while tracking and reconstructing them even when they move independently in front of the monocular camera. In contrast to related work, we achieve real-time computational performance on semantic instance segmentation with a novel method combining 2D object detection and 3D geometric segmentation. In addition, we propose a method for detecting and segmenting the motion of semantically unknown objects, thus further improving the accuracy of camera tracking and map reconstruction. We show that our method performs on par or better than previous work in terms of localization and object reconstruction accuracy, while achieving about 20 fps even if the objects are segmented in each frame.

Deep Selection: A Fully Supervised Camera Selection Network for Surgery Recordings

Hachiuma R., Shimizu T., Saito H., Kajita H., Takatsume Y.

ISSN 9783030597153

© 2020, Springer Nature Switzerland AG. Recording surgery in operating rooms is an essential task for education and evaluation of medical treatment. However, recording the desired targets, such as the surgery field, surgical tools, or doctor’s hands, is difficult because the targets are heavily occluded during surgery. We use a recording system in which multiple cameras are embedded in the surgical lamp, and we assume that at least one camera is recording the target without occlusion at any given time. As the embedded cameras obtain multiple video sequences, we address the task of selecting the camera with the best view of the surgery. Unlike the conventional method, which selects the camera based on the area size of the surgery field, we propose a deep neural network that predicts the camera selection probability from multiple video sequences by learning the supervision of the expert annotation. We created a dataset in which six different types of plastic surgery are recorded, and we provided the annotation of camera switching. Our experiments show that our approach successfully switched between cameras and outperformed three baseline methods.

Access to Document (DOI)

Surgery recording without occlusions by multi-view surgical videos

Shimizu T., Oishi K., Hachiuma R., Kajita H., Takatsume Y., Saito H.

VISIGRAPP 2020 - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications （VISIGRAPP 2020 - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications） 5 837 - 844 2020年

ISSN 9789897584022

Copyright © 2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved. Recording surgery is important for sharing various operating techniques. In most surgical rooms, fixed surgical cameras are already installed, but it is almost impossible to capture the surgical field because of occlusion by the surgeon’s head and body. In order to capture the surgical field, we propose the installation of multiple cameras in a surgical lamp system, so that at least one camera can capture the surgical field even when the surgeon’s head and body occlude other cameras. In this paper, we present a method for automatic viewpoint switching from multi-view surgical videos, so that the surgical field can always be recorded. We employ a method for learning-based object detection from videos for automatic evaluation of the surgical field from multi-view surgical videos. In general, frequent camera switching degrades the video quality of view (QoV). Therefore, we apply Dijkstra’s algorithm, widely used in the shortest path problem, as an optimization method for this problem. Our camera scheduling method works so that camera switching is not performed for the minimum frame we specified, and therefore the surgical field observed in the entire video is maximized.

MMSports'19: 2<sup>nd</sup> ACM international workshop on multimedia content analysis in sports

Lienhart R., Moeslund T.B., Saito H.

MM 2019 - Proceedings of the 27th ACM International Conference on Multimedia （MM 2019 - Proceedings of the 27th ACM International Conference on Multimedia） 2732 - 2733 2019年10月

ISSN 9781450368896

© 2019 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-6889-6/19/10. The second ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'19) is held in Nice, France on October 25th, 2019 co-located with the ACM International Conference on Multimedia 2019 (ACM Multimedia 2019). The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding and visualizing the multimedia/multimodal data in sports. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation, understanding, statistical analysis and evaluation, and sensor fusion. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports.

Access to Document (DOI)

Prediction of future shot direction using pose and position of tennis player

Shimizu T., Hachiuma R., Saito H., Yoshikawa T., Lee C.

MMSports 2019 - Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports, co-located with MM 2019 （MMSports 2019 - Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports, co-located with MM 2019） 59 - 66 2019年10月

ISSN 9781450369114

© 2019 Copyright held by the owner/author(s). In this paper, we propose a method to predict the future shot direction in a tennis match using pose information and player position. As far as we know, there is no work that deals with such a predictive task, so there is no shot direction dataset as yet. Therefore, using a YouTube tennis match video, we construct an time of impact and shot direction dataset. To reduce annotation costs, we propose a method to automatically label the shot direction. Moreover, we propose a method to predict the future shot direction using the constructed dataset. The shot direction is predicted using LSTM(long short-time memory), from sequential pose information up to the time of impact and the player position. We employ OpenPose to extract the position of skeleton joints. In the experiment, we evaluate the accuracy of shot direction prediction and verify the effectiveness of the proposed method. Since there are no studies that predict future shot direction, we set four baseline methods to evaluate the effectiveness of our proposed method.

Access to Document (DOI)

Volumetric representation of human body parts using superquadrics

Hachiuma R., Saito H.

Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019 （Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019） 187 - 188 2019年10月

ISSN 9781728147659

© 2019 IEEE. Superquadrics are one of the ideal shape representations for adapting various kinds of primitive shapes with a single equation. This paper revisits the task of representing a 3D human body with multiple superquadrics. As a single superquadric surface can only represent symmetric primitive shapes, we present a method that segments the human body into body parts to estimate their superquadric parameters. Moreover, we propose a novel initial parameter estimation method by using 3D skeleton joints. The results show that superquadric parameters are estimated, which represent human body parts volumetrically.

Access to Document (DOI)

Automatic viewpoint switching for multi-view surgical videos

Shimizu T., Oishi K., Saito H., Kajita H., Takatsume Y.

ISSN 9781728147659

© 2019 IEEE. Recording medical surgery operations is important for sharing the various operating techniques. In most operating rooms, fixed surgery cameras are already installed, but it is almost impossible to capture the surgical field because of occlusion by the surgeon's head and body. In order to capture the surgical field in real surgery operations, we proposed the installation of multiple cameras in a surgical lighting system, so that at least one camera can capture the target surgical field even when the surgeon's head and body occlude other cameras. In this paper, we present a method for automatic viewpoint selection from multi-view surgical videos, so that the surgical field can always be recorded in the output video. We employ a method for learning-based object detection from videos for automatic evaluation of the surgical field area from multiple input videos. By selecting the viewpoint with the largest area of the surgical field, we can virtually reduce the area of the surgeon's head and hands. In general, frequent camera switching degrades the video quality of view (QoV). Therefore, we apply the Dijkstra method widely used in the shortest path problem as a combinatorial optimization method for this problem. Our camera scheduling method is that the camera switching is not performed for a certain period of time, and the surgical field observed in the entire video is maximized.

Access to Document (DOI)

Joint inpainting of RGB and depth images by generative adversarial network with a late fusion approach

Fujii R., Hachiuma R., Saito H.

ISSN 9781728147659

© 2019 IEEE. Image inpainting aims to restore texture of missing regions in scene from an RGB image. In this paper, we aim to restore not only the texture but also the geometry of the missing regions in scene from a pair of RGB and depth images. Inspired by the recent development of generative adversarial network, we employ an encoder-decoderbased generative adversarial network with the input of RGB and depth image. The experimental results show that our method restores the missing region of both RGB and depth image.

Access to Document (DOI)

Incremental class discovery for semantic segmentation with rgbd sensing

Nakajima Y., Kang B., Saito H., Kitani K.

Proceedings of the IEEE International Conference on Computer Vision （Proceedings of the IEEE International Conference on Computer Vision） 2019-October 972 - 981 2019年10月

ISSN 9781728148038

© 2019 IEEE. This work addresses the task of open world semantic segmentation using RGBD sensing to discover new semantic classes over time. Although there are many types of objects in the real-word, current semantic segmentation methods make a closed world assumption and are trained only to segment a limited number of object classes. Towards a more open world approach, we propose a novel method that incrementally learns new classes for image segmentation. The proposed system first segments each RGBD frame using both color and geometric information, and then aggregates that information to build a single segmented dense 3D map of the environment. The segmented 3D map representation is a key component of our approach as it is used to discover new object classes by identifying coherent regions in the 3D map that have no semantic label. The use of coherent region in the 3D map as a primitive element, rather than traditional elements such as surfels or voxels, also significantly reduces the computational complexity and memory use of our method. It thus leads to semi-real-time performance at 10.7 Hz when incrementally updating the dense 3D map at every frame. Through experiments on the NYUDv2 dataset, we demonstrate that the proposed method is able to correctly cluster objects of both known and unseen classes. We also show the quantitative comparison with the state-of-the-art supervised methods, the processing time of each step, and the influences of each component.

Access to Document (DOI)

Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction

Maxence R., Uchiyama H., Kawasaki H., Thomas D., Nozick V., Saito H.

Proceedings - 2019 International Conference on 3D Vision, 3DV 2019 （Proceedings - 2019 International Conference on 3D Vision, 3DV 2019） 574 - 582 2019年09月

ISSN 9781728131313

© 2019 IEEE. The standard photometric stereo is a technique to densely reconstruct objects' surfaces using light variation under the assumption of a static camera with a moving light source. In this work, we use photometric stereo to reconstruct dense 3D scenes while moving the camera and the light altogether. In such non-static case, camera poses as well as correspondences between pixels of each frame to apply photometric stereo are required. ORB-SLAM is a technique that can be used to estimate camera poses. To retrieve correspondences, our idea is to start from a sparse 3D mesh obtained with ORB SLAM and then densify the mesh by a plane sweep method using a multi-view photometric consistency. By combining ORB-SLAM and photometric stereo, it is possible to reconstruct dense 3D scenes with a off-the-shelf smartphone and its embedded torchlight. Note that SLAM systems usually struggle with textureless object, which is effectively compensated by the photometric stereo in our method. Experiments are conducted to show that our proposed method gives better results than SLAM alone or COLMAP, especially for partially textureless surfaces.

Access to Document (DOI)

　前のページ - 次のページ　

斎藤 英雄

論文 - 斎藤 英雄

斎藤　英雄

論文 - 斎藤　英雄