精华内容
下载资源
问答
  • 最近总结了基于视觉的机器人抓取的相关论文及代码,同步于github。根据抓取的类别,基于视觉的机器人抓取可以分为两类:2D平面抓取以及6D空间抓取。这个页面总结了近年来涉及到的各种各样的方法,其中大部分都使用了...

    本文同步于微信公众号:3D视觉前沿,欢迎大家关注。

    本文总结了基于视觉的机器人抓取的相关论文及代码,同步于 GitHub

    机器人抓取必需的信息是相机系下抓取器的6DoF位姿,包括抓取器的3D位置和抓取器的3D空间朝向。通过控制机械臂移动使抓取器到该位置和旋转,就可以执行抓取操作。基于视觉的机器人抓取,是指给机器人安装RGB-D相机,通过人工智能算法,获得抓取器的目标抓取位姿。按照不同的抓取方式,可以分为 2D平面抓取6D空间抓取

    2D平面抓取 是指目标物体放置在水平工作台上,抓取器只能从一个方向进行抓取。由于存在这样的限制,抓取器的6D位姿简化为3D,包含平面内的2D位置和平面内的1D旋转角度。这类方法存在两类,一类通过 评估抓取接触点的质量,一类 评估带朝向的抓取四边形

    6D空间抓取 是指抓取器可以在3D空间从各个角度抓取目标物体,此时抓取器的6D位姿不能简化。按照依赖物体的完整形状还是物体的部分点云,可以将方法分为基于部分点云的方法和基于完整形状的方法两类。基于部分点云的方法 包括评估候选抓取位姿的方法和从已有抓取库中迁移抓取的方法。基于完整形状的方法 包括评估物体6D位姿的方法和基于形状补全的方法。当前大多数6D空间抓取方法都是针对已知3D模型的物体,这些物体的最优抓取位置可以通过人工指定或者仿真预先得到,此时,问题转化为估计物体的6D位姿。

    另外,当前大部分机器人抓取方法需要先从输入数据中获得目标物体的位置,这可以分为三个层次,物体定位但不识别、物体检测、物体实例分割。物体定位但不识别是指获得目标物体的2D/3D范围但是不知道物体的类别;目标检测是指得到目标物体的2D/3D包围盒,同时识别目标物体的类别;目标实例分割提供目标物体所占有的像素或者点级别的区域信息,同时识别目标物体的类别。

    在这里对以上涉及的所有技术进行了归类整理,有传统方法和深度学习的方法,有基于2D和基于3D的方法,也包含一些相关的技术例如3D重建、形状补全、深度图估计、数据生成、灵巧手、强化学习等等,列出了最新的论文及链接,并且保持每周更新,希望能够对这个领域内的朋友们有帮助。总结的中文框架结构如下:

    0. 综述文章
    1. 物体定位

    1.1 定位不识别-包括基于2D/3D数据,拟合形状基元或检测显著性区域
    1.2 目标检测-包括基于2D/3D数据,进行两阶段和单阶段目标检测
    1.3 目标实例分割-包括基于2D/3D数据,进行两阶段和单阶段目标实例分割

    2. 物体6D位姿估计

    2.1 基于RGB-D图像的方法-包括基于对应、模板或者投票的方法
    2.2 基于点云的方法-包括基于对应、模板或者投票的方法
    2.3 类别级位姿估计方法-包括类别级方法、基于图像的3D重建以及3D渲染

    3. 2D平面抓取

    3.1 估计抓取接触点的方法
    3.2 估计带朝向四边形的方法

    4. 6D空间抓取

    4.1 基于单个视角点云的方法-包括评估候选抓取位姿的方法和迁移抓取经验的方法
    4.2 基于完整形状的方法-包括估计物体6D位姿的方法和基于形状补全的方法

    5.任务导向的抓取-包括任务导向的抓取、抓取支撑以及3D部件分割
    6.灵巧手
    7.数据生成-包括虚拟到真实,以及自监督方法
    8.多源信息
    9.动作规划-包括视觉伺服和路径规划
    10.模仿学习
    11.强化学习
    12.领域专家

    具体相关论文如下:

    Vision-based Robotic Grasping: Papers and Codes

    0. Review Papers

    [arXiv] 2020-Affordances in Robotic Tasks - A Survey, [paper]

    [arXiv] 2019-A Review of Robot Learning for Manipulation- Challenges, Representations, and Algorithms, [paper]

    [arXiv] 2019-Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review, [paper]

    [MTI] 2018-Review of Deep Learning Methods in Robotic Grasp Detection, [paper]

    [ToR] 2016-Data-Driven Grasp Synthesis - A Survey, [paper]

    [RAS] 2012-An overview of 3D object grasp synthesis algorithms - A Survey, [paper]

    1. Object Localization

    1.1 Object Localization without Classification

    1.1.1 2D-based Methods

    a.Fitting 2D Shape Primitives

    [BMVC] A buyer’s guide to conic fitting, [paper] [code]

    [IJGIG] Algorithms for the reduction of the number of points required to represent a digitized line or its caricature, [paper] [code]

    b. Saliency Detection

    Survey papers:

    [arXiv] 2019-Salient object detection in the deep learning era: An in-depth survey, [paper]

    [CVM] 2014-Salient object detection: A survey, [paper]

    2020:

    [arXiv] UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders, [paper]

    [arXiv] Cross-layer Feature Pyramid Network for Salient Object Detection, [paper]

    [arXiv] Depth Potentiality-Aware Gated Attention Network for RGB-D Salient Object Detection, [paper]

    [arXiv] Weakly-Supervised Salient Object Detection via Scribble Annotations, [paper]

    [arXiv] Highly Efficient Salient Object Detection with 100K Parameters, [paper]

    [arXiv] Global Context-Aware Progressive Aggregation Network for Salient Object Detection, [paper]

    [arXiv] Adaptive Graph Convolutional Network with Attention Graph Clustering for Co-saliency Detection, [paper]

    2019:

    [ICCV] Employing deep part-object relationships for salient object detection, [paper]

    [ICME] Multi-scale capsule attention-based salient object detection with multi-crossed layer connections, [paper]

    2018:

    [CVPR] Picanet: Learning pixel-wise contextual attention for saliency detection, [paper]

    [SPM] Advanced deep-learning techniques for salient and category-specific object detection: a survey, [paper]

    2017:

    [CVPR] Deeply supervised salient object detection with short connections, [paper]

    [TOC] Video saliency detection using object proposals, [paper]

    2016:

    [CVPR] Unconstrained salient object detection via proposal subset optimization, [paper]

    [CVPR] Deep hierarchical saliency network for salient object detection, [paper]

    [TPAMI] Salient object detection via structured matrix decomposition, [paper]

    [TIP] Correspondence driven saliency transfer, [paper]

    2015:

    [CVPR] Saliency detection by multi-context deep learning, [paper]

    [TPAMI] Hierarchical image saliency detection on extended CSSD, [paper]

    2014:

    [CVPR] Saliency optimization from robust background detection, [paper]

    [TPAMI] Global contrast based salient region detection, [paper]

    2013:

    [CVPR] Salient object detection: A discriminative regional feature integration approach, [paper]

    [CVPR] Saliency detection via graph-based manifold ranking, [paper]

    2012:

    [ECCV] Geodesic saliency using background priors, [paper]

    1.1.2 3D-based Methods

    a.Fitting 3D Shape Primitives

    Survey papers:

    [CGF] 2019-A survey of simple geometric primitives detection methods for captured 3d data, [paper]

    2020:

    [arXiv] ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds, [paper]

    2015:

    [CVPR] Separating objects and clutter in indoor scenes, [paper]

    2013:

    [CVPR] A linear approach to matching cuboids in rgbd images, [paper]

    2012:

    [GCR] Robustly segmenting cylindrical and box-like objects in cluttered scenes using depth cameras, [paper]

    2009:

    [IROS] Close-range scene segmentation and reconstruction of 3d point cloud maps for mobile manipulation in domestic environments, [paper]

    2005:

    [ISPRS] Efficient hough transform for automatic detection of cylinders in point clouds, [paper]

    1981:

    [COM] Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, [paper]

    b. Saliency Detection

    2019:

    [PR] Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection, [paper]

    [ICCV] Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection, [paper]

    [ICCV] Pointcloud saliency maps, [paper]

    [arXiv] CNN-based RGB-D Salient Object Detection: Learn, Select and Fuse, [paper]

    2018:

    [CVPR] Progressively complementarity-aware fusion network for RGB-D salient object detection, [paper]

    2017:

    [TIP] RGBD salient object detection via deep fusion, [paper]

    2015:

    [CVPRW] Exploiting global priors for RGB-D saliency detection, [paper]

    2014:

    [ECCV] Rgbd salient object detection: a benchmark and algorithms, [paper]

    2013:

    [JSIP] Segmenting salient objects in 3d point clouds of indoor scenes using geodesic distances, [paper]

    2008:

    [WACV] Segmentation of salient regions in outdoor scenes using imagery and 3-d data, [paper]

    1.2 Object Detection

    Detailed paper lists can refer to hoya012 or amusi.

    1.2.1 2D Object Detection

    Survey papers:

    2020:

    [arXiv] Deep Domain Adaptive Object Detection: a Survey, [paper]

    [IJCV] Deep Learning for Generic Object Detection: A Survey, [paper]

    2019:

    [arXiv] Object Detection in 20 Years A Survey, [paper]

    [arXiv] Object Detection with Deep Learning: A Review, [paper]

    [arXiv] A Review of Object Detection Models based on Convolutional Neural Network, [paper]

    [arXiv] A Review of methods for Textureless Object Recognition, [paper]

    a. Two-stage methods

    2020:

    [arXiv] Instance-Aware, Context-Focused, and Memory-Efficient Weakly Supervised Object Detection, [paper]

    [arXiv] Scalable Active Learning for Object Detection, [paper]

    [arXiv] Any-Shot Object Detection, [paper]

    [arXiv] Frustratingly Simple Few-Shot Object Detection, [paper]

    [arXiv] Rethinking the Route Towards Weakly Supervised Object Localization, [paper]

    [arXiv] Universal-RCNN: Universal Object Detector via Transferable Graph R-CNN, [paper]

    [arXiv] Unsupervised Image-generation Enhanced Adaptation for Object Detection in Thermal images, [paper]

    [arXiv] PCSGAN: Perceptual Cyclic-Synthesized Generative Adversarial Networks for Thermal and NIR to Visible Image Transformation, [paper]

    [arXiv] SpotNet: Self-Attention Multi-Task Network for Object Detection, [paper]

    [arXiv] Real-Time Object Detection and Recognition on Low-Compute Humanoid Robots using Deep Learning, [paper]

    [arXiv] FedVision: An Online Visual Object Detection Platform Powered by Federated Learning, [paper]

    2019:

    [arXiv] Combining Deep Learning and Verification for Precise Object Instance Detection, [paper]

    [arXiv] cmSalGAN: RGB-D Salient Object Detection with Cross-View Generative Adversarial Networks, [paper]

    [arXiv] OpenLORIS-Object: A Dataset and Benchmark towards Lifelong Object Recognition, [paper] [project]

    [IROS] Look Further to Recognize Better: Learning Shared Topics and Category-Specific Dictionaries for Open-Ended 3D Object Recognition, [paper]

    [IROS] Recurrent Convolutional Fusion for RGB-D Object Recognition, [paper] [code]

    [ICCVW] An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Detection, [paper]

    2017:

    [CVPR] FPN: Feature pyramid networks for object detection, [paper]

    [arXiv] Light-Head R-CNN: In Defense of Two-Stage Object Detector, [paper] [code]

    2016:

    [NeurIPS] R-FCN: Object Detection via Region-based Fully Convolutional Networks, [paper] [code]

    [TPAMI] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, [paper] [code]

    [ECCV] Visual relationship detection with language priors, [paper]

    2015:

    [ICCV] Fast R-CNN, [paper] [code]

    2014:

    [ECCV] SPPNet: Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition, [paper] [code]

    [CVPR] R-CNN: Rich feature hierarchies for accurate object detection and semantic segmentation, [paper] [code]

    [CVPR] Scalable object detection using deep neural networks, [paper]

    [arXiv] Scalable, high-quality object detection, [paper]

    [ICLR] OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks, [paper] [code]

    2011:

    [ICCV] ORB: An efficient alternative to SIFT or SURF, [paper]

    2006:

    [ECCV] SURF: Speeded up robust features, [paper]

    2005:

    [ICCV] FAST: Fusing points and lines for high performance tracking, [paper]

    1999:

    [ICCV] SIFT: Object Recognition from Local Scale-Invariant Features, [paper]

    b. Single-stage methods

    2020:

    [arXiv] SaccadeNet: A Fast and Accurate Object Detector, [paper]

    [arXiv] CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection, [paper]

    [arXiv] Extended Feature Pyramid Network for Small Object Detection, [paper]

    [arXiv] Real Time Detection of Small Objects, [paper]

    [arXiv] OS2D: One-Stage One-Shot Object Detection by Matching Anchor Features, [paper]

    2019:

    [arXiv] CenterNet: Objects as Points, [paper]

    [arXiv] CenterNet: Keypoint Triplets for Object Detection, [paper]

    [ECCV] CornerNet: Detecting Objects as Paired Keypoints, [paper]

    [arXiv] FCOS: Fully Convolutional One-Stage Object Detection, [paper]

    [arXiv] Bottom-up Object Detection by Grouping Extreme and Center Points, [paper]

    2018:

    [arXiv] YOLOv3: An Incremental Improvement, [paper] [code]

    2017:

    [CVPR] YOLO9000: Better, Faster, Stronger, [paper] [code]

    [ICCV] RetinaNet: Focal loss for dense object detection, [paper]

    2016:

    [CVPR] YOLO: You only look once: Unified, real-time object detection, [paper] [code]

    [ECCV] SSD: Single Shot MultiBox Detector, [paper] [code]


    Dataset:

    PASCAL VOC: The PASCAL Visual Object Classes (VOC) Challenge, [paper]

    ILSVRC: ImageNet large scale visual recognition challenge, [paper]

    Microsoft COCO: Common Objects in Context, is a large-scale object detection, segmentation, and captioning dataset, [paper]

    Open Images: a collaborative release of ~9 million images annotated with labels spanning thousands of object categories, [paper]

    1.2.2 3D Object Detection

    This kind of methods can be divided into three kinds: RGB-based methods, point cloud-based methods, and fusion methods which consume images and point cloud. Most of these works are focus on autonomous driving.

    a. RGB-based methods

    Most of this kind of methods estimate depth images from RGB images, and then conduct 3D detection.

    2020:

    [arXiv] Disp R-CNN: Stereo 3D Object Detection via Shape Prior Guided Instance Disparity Estimation, [paper]

    [arXiv] End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection, [paper]

    [arXiv] Confidence Guided Stereo 3D Object Detection with Split Depth Estimation, [paper]

    [arXiv] Monocular 3D Object Detection in Cylindrical Images from Fisheye Cameras, [paper]

    [arXiv] ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection, [paper]

    [arXiv] MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships, [paper]

    [arXiv] Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image, [paper]

    [arXiv] SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation, [paper]

    [arXiv] siaNMS: Non-Maximum Suppression with Siamese Networks for Multi-Camera 3D Object Detection, [paper]

    [AAAI] Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation, [paper]

    [arXiv] SDOD: Real-time Segmenting and Detecting 3D Objects by Depth, [paper]

    [arXiv] DSGN: Deep Stereo Geometry Network for 3D Object Detection, [paper]

    [arXiv] RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving, [paper]

    2019:

    [NeurIPS] PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points, [paper]

    [arXiv] Single-Stage Monocular 3D Object Detection with Virtual Cameras, [paper]

    [arXiv] Environment reconstruction on depth images using Generative Adversarial Networks, [paper] [code]

    [arXiv] Learning Depth-Guided Convolutions for Monocular 3D Object Detection, [paper]

    [arXiv] RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Driving, [paper]

    [IROS] Look Further to Recognize Better: Learning Shared Topics and Category-Specific Dictionaries for Open-Ended 3D Object Recognition, [paper]

    [arXiv] Task-Aware Monocular Depth Estimation for 3D Object Detection, [paper]

    [CVPR] Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving, [paper] [code]

    [AAAI] MonoGRNet: A Geometric Reasoning Network for 3D Object Localization, [paper] [code]

    [ICCV] Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving, [paper]

    [ICCV] M3D-RPN: Monocular 3D Region Proposal Network for Object Detection, [paper]

    [ICCVW] Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud, [paper]

    [arXiv] Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Loss, [paper]

    [arXiv] Monocular 3D Object Detection via Geometric Reasoning on Keypoints, [paper]

    b. Point cloud-based methods

    This kind of methods only consume the 3D point cloud data.

    Survey papers:

    [arXiv] 2019-Deep Learning for 3D Point Clouds: A Survey, [paper]

    2020:

    [arXiv] MLCVNet: Multi-Level Context VoteNet for 3D Object Detection, [paper]

    [arXiv] 3D IoU-Net: IoU Guided 3D Object Detector for Point Clouds, [paper]

    [arXiv] Finding Your (3D) Center: 3D Object Detection Using a Learned Loss, [paper]

    [arXiv] LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention, [paper]

    [arXiv] Quantifying Data Augmentation for LiDAR based 3D Object Detection, [paper]

    [arXiv] DOPS: Learning to Detect 3D Objects and Predict their 3D Shapes, [paper]

    [arXiv] Improving 3D Object Detection through Progressive Population Based Augmentation, [paper]

    [arXiv] Boundary-Aware Dense Feature Indicator for Single-Stage 3D Object Detection from Point Clouds, [paper]

    [arXiv] Physically Realizable Adversarial Examples for LiDAR Object Detection, [paper]

    [arXiv] Real-time 3D object proposal generation and classification under limited processing resources, [paper]

    [arXiv] 3D Object Detection From LiDAR Data Using Distance Dependent Feature Extraction, [paper]

    [arXiv] HVNet: Hybrid Voxel Network for LiDAR Based 3D Object Detection, [paper]

    [arXiv] Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud, [paper]

    [arXiv] PointTrackNet: An End-to-End Network for 3-D Object Detection and Tracking from Point Clouds, [paper]

    [arXiv] 3DSSD: Point-based 3D Single Stage Object Detector, [paper]

    [ariv] SegVoxelNet: Exploring Semantic Context and Depth-aware Features for 3D Vehicle Detection from Point Cloud, [paper]

    [arXiv] Investigating the Importance of Shape Features, Color Constancy, Color Spaces and Similarity Measures in Open-Ended 3D Object Recognition, [paper]

    [arXiv] Probabilistic 3D Multi-Object Tracking for Autonomous Driving, [paper]

    [AAAI] TANet: Robust 3D Object Detection from Point Clouds with Triple Attention, [paper]

    2019:

    [arXiv] Class-balanced grouping and sampling for point cloud 3d object detection, [paper] [code]

    [arXiv] SESS: Self-Ensembling Semi-Supervised 3D Object Detection, [paper]

    [arXiv] Deep SCNN-based Real-time Object Detection for Self-driving Vehicles Using LiDAR Temporal Data, [paper]

    [arXiv] Pillar in Pillar: Multi-Scale and Dynamic Feature Extraction for 3D Object Detection in Point Clouds, [paper]

    [arXiv] What You See is What You Get: Exploiting Visibility for 3D Object Detection, [paper]

    [NeurIPSW] Patch Refinement – Localized 3D Object Detection, [paper]

    [CoRL] End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds, [paper]

    [ICCV] Deep Hough Voting for 3D Object Detection in Point Clouds, [paper] [code]

    [arXiv] Part-A2 Net: 3D Part-Aware and Aggregation Neural Network for Object Detection from Point Cloud, [paper]

    [ICCV] STD: Sparse-to-Dense 3D Object Detector for Point Cloud, [paper]

    [CVPR] PointPillars: Fast Encoders for Object Detection from Point Clouds, [paper]

    [arXiv] StarNet: Targeted Computation for Object Detection in Point Clouds, [paper]

    2018:

    [CVPR] PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud, [paper] [code]

    [CVPR] PIXOR: Real-time 3D Object Detection from Point Clouds, [paper] [code]

    [ECCVW] Complex-YOLO: Real-time 3D Object Detection on Point Clouds, [paper] [code]

    [ECCVW] YOLO3D: End-to-end real-time 3D Oriented Object Bounding Box Detection from LiDAR Point Cloud, [paper]

    c. Fusion methods

    This kind of methods utilize both rgb images and depth images/point clouds. There exist early fusion methods, late fusion methods, and dense fusion methods.

    2020:

    [arXiv] ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes, [paper]

    [arXiv] JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset, [paper]

    [AAAI] PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Module, [paper]

    2019:

    [arXiv] PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection, [paper]

    [arXiv] Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots, [paper]

    [arXiv] ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language, [paper]

    [arXiv] Relation Graph Network for 3D Object Detection in Point Clouds, [paper]

    [arXiv] PointPainting: Sequential Fusion for 3D Object Detection, [paper]

    [ICCV] Transferable Semi-Supervised 3D Object Detection From RGB-D Data, [paper]

    [arXiv] Adaptive and Azimuth-Aware Fusion Network of Multimodal Local Features for 3D Object Detection, [paper]

    [arXiv] Frustum VoxNet for 3D object detection from RGB-D or Depth images, [paper]

    [IROS] Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection, [paper]

    [CVPR] Multi-Task Multi-Sensor Fusion for 3D Object Detection, [paper]

    2018:

    [CVPR] Frustum PointNets for 3D Object Detection from RGB-D Data, [paper] [code]

    [ECCV] Deep Continuous Fusion for Multi-Sensor 3D Object Detection, [paper]

    [IROS] Joint 3D Proposal Generation and Object Detection from View Aggregation, [paper] [code]

    [CVPR] PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation, [paper]

    [ICRA] A General Pipeline for 3D Detection of Vehicles, [paper]

    2017:
    [CVPR] Multi-View 3D Object Detection Network for Autonomous Driving, [paper] [code]

    2016:

    [CVPR] Deep sliding shapes for amodal 3d object detection in rgb-d images, [paper]

    1.3 Object Instance Segmentation

    1.3.1 2D Instance Segmentation

    a. Survey papers

    2020:

    [arXiv] Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey, [paper]

    [arXiv] Image Segmentation Using Deep Learning: A Survey, [paper]

    b. Two-stage methods

    2020:

    [arXiv] 1st Place Solutions for OpenImage2019 - Object Detection and Instance Segmentation, [paper]

    [arXiv] Fully Convolutional Networks for Automatically Generating Image Masks to Train Mask R-CNN, [paper]

    [arXiv] FGN: Fully Guided Network for Few-Shot Instance Segmentation, [paper]

    [arXiv] PointRend: Image Segmentation as Rendering, [paper]

    2019:

    [CVPR] HTC: Hybrid task cascade for instance segmentation, [paper]

    2018:

    [CVPR] PANet: Path aggregation network for instance segmentation, [paper]

    [CVPR] Masklab: Instance segmentation by refining object detection with semantic and direction features, [paper]

    2017:

    [ICCV] Mask r-cnn, [paper] [code]

    2016:

    [CVPR] Instance-aware semantic segmentation via multi-task network cascades, [paper]

    2014:

    [ECCV] Simultaneous detection and segmentation, [paper]

    c. One-stage methods

    2020:

    [CVPR] CenterMask: single shot instance segmentation with point representation, [paper]

    [arXiv] BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation, [paper]

    [arXiv] SOLOv2: Dynamic, Faster and Stronger, [paper] [code]

    [arXiv] Mask Encoding for Single Shot Instance Segmentation, [paper]

    [arXiv] Deep Affinity Net: Instance Segmentation via Affinity, [paper]

    [arXiv] PointINS: Point-based Instance Segmentation, [paper]

    [arXiv] Conditional Convolutions for Instance Segmentation, [paper]

    [arXiv] Real-time Semantic Background Subtraction, [paper]

    [arXiv] FourierNet: Compact mask representation for instance segmentation using differentiable shape decoders, [paper]

    2019:

    [arXiv] CenterMask:Real-Time Anchor-Free Instance Segmentation, [paper] [code]

    [arXiv] SAIS: Single-stage Anchor-free Instance Segmentation, [paper]

    [arXiv] YOLACT++ Better Real-time Instance Segmentation, [paper] [code]

    [ICCV] YOLACT: Real-time Instance Segmentation, [paper] [code]

    [ICCV] TensorMask: A Foundation for Dense Object Segmentation, [paper] [code]

    [CASE] Deep Workpiece Region Segmentation for Bin Picking, [paper]

    2018:

    [CVPR] PANet: Path Aggregation Network for Instance Segmentation, [paper] [code]

    [CVPR] MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features, [paper]

    2017:

    [CVPR] Fully Convolutional Instance-aware Semantic Segmentation, [paper]

    2016:

    [ECCV] SharpMask: Learning to Refine Object Segments, [paper] [code]

    [BMVC] MultiPathNet: A MultiPath Network for Object Detection, [paper] [code]

    [CVPR] MNC: Instance-aware Semantic Segmentation via Multi-task Network Cascades, [paper]

    2015:

    [NeurIPS] DeepMask: Learning to Segment Object Candidates, [paper] [code]

    [CVPR] Hypercolumns for Object Segmentation and Fine-grained Localization, [paper]

    2014:

    [ECCV] SDS: Simultaneous Detection and Segmentation, [paper]

    Applications in Robotics:

    2020:

    [arXiv] Self-Supervised Object-in-Gripper Segmentation from Robotic Motions, [paper]

    [arXiv] Segmenting unseen industrial components in a heavy clutter using rgb-d fusion and synthetic data, [paper]

    [arXiv] Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects, [paper]

    [arXiv] Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter, [paper]

    d. Panoptic segmentation

    2020:

    [arXiv] BANet: Bidirectional Aggregation Network with Occlusion Handling for Panoptic Segmentation, [paper]

    [arXiv] Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation, [paper]

    [arXiv] Towards Bounding-Box Free Panoptic Segmentation, [paper]

    2019:

    [CVPR] An End-to-End Network for Panoptic Segmentation, [paper]

    [CVPR] Panoptic Segmentation, [paper]

    [CVPR] Panoptic Feature Pyramid Networks, [paper]

    [CVPR] UPSNet: A Unified Panoptic Segmentation Network, [paper]

    [IV] Single Network Panoptic Segmentation for Street Scene Understanding, [paper] [code]

    [ITSC] Multi-task Network for Panoptic Segmentation in Automated Driving, [paper]

    1.3.2 3D Instance Segmentation

    a. Two-stage methods

    2020:

    [arXiv] PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation, [paper]

    [arXiv] 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation, [paper]

    [arXiv] OccuSeg: Occupancy-aware 3D Instance Segmentation, [paper]

    [arXiv] Learning to Segment 3D Point Clouds in 2D Image Space, [paper]

    [arXiv] Bi-Directional Attention for Joint Instance and Semantic Segmentation in Point Clouds, [paper]

    [arXiv] 3DCFS: Fast and Robust Joint 3D Semantic-Instance Segmentation via Coupled Feature Selection, [paper]

    [RAL] From Planes to Corners: Multi-Purpose Primitive Detection in Unorganized 3D Point Clouds, [paper]

    [arXiv] Learning and Memorizing Representative Prototypes for 3D Point Cloud Semantic and Instance Segmentation, [paper]

    [WACV] FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data, [paper]

    2019:

    [arXiv] Point2Node: Correlation Learning of Dynamic-Node for Point Cloud Feature Modeling, [paper]

    [arXiv] LatticeNet: Fast Point Cloud Segmentation Using Permutohedral Lattices, [paper]

    [arXiv] Learning to Optimally Segment Point Clouds, [paper]

    [arXiv] Point Cloud Instance Segmentation using Probabilistic Embeddings, [paper]

    [NeurIPS] Exploiting Local and Global Structure for Point Cloud Semantic Segmentation with Contextual Point Representations, [paper]

    [arXiv] Addressing the Sim2Real Gap in Robotic 3D Object Classification, [paper]

    [IROS] LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images, [paper]

    [CoRL] The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation, [paper] [code]

    [arXiv] GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud, [paper]

    b. One-stage Methods

    2020:

    [AAAI] JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds, [paper] [code]

    [ICRA] LiDARSeg: Instance segmentation of lidar point clouds, [paper]

    2019:

    [NeurIPS] 3D-BoNet: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds, [paper] [code]

    [arXiv] MASC: multi-scale affinity with sparse convolution for 3d instance segmentation, [paper]

    [CVPR] ASIS: Associatively segmenting instances and semantics in point clouds, [paper]

    [CVPR] SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation, [paper]

    [CVPR] JSIS3D: joint semantic-instance segmentation of 3d point clouds with multi-task pointwise networks and multi-value conditional random fields, [paper]

    c. 3D deep learning networks

    2020:

    [arXiv] LightConvPoint: convolution for points, [paper]

    [arXiv] Review: deep learning on 3D point clouds, [paper]

    [arXiv] Improving Semantic Analysis on Point Clouds via Auxiliary Supervision of Local Geometric Priors, [paper]

    2019:

    [arXiv] QUATERNION EQUIVARIANT CAPSULE NETWORKS FOR 3D POINT CLOUDS, [paper]

    [arXiv] Geometry Sharing Network for 3D Point Cloud Classification and Segmentation, [paper]

    [arXiv] Geometric Capsule Autoencoders for 3D Point Clouds, [paper]

    [arXiv] Utility Analysis of Network Architectures for 3D Point Cloud Processing, [paper]

    [arXiv] Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research, [paper] [code]

    [ICCV] DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing, [paper] [code]

    [TOG] Dynamic Graph CNN for Learning on Point Clouds, [paper] [code]

    [ICCV] DeepGCNs: Can GCNs Go as Deep as CNNs?, [paper] [code]

    [ICCV] KPConv: Flexible and Deformable Convolution for Point Clouds, [paper] [code]

    [MM] SRINet: Learning Strictly Rotation-Invariant Representations for Point Cloud Classification and Segmentation, [paper]

    [CVPR] PointConv: Deep Convolutional Networks on 3D Point Clouds, [paper] [code]

    [CVPR] PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing, [paper] [code]

    [CVPR] Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN, [paper] [code]

    [arXiv] SAWNet: A Spatially Aware Deep Neural Network for 3D Point Cloud Processing, [paper]

    [arXiv] PyramNet: Point Cloud Pyramid Attention Network and Graph Embedding Module for Classification and Segmentation, [paper]

    [ICCV] Interpolated Convolutional Networks for 3D Point Cloud Understanding, [paper]

    [arXiv] A survey on Deep Learning Advances on Different 3D Data Representations, [paper]

    2018:

    [TOG] MCCNN: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds, [paper] [code]

    [NeurIPS] PointCNN: Convolution On X-Transformed Points, [paper] [code]

    [CVPR] Mining Point Cloud Local Structures by Kernel Correlation and Graph Pooling, [paper] [code]

    [CVPR] SO-Net: Self-Organizing Network for Point Cloud Analysis, [paper] [code]

    [CVPR] SPLATNet: Sparse Lattice Networks for Point Cloud Processing, [paper] [code]

    [arXiv] Point Convolutional Neural Networks by Extension Operators, [paper]

    2017:

    [ICCV] Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models, [paper] [code]

    [CVPR] PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, [paper] [code]

    [NeurIPS] PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, [paper] [code]

    [CVPR] SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation, [paper]

    2. Object Pose Estimation

    This part mainly discuss 6D object pose estimation methods, which can be categorized into RGB-D image-based methods and point cloud-based methods. RGB-D image-based methods mainly utilized the 2D RGB image and the 2.5D Depth image. Point cloud-based methods utilize registration-based methods.

    2.1 RGB-D Image-based Methods

    Survey papers:

    2020:

    [arXiv] A Review on Object Pose Recovery: from 3D Bounding Box Detectors to Full 6D Pose Estimators, [paper]

    2016:

    [ECCVW] A Summary of the 4th International Workshop on Recovering 6D Object Pose, [paper]

    2.1.1 Correspondence-based Methods

    a. Match 2D feature points

    2020:

    [arXiv] S2DNet: Learning Accurate Correspondences for Sparse-to-Dense Feature Matching, [paper]

    [arXiv] SK-Net: Deep Learning on Point Cloud via End-to-end Discovery of Spatial Keypoints, [paper]

    [arXiv] LRC-Net: Learning Discriminative Features on Point Clouds by Encoding Local Region Contexts, [paper]

    [arXiv] Table-Top Scene Analysis Using Knowledge-Supervised MCMC, [paper]

    [arXiv] AprilTags 3D: Dynamic Fiducial Markers for Robust Pose Estimation in Highly Reflective Environments and Indirect Communication in Swarm Robotics, [paper]

    [AAAI] LCD: Learned Cross-Domain Descriptors for 2D-3D Matching, [paper] [project]

    2019:

    [ICCV] GLAMpoints: Greedily Learned Accurate Match points, [paper]

    2016:

    [ECCV] LIFT: Learned Invariant Feature Transform, [paper]

    2012:

    [3DIMPVT] 3D Object Detection and Localization using Multimodal Point Pair Features, [paper]

    b. Regress 2D projections

    2020:

    [arXiv] EPOS: Estimating 6D Pose of Objects with Symmetries, [paper]

    [arXiv] Tackling Two Challenges of 6D Object Pose Estimation: Lack of Real Annotated RGB Images and Scalability to Number of Objects, [paper]

    [arXiv] Squeezed Deep 6DoF Object Detection using Knowledge Distillation, [paper]

    [arXiv] Learning 2D–3D Correspondences To Solve The Blind Perspective-n-Point Problem, [paper]

    [arXiv] PnP-Net: A hybrid Perspective-n-Point Network, [paper]

    [arXiv] Object 6D Pose Estimation with Non-local Attention, [paper]

    [arXiv] 6DoF Object Pose Estimation via Differentiable Proxy Voting Loss, [paper]

    2019:

    [arXiv] DPOD: 6D Pose Object Detector and Refiner, [paper]

    [CVPR] Segmentation-driven 6D Object Pose Estimation, [paper]

    [arXiv] Single-Stage 6D Object Pose Estimation, [paper]

    [arXiv] W-PoseNet: Dense Correspondence Regularized Pixel Pair Pose Regression, [paper]

    [arXiv] KeyPose: Multi-view 3D Labeling and Keypoint Estimation for Transparent Objects, [paper]

    2018:

    [CVPR] Real-time seamless single shot 6d object pose prediction, [paper] [code]

    [arXiv] Estimating 6D Pose From Localizing Designated Surface Keypoints, [paper]

    2017:

    [ICCV] BB8: a scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth, [paper]

    [ICCV] SSD-6D: Making rgb-based 3d detection and 6d pose estimation great again, [paper] [code]

    [ICRA] 6-DoF Object Pose from Semantic Keypoints, [paper]

    2.1.2 Template-based Methods

    This kind of methods can be regarded as regression-based methods.

    2020:

    [arXiv] Self6D: Self-Supervised Monocular 6D Object Pose Estimation, [paper]

    [arXiv] A Novel Pose Proposal Network and Refinement Pipeline for Better Object Pose Estimation, [paper]

    [arXiv] G2L-Net: Global to Local Network for Real-time 6D Pose Estimation with Embedding Vector Features, [paper]

    [arXiv] Neural Mesh Refiner for 6-DoF Pose Estimation, [paper]

    [arXiv] MobilePose: Real-Time Pose Estimation for Unseen Objects with Weak Shape Supervision, [paper]

    [arXiv] Robust 6D Object Pose Estimation by Learning RGB-D Features, [paper]

    [arXiv] HybridPose: 6D Object Pose Estimation under Hybrid Representations, [paper]

    2019:

    [arXiv] P2GNet: Pose-Guided Point Cloud Generating Networks for 6-DoF Object Pose Estimation, [paper]

    [arXiv] ConvPoseCNN: Dense Convolutional 6D Object Pose Estimation, [paper]

    [arXiv] PointPoseNet: Accurate Object Detection and 6 DoF Pose Estimation in Point Clouds, [paper]

    [RSS] PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking, [paper]

    [arXiv] Multi-View Matching Network for 6D Pose Estimation, [paper]

    [arXiv] Fast 3D Pose Refinement with RGB Images, [paper]

    [arXiv] MaskedFusion: Mask-based 6D Object Pose Detection, [paper]

    [CoRL] Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects, [paper]

    [IROS] Learning to Estimate Pose and Shape of Hand-Held Objects from RGB Images, [paper]

    [IROSW] Motion-Nets: 6D Tracking of Unknown Objects in Unseen Environments using RGB, [paper]

    [ICCV] DPOD: 6D Pose Object Detector and Refiner, [paper]

    [ICCV] CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation, [paper]

    [ICCV] Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation, [paper]

    [ICCV] Explaining the Ambiguity of Object Detection and 6D Pose From Visual Data, [paper]

    [arXiv] Active 6D Multi-Object Pose Estimation in Cluttered Scenarios with Deep Reinforcement Learning, [paper]

    [arXiv] Accurate 6D Object Pose Estimation by Pose Conditioned Mesh Reconstruction, [paper]

    [arXiv] Learning Object Localization and 6D Pose Estimation from Simulation and Weakly Labeled Real Images, [paper]

    [ICHR] Refining 6D Object Pose Predictions using Abstract Render-and-Compare, [paper]

    [arXiv] Deep-6dpose: recovering 6d object pose from a single rgb image, [paper]

    [arXiv] Real-time Background-aware 3D Textureless Object Pose Estimation, [paper]

    [IROS] SilhoNet: An RGB Method for 6D Object Pose Estimation, [paper]

    2018:

    [ECCV] AAE: Implicit 3D Orientation Learning for 6D Object Detection From RGB Images, [paper] [code]

    [ECCV] DeepIM:Deep Iterative Matching for 6D Pose Estimation, [paper] [code]

    [RSS] Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes, [paper] [code]

    [IROS] Robust 6D Object Pose Estimation in Cluttered Scenes using Semantic Segmentation and Pose Regression Networks, [paper]

    2012:

    [ACCV] Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes, [paper]

    2.1.3 Voting-based Methods

    2019:

    [CVPR] PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation, [paper] [code]

    2017:

    [TPAMI] Robust 3D Object Tracking from Monocular Images Using Stable Parts, [paper]

    [Access] Fast Object Pose Estimation Using Adaptive Threshold for Bin-picking, [paper]

    2014:

    [ECCV] Learning 6d object pose estimation using 3d object coordinate, [paper]

    [ECCV] Latent-class hough forests for 3d object detection and pose estimation, [paper]

    Datasets:

    LineMOD: Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes, ACCV, 2012 [paper] [database]

    YCB Datasets: The YCB Object and Model Set: Towards Common Benchmarks for Manipulation Research, IEEE International Conference on Advanced Robotics (ICAR), 2015 [paper]

    T-LESS Datasets: T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects, IEEE Winter Conference on Applications of Computer Vision (WACV), 2017 [paper]

    HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects, ICCVW, 2019 [paper]

    2.2 Point Cloud-based Methods

    The partial-view point cloud will be aligned to the complete shape in order to obtain the 6D pose. Generally, coarse registration should be conduct firstly to provide an intial alignment, and dense registration methods like ICP (Iterative Closest Point) will be conducted to obtain the final 6D pose.

    2.2.1 Correspondence-based Methods

    2020:

    [arXiv] RPM-Net: Robust Point Matching using Learned Features, [paper]

    [arXiv] End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds, [paper]

    [arXiv] D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features, [paper]

    [arXiv] Self-supervised Point Set Local Descriptors for Point Cloud Registration, [paper]

    [arXiv] StickyPillars: Robust feature matching on point clouds using Graph Neural Networks, [paper]

    2019:

    [arXiv] 3DRegNet: A Deep Neural Network for 3D Point Registration, [paper] [code]

    [CVPR] The Perfect Match: 3D Point Cloud Matching with Smoothed Densities, [paper]

    [arXiv] LCD: Learned Cross-Domain Descriptors for 2D-3D Matching, [paper]

    2018:

    [arXiv] Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation, [paper]

    [ECCV] 3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration, [paper] [code]

    2017:

    [CVPR] 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions, [paper] [code]

    2016:

    [arXiv] Lessons from the Amazon Picking Challenge, [paper]

    [arXiv] Team Delft’s Robot Winner of the Amazon Picking Challenge 2016, [paper]

    [IJCV] A comprehensive performance evaluation of 3D local feature descriptors, [paper]

    2014:

    [CVIU] SHOT: Unique signatures of histograms for surface and texture description, [paper)]

    2011:

    [ICCVW] CAD-model recognition and 6DOF pose estimation using 3D cues, [paper]

    2009:

    [ICRA] Fast Point Feature Histograms (FPFH) for 3D registration, [paper]

    2.2.2 Template-based Methods

    Survey papers:

    2020:

    [arXiv] When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs), [paper]

    [arXiv] Least Squares Optimization: from Theory to Practice, [paper]

    2020:

    [arXiv] A Benchmark for Point Clouds Registration Algorithms, [paper] [code]

    [arXiv] PointGMM: a Neural GMM Network for Point Clouds, [paper]

    [arXiv] SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans, [paper]

    [arXiv] TEASER: Fast and Certifiable Point Cloud Registration, [paper] [code]

    [arXiv] Plane Pair Matching for Efficient 3D View Registration, [paper]

    [arXiv] Learning multiview 3D point cloud registration, [paper]

    [arXiv] Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands, [paper]

    [arXiv] Non-iterative One-step Solution for Point Set Registration Problem on Pose Estimation without Correspondence, [paper]

    [arXiv] 6D Object Pose Regression via Supervised Learning on Point Clouds, [paper]

    2019:

    [arXiv] One Framework to Register Them All: PointNet Encoding for Point Cloud Alignment, [paper]

    [arXiv] DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration, [paper]

    [NeurIPS] PRNet: Self-Supervised Learning for Partial-to-Partial Registration, [paper]

    [CVPR] PointNetLK: Robust & Efficient Point Cloud Registration using PointNet, [paper] [code]

    [ICCV] End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans, [paper]

    [arXiv] Iterative Matching Point, [paper]

    [arXiv] Deep Closest Point: Learning Representations for Point Cloud Registration, [paper] [code]

    [arXiv] PCRNet: Point Cloud Registration Network using PointNet Encoding, [paper] [code]

    2016:

    [TPAMI] Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration, [paper] [code]

    2014:

    [SGP] Super 4PCS Fast Global Pointcloud Registration via Smart Indexing, [paper] [code]

    2.2.3 Voting-based Methods

    2020:

    [arXiv] MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion, [paper]

    [arXiv] YOLOff: You Only Learn Offsets for robust 6DoF object pose estimation, [paper]

    [arXiv] LRF-Net: Learning Local Reference Frames for 3D Local Shape Description and Matching, [paper]

    2019:

    [arXiv] PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation, [paper]

    [CVPR] Densefusion: 6d object pose estimation by iterative dense fusion, [paper] [code]

    2.3 Category-level Methods

    2.3.1 Category-level 6D pose estimation

    2020:

    [arXiv] CPS: Class-level 6D Pose and Shape Estimation From Monocular Images, [paper]

    [arXiv] Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation, [paper]

    2019:

    [arXiv] Category-Level Articulated Object Pose Estimation, [paper]

    [arXiv] LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation, [paper]

    [arXiv] 6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints, [paper] [code]

    [arXiv] Self-Supervised 3D Keypoint Learning for Ego-motion Estimation, [paper]

    [CVPR] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation, [paper] [code]

    [arXiv] Instance- and Category-level 6D Object Pose Estimation, [paper]

    [arXiv] kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation, [paper]

    2.3.2 3D shape reconstruction from images

    2020:

    [arXiv] Few-Shot Single-View 3-D Object Reconstruction with Compositional Priors, [paper]

    [arXiv] Neural Object Descriptors for Multi-View Shape Reconstruction, [paper]

    [arXiv] Leveraging 2D Data to Learn Textured 3D Mesh Generation, [paper]

    [arXiv] Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images, [paper]

    [arXiv] Self-Supervised 2D Image to 3D Shape Translation with Disentangled Representations, [paper]

    [arXiv] Atlas: End-to-End 3D Scene Reconstruction from Posed Images, [paper]

    [arXiv] Instant recovery of shape from spectrum via latent space connections, [paper]

    [arXiv] Self-supervised Single-view 3D Reconstruction via Semantic Consistency, [paper]

    [arXiv] Meta3D: Single-View 3D Object Reconstruction from Shape Priors in Memory, [paper]

    [arXiv] STD-Net: Structure-preserving and Topology-adaptive Deformation Network for 3D Reconstruction from a Single Image, [paper]

    [arXiv] Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data, [paper]

    [arXiv] Deep NRSfM++: Towards 3D Reconstruction in the Wild, [paper]

    [arXiv] Learning to Correct 3D Reconstructions from Multiple Views, [paper]

    2019:

    [arXiv] Boundary Cues for 3D Object Shape Recovery, [paper]

    [arXiv] Learning to Generate Dense Point Clouds with Textures on Multiple Categories, [paper]

    [arXiv] Front2Back: Single View 3D Shape Reconstruction via Front to Back Prediction, [paper]

    [arXiv] Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision, [paper]

    [arXiv] SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization, [paper]

    [arXiv] 3D-GMNet: Learning to Estimate 3D Shape from A Single Image As A Gaussian Mixture, [paper]

    [arXiv] Deep-Learning Assisted High-Resolution Binocular Stereo Depth Reconstruction, [paper]

    2.3.3 3D shape rendering

    2019:

    [arXiv] SynSin: End-to-end View Synthesis from a Single Image, [paper] [project]

    [arXiv] Neural Point Cloud Rendering via Multi-Plane Projection, [paper]

    [arXiv] Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool, [paper]

    3. 2D Planar Grasp

    3.1 Estimating Grasp Contact Points

    2019:

    [IROS] GQ-STN: Optimizing One-Shot Grasp Detection based on Robustness Classifier, [paper]

    [ICRA] Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter, [paper]

    [ICRA] MetaGrasp: Data Efficient Grasping by Affordance Interpreter Network, [paper]

    [IROS] GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter, [paper]

    [ICRA] Multi-View Picking: Next-best-view Reaching for Improved Grasping in Clutter, [paper] [code]

    2018:

    [RSS] Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach, [paper]

    [BMVC] EnsembleNet: Improving Grasp Detection using an Ensemble of Convolutional Neural Networks, [paper]

    [ICRA] Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching, [paper] [code]

    2017:

    [RSS] Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics, [paper] [code]

    2014:

    [ICRA] Fast graspability evaluation on single depth maps for bin picking with general grippers, [paper]

    Dataset:

    Dex-Net, a synthetic dataset of 6.7 million point clouds, grasps, and robust analytic grasp metrics generated from thousands of 3D models.

    3.2 Estimating Oriented Rectangles

    2020:

    [arXiv] Online Self-Supervised Learning for Object Picking: Detecting Optimum Grasping Position using a Metric Learning Approach, [paper]

    [arXiv] A Multi-task Learning Framework for Grasping-Position Detection and Few-Shot Classification, [paper]

    [arXiv] Rigid-Soft Interactive Learning for Robust Grasping*, [paper]

    [arXiv] Optimizing Correlated Graspability Score and Grasp Regression for Better Grasp Prediction, [paper]

    [arXiv] Domain Independent Unsupervised Learning to grasp the Novel Objects, [paper]

    [arXiv] Real-time Grasp Pose Estimation for Novel Objects in Densely Cluttered Environment, [paper]

    [arXiv] Semi-supervised Grasp Detection by Representation Learning in a Vector Quantized Latent Space, [paper]

    2019:

    [arXiv] Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network, [paper]

    [IROS] Domain Independent Unsupervised Learning to grasp the Novel Objects, [paper]

    [Sensors] Vision for Robust Robot Manipulation, [paper]

    [arXiv] Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly, [paper] [code]

    [IROS] GRIP: Generative Robust Inference and Perception for Semantic Robot Manipulation in Adversarial Environments, [paper]

    [arXiv] Efficient Fully Convolution Neural Network for Generating Pixel Wise Robotic Grasps With High Resolution Images, [paper]

    [arXiv] A Single Multi-Task Deep Neural Network with Post-Processing for Object Detection with Reasoning and Robotic Grasp Detection, [paper]

    [IROS] ROI-based Robotic Grasp Detection for Object Overlapping Scenes, [paper]

    [RO-MAN] Real-time Grasp Pose Estimation for Novel Objects in Densely Cluttered Environment, [paper]

    2018:

    [IROS] Fully convolutional grasp detection network with oriented anchor box, [paper]

    [arXiv] Real-Time, Highly Accurate Robotic Grasp Detection using Fully Convolutional Neural Networks with High-Resolution Images, [paper]

    [arXiv] Real-world Multi-object, Multi-grasp Detection, [paper]

    [arXiv] Classification based grasp detection using spatial transformer network, [paper]

    [arXiv] A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes, [paper]

    2017:

    [IROS] Robotic Grasp Detection using Deep Convolutional Neural Networks, [paper]

    [ICMITE] Robust Robot Grasp Detection in Multimodal Fusion, [paper]

    [ICRA] A hybrid deep architecture for robotic grasp detection, [paper]

    2016:

    [ICRA] Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours, [paper]

    [ICRA] Object discovery and grasp detection with a shared convolutional neural network, [paper]

    2015:

    [ICRA] Real-time grasp detection using convolutional neural networks, [paper] [code]

    [IJRR] Deep Learning for Detecting Robotic Grasps, [paper]

    2011:

    [ICRA] Efficient grasping from rgbd images: Learning using a new rectangle representation, [paper]


    Datasets:

    Cornell dataset, the dataset consists of 1035 images of 280 different objects.

    Jacquard Dataset, Jacquard: A Large Scale Dataset for Robotic Grasp Detection” in IEEE International Conference on Intelligent Robots and Systems, 2018, [paper]

    4. 6DoF Grasp

    Grasp Representation:
    The grasp is represented as 6DoF pose in 3D domain, and the gripper can grasp the object from various angles. The input to this task is 3D point cloud from RGB-D sensors, and this task contains two stages. In the first stage, the target object should be extracted from the scene. In the second stage, if there exists an existing 3D model, the 6D pose of the object could be computed. If there exists no 3D models, the 6DoF grasp pose will be computed from some other methods.

    4.1 Methods based on Single-view Point Cloud

    In this situation, there exist no 3D models, an the 6-DoF grasps are estimated from available partial data. This can be implemented by directly estimating from partial view point cloud, or indirectly estimating after shape completion.

    4.1.1 Methods of Estimating Candidate Grasps

    2020:

    [arXiv] Go Fetch: Mobile Manipulation in Unstructured Environments, [paper]

    [arXiv] Real-time Fruit Recognition and Grasp Estimation for Autonomous Apple harvesting, [paper]

    [arXiv] PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds, [paper]

    [arXiv] EGAD! an Evolved Grasping Analysis Dataset for diversity and reproducibility in robotic manipulation, [paper]

    [ariXiv] REGNet: REgion-based Grasp Network for Single-shot Grasp Detection in Point Clouds, [paper]

    [RAL] GRASPA 1.0: GRASPA is a Robot Arm graSping Performance benchmArk, [paper] [code]

    [arXiv] GraspNet: A Large-Scale Clustered and Densely Annotated Dataset for Object Grasping, [paper]

    2019:

    [ISRR] A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set, [paper] [project]

    [arXiv] 6-DOF Grasping for Target-driven Object Manipulation in Clutter, [paper]

    [IROS] Grasping Unknown Objects Based on Gripper Workspace Spheres, [paper]

    [arXiv] Learning to Generate 6-DoF Grasp Poses with Reachability Awareness, [paper]

    [CoRL] S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes, [paper]

    [ICCV] 6-DoF GraspNet: Variational Grasp Generation for Object Manipulation, [paper] [code]

    [ICRA] PointNetGPD: Detecting Grasp Configurations from Point Sets, [paper] [code]

    [IJARS] Fast geometry-based computation of grasping points on three-dimensional point clouds, [paper]

    2017:

    [IJRR] Grasp Pose Detection in Point Clouds, [paper] [code]

    [ICINCO] Using geometry to detect grasping points on 3D unknown point cloud, [paper]

    2015:

    [arXiv] Using geometry to detect grasps in 3d point clouds, [paper]

    2010:

    [RAS] Learning grasping points with shape context, [paper]

    4.1.2 Methods of Transferring Grasps

    a. Grasp transfer

    2020:

    [arXiv] DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-based Robotic Grasping, [paper]

    2019:

    [arXiv] Using Synthetic Data and Deep Networks to Recognize Primitive Shapes for Object Grasping, [paper]

    [ICRA] Transferring Grasp Configurations using Active Learning and Local Replanning, [paper]

    2017:

    [AIP] Fast grasping of unknown objects using principal component analysis, [paper]

    2016:

    [Humanoids] Part-based grasp planning for familiar objects, [paper]

    2015:

    [RAS] Category-based task specific grasping, [paper]

    2003:

    [ICRA] Automatic grasp planning using shape primitives, [paper]

    b. Non-rigid registration

    2020:

    [arXiv] Quasi-Newton Solver for Robust Non-Rigid Registration, [paper]

    [arXiv] MINA: Convex Mixed-Integer Programming for Non-Rigid Shape Alignment, [paper]

    2019:

    [arXiv] Non-Rigid Point Set Registration Networks, [paper] [code]

    2018:

    [RAL] Transferring Category-based Functional Grasping Skills by Latent Space Non-Rigid Registration, [paper]

    [RAS] Learning Postural Synergies for Categorical Grasping through Shape Space Registration, [paper]

    [RAS] Autonomous Dual-Arm Manipulation of Familiar Objects, [paper]

    c. Shape correspondence

    2020:

    [arXiv] Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences, [paper]

    [arXiv] Deep Geometric Functional Maps: Robust Feature Learning for Shape Correspondence, [paper] [code]

    [arXiv] Efficient and Robust Shape Correspondence via Sparsity-Enforced Quadratic Assignment, [paper]

    [CVM] Learning local shape descriptors for computing non-rigid dense correspondence, [paper]

    [JCDE] Embedded spectral descriptors: learning the point-wise correspondence metric via Siamese neural networks, [paper]

    [arXiv] SAPIEN: A SimulAted Part-based Interactive ENvironment, [paper]

    [TVCG] Voting for Distortion Points in Geometric Processing, [paper]

    [arXiv] SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence, [paper]

    2019:

    [arXiv] Fine-grained Object Semantic Understanding from Correspondences, [paper]

    [IROS] Multi-step Pick-and-Place Tasks Using Object-centric Dense Correspondences, [code]

    [arXiv] Unsupervised cycle-consistent deformation for shape matching, [paper]

    [arXiv] ZoomOut: Spectral Upsampling for Efficient Shape Correspondence, [paper]

    [C&G] Partial correspondence of 3D shapes using properties of the nearest-neighbor field, [paper]

    4.2 Methods based on Complete Shape

    4.2.1 Methods of Estimating 6D Object Pose

    2017:

    [IROS] SegICP: Integrated Deep Semantic Segmentation and Pose Estimation, [paper]

    [ICRA] Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge, [paper] [code]

    2010:

    [SIMPAR] OpenGRASP: A Toolkit for Robot Grasping Simulation, [paper]

    2009:

    [ICAR] An automatic grasp planning system for service robots, [paper]

    2004:

    [RAM] Graspit! A versatile simulator for robotic grasping, [paper]

    4.2.2 Methods of Shape Completion

    a. Shape Completion-based Grasp

    2020:

    [arXiv] Robotic Grasping through Combined Image-Based Grasp Proposal and 3D Reconstruction, [paper]

    2019:

    [arXiv] ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation, [paper]

    [arXiv] kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion, [paper] [code]

    [arXiv] Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks, [paper]

    [IROS] Robust Grasp Planning Over Uncertain Shape Completions, [paper]

    [arXiv] Multi-Modal Geometric Learning for Grasping and Manipulation, [paper]

    2018:

    [ICRA] Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations, [paper]

    [IROS] 3D Shape Perception from Monocular Vision, Touch, and Shape Priors, [paper]

    2017:

    [IROS] Shape Completion Enabled Robotic Grasping, [paper]

    b. Shape Completion or Generation

    2020:

    [arXiv] Anisotropic Convolutional Networks for 3D Semantic Scene Completion, [paper]

    [arXiv] Cascaded Refinement Network for Point Cloud Completio, [paper]

    [arXiv] Generative PointNet: Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification, [paper]

    [arXiv] Intrinsic Point Cloud Interpolation via Dual Latent Space Navigation, [paper]

    [arXiv] Modeling 3D Shapes by Reinforcement Learning, [paper]

    [arXiv] PF-Net: Point Fractal Network for 3D Point Cloud Completion, [paper]

    [arXiv] Hypernetwork approach to generating point clouds, [paper]

    [arXiv] Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion, [paper]

    [arXiv] PolyGen: An Autoregressive Generative Model of 3D Meshes, [paper]

    [arXiv] BlockGAN Learning 3D Object-aware Scene Representations from Unlabelled Images, [paper]

    [arXiv] Implicit Geometric Regularization for Learning Shapes, [paper]

    [arXiv] The Whole Is Greater Than the Sum of Its Nonrigid Parts, [paper]

    [arXiv] PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions, [paper]

    [arXiv] Multimodal Shape Completion via Conditional Generative Adversarial Networks, [paper]

    [arXiv] Symmetry Detection of Occluded Point Cloud Using Deep Learning, [paper]

    2019:

    [arXiv] Inferring Occluded Geometry Improves Performance when Retrieving an Object from Dense Clutter, [paper]

    2018:

    [3DORW] Completion of Cultural Heritage Objects with Rotational Symmetry, [paper]

    c. Depth Completion and Estimation

    2020:

    [arXiv] DepthNet Nano: A Highly Compact Self-Normalizing Neural Network for Monocular Depth Estimation, [paper]

    [arXiv] RealMonoDepth: Self-Supervised Monocular Depth Estimation for General Scenes, [paper]

    [arXiv] Monocular Depth Estimation with Self-supervised Instance Adaptation, [paper]

    [arXiv] Guiding Monocular Depth Estimation Using Depth-Attention Volume, [paper]

    [arXiv] 3D Photography using Context-aware Layered Depth Inpainting, [paper]

    [arXiv] Occlusion-Aware Depth Estimation with Adaptive Normal Constraints, [paper]

    [arXiv] The Edge of Depth: Explicit Constraints between Segmentation and Depth, [paper]

    [arXiv] Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume, [paper]

    [arXiv] DeFeat-Net: General Monocular Depth via Simultaneous Unsupervised Representation Learning, [paper]

    [arXiv] Adversarial Attacks on Monocular Depth Estimation, [paper]

    [arXiv] Monocular Depth Prediction Through Continuous 3D Loss, [paper]

    [arXiv] 3dDepthNet: Point Cloud Guided Depth Completion Network for Sparse Depth and Single Color Image, [paper]

    [arXiv] Depth Estimation by Learning Triangulation and Densification of Sparse Points for Multi-view Stereo, [paper]

    [arXiv] Monocular Depth Estimation Based On Deep Learning: An Overview, [paper]

    [arXiv] Scene Completenesss-Aware Lidar Depth Completion for Driving Scenario, [paper]

    [arXiv] Fast Depth Estimation for View Synthesis, [paper]

    [arXiv] Active Depth Estimation: Stability Analysis and its Applications, [paper]

    [arXiv] Uncertainty depth estimation with gated images for 3D reconstruction, [paper]

    [arXiv] Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry, [paper]

    [arXiv] A-TVSNet: Aggregated Two-View Stereo Network for Multi-View Stereo Depth Estimation, [paper]

    [arXiv] Predicting Sharp and Accurate Occlusion Boundaries in Monocular Depth Estimation Using Displacement Fields, [paper]

    [ICLR] SEMANTICALLY-GUIDED REPRESENTATION LEARNING FOR SELF-SUPERVISED MONOCULAR DEPTH, [paper]

    [arXiv] 3D Gated Recurrent Fusion for Semantic Scene Completion, [paper]

    [arXiv] Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot, [paper]

    [arXiv] Fast Generation of High Fidelity RGB-D Images by Deep-Learning with Adaptive Convolution, [paper]

    [arXiv] DiverseDepth: Affine-invariant Depth Prediction Using Diverse Data, [paper]

    [arXiv] Depth Map Estimation of Dynamic Scenes Using Prior Depth Information, [paper]

    [arXiv] FIS-Nets: Full-image Supervised Networks for Monocular Depth Estimation, [paper]

    [ICRA] Depth Based Semantic Scene Completion with Position Importance Aware Loss, [paper]

    [arXiv] ResDepth: Learned Residual Stereo Reconstruction, [paper]

    [arXiv] Single Image Depth Estimation Trained via Depth from Defocus Cues, [paper]

    [arXiv] RoutedFusion: Learning Real-time Depth Map Fusion, [paper]

    [arXiv] Don’t Forget The Past: Recurrent Depth Estimation from Monocular Video, [paper]

    [AAAI] Morphing and Sampling Network for Dense Point Cloud Completion, [paper] [code]

    [AAAI] CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion, [paper]

    2019:

    [arXiv] Normal Assisted Stereo Depth Estimation, [paper]

    [arXiv] GEOMETRY-AWARE GENERATION OF ADVERSARIAL AND COOPERATIVE POINT CLOUDS, [paper]

    [arXiv] DeepSFM: Structure From Motion Via Deep Bundle Adjustment, [paper]

    [CVIU] On the Benefit of Adversarial Training for Monocular Depth Estimation, [paper]

    [ICCV] Learning Joint 2D-3D Representations for Depth Completion, [paper]

    [ICCV] Deep Optics for Monocular Depth Estimation and 3D Object Detection, [paper]

    [arXiv] Deep Classification Network for Monocular Depth Estimation, [paper]

    [ICCV] Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints, [paper]

    [arXiv] Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era, [paper]

    [arXiv] Real-time Vision-based Depth Reconstruction with NVidia Jetson, [paper]

    [IROS] Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics, [paper]

    [arXiv] Mesh R-CNN, [paper]

    [arXiv] Monocular depth estimation: a survey, [paper]

    2018:

    [3DV] PCN: Point Completion Network, [paper] [code]

    [NeurIPS] Learning to Reconstruct Shapes from Unseen Classes, [paper] [code]

    [ECCV] Learning Shape Priors for Single-View 3D Completion and Reconstruction, [paper] [code]

    [CVPR] Deep Depth Completion of a Single RGB-D Image, [paper] [code]

    d. Point Cloud Denoising and Samping

    2020:

    [arXiv] Self-Supervised Learning for Domain Adaptation on Point-Clouds, [paper]

    [arXiv] Non-Local Part-Aware Point Cloud Denoising, [paper]

    [arXiv] PUGeo-Net: A Geometry-centric Network for 3D Point Cloud Upsampling, [paper]

    2019:

    [arXiv] CNN-based Lidar Point Cloud De-Noising in Adverse Weather, [paper]

    [arXiv] PU-GCN: Point Cloud Upsampling using Graph Convolutional Networks, [paper] [code]

    [ICCV] PU-GAN: a Point Cloud Upsampling Adversarial Network, [paper] [code]

    [CVPR] Patch-based Progressive 3D Point Set Upsampling, [paper] [code]

    [arXiv] SampleNet: Differentiable Point Cloud Sampling, [paper] [code]

    2018:

    [CVPR] PU-Net: Point Cloud Upsampling Network, [paper] [code]

    5. Task-oriented Methods

    5.1 Task-oriented Manipulation

    2020:

    [arXiv] Neuromorphic Event-Based Slip Detection and Suppression in Robotic Grasping and Manipulation, [paper]

    [arXiv] Combinatorial 3D Shape Generation via Sequential Assembly, [paper]

    [arXiv] Learning visual policies for building 3D shape categories, [paper]

    [arXiv] Where to relocate?: Object rearrangement inside cluttered and confined environments for robotic manipulation, [paper]

    [arXiv] Correspondence Networks with Adaptive Neighbourhood Consensus, [paper]

    [arXiv] Development of a Robotic System for Automated Decaking of 3D-Printed Parts, [paper]

    [arXiv] Team O2AS at the World Robot Summit 2018: An Approach to Robotic Kitting and Assembly Tasks using General Purpose Grippers and Tools, [paper]

    [arXiv] Towards Mobile Multi-Task Manipulation in a Confined and Integrated Environment with Irregular Objects, [paper]

    [arXiv] Autonomous Industrial Assembly using Force, Torque, and RGB-D sensing, [[paper](Autonomous Industrial Assembly using Force, Torque, and RGB-D sensing)]

    [RAL] A Deep Learning Approach to Grasping the Invisible, [paper] [code]

    2019:

    [arXiv] KETO: Learning Keypoint Representations for Tool Manipulation, [paper]

    [arXiv] Learning Task-Oriented Grasping from Human Activity Datasets, [paper]

    5.2 Grasp Affordance

    2020:

    [arXiv] Learning to Grasp 3D Objects using Deep Residual U-Nets, [paper]

    2019:

    [IROS] Detecting Robotic Affordances on Novel Objects with Regional Attention and Attributes, [paper]

    [IROS] Learning Grasp Affordance Reasoning through Semantic Relations, [paper]

    [arXiv] Automatic pre-grasps generation for unknown 3D objects, [paper]

    [IECON] A novel object slicing based grasp planner for 3D object grasping using underactuated robot gripper, [paper]

    2018:

    [ICRA] AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection, [paper]

    [arXiv] Workspace Aware Online Grasp Planning, [paper]

    5.3 3D Part Segmentation

    2020:

    [arXiv] Learning Unsupervised Hierarchical Part Decomposition of 3D Objects from a Single RGB Image, [paper]

    [arXiv] Learning 3D Part Assembly from a Single Image, [paper]

    [ICLR] Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories, [paper]

    2019:

    [arXiv] Skeleton Extraction from 3D Point Clouds by Decomposing the Object into Parts, [paper]

    [arXiv] Neural Shape Parsers for Constructive Solid Geometry, [paper]

    [arXiv] PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes, [paper]

    [CVPR] PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation, [paper] [code]

    [C&G] Autoencoder-based part clustering for part-in-whole retrieval of CAD models, [paper]

    2016:

    [SiggraphAsia] A Scalable Active Framework for Region Annotation in 3D Shape Collections, [paper]

    6. Dexterous Grippers

    2020:

    [arXiv] HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map, [paper]

    [arXiv] Functionally Divided Manipulation Synergy for Controlling Multi-fingered Hands, [paper]

    [arXiv] The State of Service Robots: Current Bottlenecks in Object Perception and Manipulation, [paper]

    [arXiv] Selecting and Designing Grippers for an Assembly Task in a Structured Approach, [paper]

    [arXiv] A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU, [paper]

    [arXiv] Robust High-Transparency Haptic Exploration for Dexterous Telemanipulation, [paper]

    [arXiv] Tactile Dexterity: Manipulation Primitives with Tactile Feedback, [paper]

    [arXiv] Deep Differentiable Grasp Planner for High-DOF Grippers, [paper]

    [arXiv] Multi-Fingered Grasp Planning via Inference in Deep Neural Networks, [paper]

    [RAL] Benchmarking In-Hand Manipulation, [paper]

    2019:

    [arXiv] GraphPoseGAN: 3D Hand Pose Estimation from a Monocular RGB Image via Adversarial Learning on Graphs, [paper]

    [arXiv] HMTNet:3D Hand Pose Estimation from Single Depth Image Based on Hand Morphological Topology, [paper]

    [arXiv] UniGrasp: Learning a Unified Model to Grasp with N-Fingered Robotic Hands, [paper]

    [ScienceRobotics] On the choice of grasp type and location when handing over an object, [paper]

    [arXiv] Solving Rubik’s Cube with a Robot Hand, [paper]

    [IJARS] Fast geometry-based computation of grasping points on three-dimensional point clouds, [paper] [code]

    [arXiv] Learning better generative models for dexterous, single-view grasping of novel objects, [paper]

    [arXiv] DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System, [paper]

    [IROS] Optimization Model for Planning Precision Grasps with Multi-Fingered Hands, [paper]

    [IROS] Generating Grasp Poses for a High-DOF Gripper Using Neural Networks, [paper]

    [arXiv] Deep Dynamics Models for Learning Dexterous Manipulation, [paper]

    [CVPR] Learning joint reconstruction of hands and manipulated objects, [paper]

    [CVPR] H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions, [paper]

    [IROS] Efficient Grasp Planning and Execution with Multi-Fingered Hands by Surface Fitting, [paper]

    [arXiv] Efficient Bimanual Manipulation Using Learned Task Schemas, [paper]

    [ICRA] High-Fidelity Grasping in Virtual Reality using a Glove-based System, [paper] [code]

    7. Data Generation

    7.1 Simulation to Reality

    2020:

    [arXiv] A Study on the Challenges of Using Robotics Simulators for Testing, [paper]

    [arXiv] Joint Supervised and Self-Supervised Learning for 3D Real-World Challenges, [paper]

    [arXiv] RoboTHOR: An Open Simulation-to-Real Embodied AI Platform, [paper]

    [arXiv] On the Effectiveness of Virtual Reality-based Training for Robotic Setup, [paper]

    [arXiv] LiDARNet: A Boundary-Aware Domain Adaptation Model for Lidar Point Cloud Semantic Segmentation, [paper]

    [arXiv] Multi-source Domain Adaptation in the Deep Learning Era: A Systematic Survey, [paper]

    [arXiv] Learning Machines from Simulation to Real World, [paper]

    [arXiv] Sim2Real2Sim: Bridging the Gap Between Simulation and Real-World in Flexible Object Manipulation, [paper]

    2019:

    [IROS] Learning to Augment Synthetic Images for Sim2Real Policy Transfer, [paper]

    [arXiv] Accept Synthetic Objects as Real-End-to-End Training of Attentive Deep Visuomotor Policies for Manipulation in Clutter, [paper]

    [RSSW] Generative grasp synthesis from demonstration using parametric mixtures, [paper]

    2018:

    [RSS] Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision, [paper]

    [CoRL] Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects, [paper]

    [arXiv] Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation, [paper]

    2017:

    [arXiv] Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping, [paper]

    7.2 Self-supervised Methods

    2019:

    [arXiv] Self-supervised 6D Object Pose Estimation for Robot Manipulation, [paper]

    2018:

    [RSS] Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision, [paper]

    8. Multi-source

    2020:

    [arXiv] Multimodal Material Classification for Robots using Spectroscopy and High Resolution Texture Imaging, [paper]

    [arXiv] Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream, [paper]

    [ToR] A Transfer Learning Approach to Cross-modal Object Recognition: from Visual Observation to Robotic Haptic Exploration, [paper]

    [arXiv] Accurate Vision-based Manipulation through Contact Reasoning, [paper]

    2019:

    [arXiv] RoboSherlock: Cognition-enabled Robot Perception for Everyday Manipulation Tasks, [paper]

    [ICRA] Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks, [paper]

    [CVPR] ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging, [paper] [code]

    2018:

    [arXiv] Learning to Grasp without Seeing, [paper]

    9. Motion Planning

    9.1 Visual servoing

    2020:

    [arXiv] Detailed 2D-3D Joint Representation for Human-Object Interaction, [paper]

    [arXiv] Neuromorphic Eye-in-Hand Visual Servoing, [paper]

    [arXiv] Predicting Target Feature Configuration of Non-stationary Objects for Grasping with Image-Based Visual Servoing, [paper]

    [AAAI] That and There: Judging the Intent of Pointing Actions with Robotic Arms, [paper]

    2019:

    [arXiv] Camera-to-Robot Pose Estimation from a Single Image, [paper]

    [ICRA] Learning Driven Coarse-to-Fine Articulated Robot Tracking, [paper]

    [CVPR] Craves: controlling robotic arm with a vision-based, economic system, [paper] [code]

    2018:

    [arXiv] Point-to-Pose Voting based Hand Pose Estimation using Residual Permutation Equivariant Layer, [paper]

    2016:

    [ICRA] Robot Arm Pose Estimation by Pixel-wise Regression of Joint Angles, [paper]

    2014:

    [ICRA] Robot Arm Pose Estimation through Pixel-Wise Part Classification, [paper]

    9.2 Path Planning

    2020:

    [arXiv] Human-Guided Planner for Non-Prehensile Manipulation, [paper)]

    [arXiv] Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation, [paper]

    [arXiv] GOMP: Grasp-Optimized Motion Planning for Bin Picking, [paper]

    [arXiv] Describing Physics For Physical Reasoning: Force-based Sequential Manipulation Planning, [paper]

    [arXiv] Reaching, Grasping and Re-grasping: Learning Fine Coordinated Motor Skills, [paper]

    2019:

    [arXiv] Manipulation Trajectory Optimization with Online Grasp Synthesis and Selection, [paper]

    [arXiv] Parareal with a Learned Coarse Model for Robotic Manipulation, [paper]

    10. Imitation Learning

    2020:

    [arXiv] HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation, [paper]

    [arXiv] SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks, [paper]

    [arXiv] A Geometric Perspective on Visual Imitation Learning, [paper]

    [arXiv] Vision-based Robot Manipulation Learning via Human Demonstrations, [paper]

    [arXiv] Gaussian-Process-based Robot Learning from Demonstration, [paper]

    2019:

    [arXiv] Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations, [paper] [project]

    [arXiv] Motion Reasoning for Goal-Based Imitation Learning, [paper]

    [IROS] Robot Learning of Shifting Objects for Grasping in Cluttered Environments, [paper] [code]

    [arXiv] Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control, [paper]

    [arXiv] Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video, [paper]

    [IROS] Learning Actions from Human Demonstration Video for Robotic Manipulation, [paper]

    [RSSW] Generative grasp synthesis from demonstration using parametric mixtures, [paper]

    2018:

    [arXiv] Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation, [paper]

    11. Reinforcement Learning

    2020:

    [arXiv] Learning Precise 3D Manipulation from Multiple Uncalibrated Cameras, [paper]

    [arXiv] The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation, [paper]

    [arXiv] Learning Pregrasp Manipulation of Objects from Ungraspable Poses, [paper]

    [arXiv] Deep Reinforcement Learning for Autonomous Driving: A Survey, [paper]

    [arXiv] Lyceum: An efficient and scalable ecosystem for robot learning, [paper]

    [arXiv] Planning an Efficient and Robust Base Sequence for a Mobile Manipulator Performing Multiple Pick-and-place Tasks, [paper]

    [arXiv] Reward Engineering for Object Pick and Place Training, [paper]

    2019:

    [arXiv] Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning, [paper] [project] [code]

    [ROBIO] Efficient Robotic Task Generalization Using Deep Model Fusion Reinforcement Learning, [paper]

    [arXiv] Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies, [paper]

    [IROS] Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity, [paper]

    [arXiv] IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data, [paper]

    [arXiv] Dynamic Cloth Manipulation with Deep Reinforcement Learning, [paper]

    [CoRL] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, [paper] [project]

    [CoRL] Asynchronous Methods for Model-Based Reinforcement Learning, [paper]

    [CoRL] Entity Abstraction in Visual Model-Based Reinforcement Learning, [paper]

    [CoRL] Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation, [paper] [project]

    [arXiv] Contextual Imagined Goals for Self-Supervised Robotic Learning, [paper]

    [arXiv] Learning to Manipulate Deformable Objects without Demonstrations, [paper] [project]

    [arXiv] A Deep Learning Approach to Grasping the Invisible, [paper]

    [arXiv] Knowledge Induced Deep Q-Network for a Slide-to-Wall Object Grasping, [paper]

    [arXiv] Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping, [paper]

    [arXiv] Adaptive Curriculum Generation from Demonstrations for Sim-to-Real Visuomotor Control, [paper]

    [arXiv] Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations, [paper]

    [arXiv] Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation, [paper]

    [arXiv] Object Perception and Grasping in Open-Ended Domains, [paper]

    [CoRL] ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots, [paper] [code]

    [RSS] End-to-End Robotic Reinforcement Learning without Reward Engineering, [paper]

    [arXiv] Learning to combine primitive skills: A step towards versatile robotic manipulation, [paper]

    [CoRL] A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots, [paper] [code]

    [ICCAS] Deep Reinforcement Learning Based Robot Arm Manipulation with Efficient Training Data through Simulation, [paper]

    [CVPR] CRAVES: Controlling Robotic Arm with a Vision-based Economic System, [paper] [code]

    [Report] A Unified Framework for Manipulating Objects via Reinforcement Learning, [paper]

    2018:

    [IROS] Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning, [paper] [code]

    [CoRL] QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation, [paper]

    [arXiv] Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods, [paper]

    [arXiv] Pick and Place Without Geometric Object Models, [paper]

    2017:

    [arXiv] Deep Reinforcement Learning for Robotic Manipulation-The state of the art, [paper]

    2016:

    [IJRR] Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning, [paper]

    2013:

    [IJRR] Reinforcement learning in robotics: A survey, [paper]

    12. Experts

    Abhinav Gupta(CMU & FAIR): Robotics, machine learning

    Andreas ten Pas(Northeastern University): Robotic Grasping, Deep Learning, Simulation-based Planning

    Andy Zeng(Princeton University & Google Brain Robotics): 3D Deep Learning, Robotic Grasping

    Animesh Garg(University of Toronto): Robotics, Reinforcement Learning

    Bugra Tekin(Microsoft MR): Pose Estimation

    Cewu Lu(SJTU): Machine Vision

    Charles Ruizhongtai Qi(Waymo(Google)): 3D Deep Learning

    Danfei Xu(Stanford University): Robotics, Computer Vision

    Deter Fox(Nvidia & University of Washington): Robotics, Artificial intelligence, State Estimation

    Fei-Fei Li(Stanford University): Computer Vision

    Guofeng Zhang(ZJU): 3D Vision, SLAM

    Hao Su(UC San Diego): 3D Deep Learning

    Jeannette Bohg(Stanford University): Perception for robotic manipulation and grasping

    Jianping Shi(SenseTime): Computer Vision

    Juxi Leitner(Australian Centre of Excellence for Robotic Vision (ACRV)): Robotic grasping

    Lerrel Pinto(UC Berkeley): Robotics

    Lorenzo Jamone(Queen Mary University of London (QMUL)): Cognitive Robotics

    Lorenzo Natale(Italian Institute of Technology): Humanoid robotic sensing and perception

    Kaiming He(Facebook AI Research (FAIR)): Deep Learning

    Kai Xu(NUDT): Graphics, Geometry

    Ken Goldberg(UC Berkeley): Robotics

    Marc Pollefeys(Microsoft & ETH): Computer Vision

    Markus Vincze(Technical University Wien (TUW)): Robotic Vision

    Matthias Nießner(TUM): 3D reconstruction, Semantic 3D Scene Understanding

    Oliver Brock(TU Berlin): Robotic manipulation

    Pascal Fua(EPFL): Computer Vision

    Peter K. Allen.(Columbia University): Robotic Grasping, 3-D vision, Modeling, Medical robotics

    Peter Corke(Queensland University of Technology): Robotic vision

    Pieter Abbeel(UC Berkeley): Artificial Intelligence, Advanced Robotics

    Raquel Urtasun(Uber ATG & University of Toronto): AI for self-driving cars, Computer Vision, Robotics

    Robert Platt(Northeastern University): Robotic manipulation

    Ruigang Yang(Baidu): Computer Vision, Robotics

    Sergey Levine(UC Berkeley): Reinforcement Learning

    Shuran Song(Columbia University), 3D Deep Learning, Robotics

    Silvio Savarese(Stanford University): Computer Vision

    Song-Chun Zhu(UCLA): Computer Vision

    Tamim Asfour(Karlsruhe Institute of Technology (KIT)): Humanoid Robotics

    Thomas Funkhouser(Princeton University): Geometry, Graphics, Shape

    Valerio Ortenzi(University of Birmingham): Robotic vision

    Vicient Lepetit(University of Bordeaux): Machine Learning, 3D Vision

    Xiaogang Wang(Chinese University of Hong Kong): Deep Learning, Computer Vision

    Xiaozhi Chen(DJI): Deep learning

    Yan Xinchen(Uber ATG): Deep Representation Learning, Generative Modeling

    Yasutaka Furukawa(SFU): 3D Reconstruction

    Yu Xiang(Nvidia): Robotics, Computer Vision

    Yue Wang(MIT): 3D Deep Learning

    展开全文
  • 针对生产线上工业机器人抓取系统中摄像机标定、目标工件识别匹配、机器人对目标工件定位抓取这3个主要步骤在现阶段研究成果进行了综述,对计算机视觉定位中涉及到相关图像预处理方法进行了分析与归纳,...
  • 基于视觉的机器人抓取--物体定位,位姿估计到抓取估计课堂笔记 杜国光博士在智东西公开课上讲了《基于视觉的机器人抓取--物体定位,位姿估计到抓取估计》的精彩课程 满满的干货,记下来,后面慢慢消化 2020.11.06...

    基于视觉的机器人抓取--物体定位,位姿估计到抓取估计课堂笔记

    杜国光博士在智东西公开课上讲了《基于视觉的机器人抓取--物体定位,位姿估计到抓取估计》的精彩课程

    满满的干货,记下来,后面慢慢消化

    2020.11.06课堂笔记

    图像 小部件

     

    一.

    二。物体定位

     

    滑窗+提取局部特征点+分来+svm(不确定是否是这个)

    基于DL:

    region proposal networkin +回归

    基于截锥体得到一块点云,区域内进行分割,回归

    三。物体位姿估计

     

     

    四。抓取物体的位姿估计

     

    五。挑战和未来研究方向

    展开全文
  • CV前沿讲座,是智东西公开课针对计算机视觉推出的一档讲座,聚焦于计算机视觉前沿领域研究成果与进展。我们将持续邀请研究者、专家与资深开发者,为大家带来直播讲解。...而基于视觉的机器人抓取,是通过给...
    893fa23b97edaa61cc7797656d4f67bc.png

    CV前沿讲座,是智东西公开课针对计算机视觉推出的一档讲座,聚焦于计算机视觉前沿领域研究成果与进展。我们将持续邀请研究者、专家与资深开发者,为大家带来直播讲解。

    抓取是机器人的基本和重要的任务之一。机器人抓取所必须的信息是相机坐标系下抓取器的6DoF位姿,包括抓取器的3D位置和抓取器的3D空间朝向,通过控制机械臂的移动使抓取器到该位置和旋转,然后执行抓取操作。而基于视觉的机器人抓取,是通过给机器人安装RGB-D相机,利用人工智能算法,获取抓取器的目标抓取位姿,按照抓取方式的不同,可以分为2D平面抓取和6D空间抓取。

    2D平面抓取是指目标物体放置在水平工作台上,抓取器只能从一个方向进行抓取。而6D空间抓取是指抓取器可以在3D空间从各个角度抓取物体。两种抓取方式,自然有不同的实现方法。

    2D平面抓取因为存在一些限制,抓取器的6D位姿可简化为3D,包括平面内的2D位置和平面内的1D旋转角度,主要分为评估抓取接触点质量和评估带朝向的抓取四边形两种方法。

    6D空间抓取按照依赖物体的完整形状还是物体的部分点云,又可以分为基于部分点云的方法和基于完整形状的方法。当前大多数6D空间抓取都是针对已知3D模型的物体,这些物体的最优抓取位置可以通过人工指定货仿真预先得到,此时,6D空间抓取就转化为了估计物体的6D位姿。

    同时,由于大部分机器人抓取的方法都需要先输入数据然后才能获得目标物体的位置,因此又可以分为三个阶段:物体定位、位姿估计和抓取位姿估计。那么基于视觉的机器人抓取到底需要什么样的技术呢?又有什么样的方法呢?不同的方法之间又有哪些优劣势呢?11月6日晚8点,智东西公开课邀请到达闼科技3D研发负责人、北京师范大学博士杜国光参与到「CV前沿讲座」第22讲,带来主题为《基于视觉的机器人抓取-从物体定位、位姿估计到抓取位姿估计》的直播讲解。

    杜国光博士将深度解析基于视觉进行机器人抓取中所涉及的三大模块,包括物体定位、物体位姿估计和抓取位姿估计。其中,物体定位包括定位不识别、目标检测以及目标实例分割;物体位姿估计包括基于对应的方法、基于模板的方法以及基于投票的方法;抓取位姿估计分为2D平面抓取和6D抓取。在最后,杜博也会详解传统的方法和基于深度学习的方法,并对相关方法进行对比,指出未来的研究方向与挑战。

    杜国光,北京师范大学博士,目前是达闼科技3D研发负责人。他的研究方向为3D视觉感知,包括3D检测/分割、物体6D位姿估计、机器人抓取等,并在相关期刊和会议上发表论文二十余篇。

    024c31bf311f076bb28f74415a2588b5.png

    课程内容

     课 程 主 题 

    8f00514974844a00398abbc70a109636.png

    《基于视觉的机器人抓取-从物体定位、位姿估计到抓取位姿估计》

     课 程 提 纲 

    1、视觉机器人抓取的流程与关键技术

    2、物体定位技术的研究

    3、物体位姿估计的方法解析

    4、2D平面与6D空间机器人抓取位姿估计

    5、方法对比与未来的研究方向和挑战

     讲 师 介 绍 

    杜国光,达闼科技3D研发负责人,北京师范大学博士,研究方向为3D视觉感知,包括3D检测/分割、物体6D位姿估计、机器人抓取等,在相关期刊和会议上发表论文二十余篇。

     直 播 信 息 

    直播时间:11月6日20:00

    直播地点:智东西公开课小程序

    答疑地址:智东西公开课讨论群

    加入讨论群

    本次课程的讲解分为主讲和答疑两部分,主讲以视频直播形式,答疑将在「智东西公开课讨论群」进行。

    加入讨论群,除了可以免费收看直播之外,还能认识讲师,与更多同行和同学一起学习,并进行深度讨论。

    扫码添加小助手糖糖(ID:hitang20)即可申请,备注“姓名-公司/学校/单位-职位/专业”的朋友将会优先审核通过哦~

    5bf012f9a7c72052724746f574ebe31f.png

    往期回顾

    「CV前沿讲座」现已完结21讲,讲解的内容包含了迁移学习中的领域适应、生成对抗网络GAN及可解释性、深度强化学习、OCR、实例分割、语义分割、知识蒸馏、视频理解以及三维重建等,课程累计观看42000余人次。点击下方图片可以观看所有课程回放。

    9fe6bf02280524d9037851cc5ba3963a.png

    点个“在看”和大家一起聊聊

    ???

    展开全文
  • 原标题:基于视觉的机器人抓取-从物体定位、位姿估计到抓取位姿估计 | 公开课预CV前沿讲座,是智东西公开课针对计算机视觉推出的一档讲座,聚焦于计算机视觉前沿领域研究成果与进展。我们将持续邀请研究者、专家与...

    原标题:基于视觉的机器人抓取-从物体定位、位姿估计到抓取位姿估计 | 公开课预

    CV前沿讲座,是智东西公开课针对计算机视觉推出的一档讲座,聚焦于计算机视觉前沿领域研究成果与进展。我们将持续邀请研究者、专家与资深开发者,为大家带来直播讲解。

    抓取是机器人的基本和重要的任务之一。机器人抓取所必须的信息是相机坐标系下抓取器的6DoF位姿,包括抓取器的3D位置和抓取器的3D空间朝向,通过控制机械臂的移动使抓取器到该位置和旋转,然后执行抓取操作。而基于视觉的机器人抓取,是通过给机器人安装RGB-D相机,利用人工智能算法,获取抓取器的目标抓取位姿,按照抓取方式的不同,可以分为2D平面抓取和6D空间抓取。

    2D平面抓取是指目标物体放置在水平工作台上,抓取器只能从一个方向进行抓取。而6D空间抓取是指抓取器可以在3D空间从各个角度抓取物体。两种抓取方式,自然有不同的实现方法。

    2D平面抓取因为存在一些限制,抓取器的6D位姿可简化为3D,包括平面内的2D位置和平面内的1D旋转角度,主要分为评估抓取接触点质量和评估带朝向的抓取四边形两种方法。

    6D空间抓取按照依赖物体的完整形状还是物体的部分点云,又可以分为基于部分点云的方法和基于完整形状的方法。当前大多数6D空间抓取都是针对已知3D模型的物体,这些物体的最优抓取位置可以通过人工指定货仿真预先得到,此时,6D空间抓取就转化为了估计物体的6D位姿。

    同时,由于大部分机器人抓取的方法都需要先输入数据然后才能获得目标物体的位置,因此又可以分为三个阶段:物体定位、位姿估计和抓取位姿估计。那么基于视觉的机器人抓取到底需要什么样的技术呢?又有什么样的方法呢?不同的方法之间又有哪些优劣势呢?11月6日晚8点,智东西公开课邀请到达闼科技3D研发负责人、北京师范大学博士杜国光参与到「CV前沿讲座」第22讲,带来主题为《基于视觉的机器人抓取-从物体定位、位姿估计到抓取位姿估计》的直播讲解。

    杜国光博士将深度解析基于视觉进行机器人抓取中所涉及的三大模块,包括物体定位、物体位姿估计和抓取位姿估计。其中,物体定位包括定位不识别、目标检测以及目标实例分割;物体位姿估计包括基于对应的方法、基于模板的方法以及基于投票的方法;抓取位姿估计分为2D平面抓取和6D抓取。在最后,杜博也会详解传统的方法和基于深度学习的方法,并对相关方法进行对比,指出未来的研究方向与挑战。

    杜国光,北京师范大学博士,目前是达闼科技3D研发负责人。他的研究方向为3D视觉感知,包括3D检测/分割、物体6D位姿估计、机器人抓取等,并在相关期刊和会议上发表论文二十余篇。

    课程内容

    课程主题

    《基于视觉的机器人抓取-从物体定位、位姿估计到抓取位姿估计》

    课程提纲

    1、视觉机器人抓取的流程与关键技术

    2、物体定位技术的研究

    3、物体位姿估计的方法解析

    4、2D平面与6D空间机器人抓取位姿估计

    5、方法对比与未来的研究方向和挑战

    讲师介绍

    杜国光,达闼科技3D研发负责人,北京师范大学博士,研究方向为3D视觉感知,包括3D检测/分割、物体6D位姿估计、机器人抓取等,在相关期刊和会议上发表论文二十余篇。

    直播信息

    直播时间:11月6日20:00

    直播地点:智东西公开课小程序

    答疑地址:智东西公开课讨论群

    加入讨论群

    本次课程的讲解分为主讲和答疑两部分,主讲以视频直播形式,答疑将在「智东西公开课讨论群」进行。

    加入讨论群,除了可以免费收看直播之外,还能认识讲师,与更多同行和同学一起学习,并进行深度讨论。

    添加小助手糖糖(ID:hitang20)即可申请,备注“姓名-公司/学校/单位-职位/专业”的朋友将会优先审核通过哦~返回搜狐,查看更多

    责任编辑:

    展开全文
  • 点击上方“3D视觉工坊”,选择“星标”干货第一时间送达“一眼就能学会动作”,或许对人而言,这样要求有点过高,然而,在机器人的身上,这个想法正在逐步实现中。马斯克(Elon Musk)创...
  • Date:2020-7-14作者:小毛来源:公众号【3D视觉工坊】原文链接:基于点云的机器人抓取识别综述欢迎加入国内最大的3D视觉交流社区,1700+的领域从业者正在一起学习~机器人作为面向未来的智能制造重点技术,其具有可...
  • Date:2020-7-14作者:小毛来源:公众号【3D视觉工坊】原文链接:基于点云的机器人抓取识别综述欢迎加入国内最大的3D视觉交流社区,1700+的领域从业者正在一起学习~机器人作为面向未来的智能制造重点技术,其具有可...
  • 基于点云的机器人抓取识别综述

    千次阅读 2020-07-14 07:00:00
    点击上方“3D视觉工坊”,选择“星标”干货第一时间送达机器人作为面向未来智能制造重点技术,其具有可控性强、灵活性高以及配置柔性等优势,被广泛应用于零件加工、协同搬运、物体抓取与部件装...
  • 作者:小毛来源:公众号 @3D视觉工坊链接:基于点云的机器人抓取识别综述 机器人作为面向未来的智能制造重点技术,其具有可控性强、灵活性高以及配置柔性等优势,被广泛的应用于零件加工、协同搬运、物体抓取与部件...
  • 作者:小毛来源:公众号 @3D视觉工坊机器人作为面向未来智能制造重点技术,其具有可控性强、灵活性高以及配置柔性等优势,被广泛应用于零件加工、协同搬运、物体抓取与部件装配等领域,如图1-1所示。然而,传统...
  • 机器视觉 机器人 智能抓取
  • 点击上方“新机器视觉”,选择“星标”干货第一时间送达机器人作为面向未来智能制造重点技术,其具有可控性强、灵活性高以及配置柔性等优势,被广泛应用于零件加工、协同搬运、物体抓取与部件装配等领域,如图1-...
  • 关注极市平台公众号,回复加群,立刻申请入群~来源|3D视觉工坊机器人作为面向未来智能制造重点技术,其具有可控性强、灵活性高以及配置柔性等优势,被广泛应用于零件加工、协同搬运、物体抓取与...
  • 文章目录前言一、物体抓取研究现状1、物体抓取研究现状2、抓取检测情况分析-2D二、目前研究1. Cornell抓取数据集1. Cornell数据集介绍2. Cornell数据集下载3. Cornell数据集训检测网络4.... 基于视觉的机器人
  • 在此问题基础上,本课题提出一套基于ROS(机器人操作系统)视觉定位机械臂智能抓取系统,使抓取目标初始位姿和最终位姿被严格限定问题得到解决。首先,采用张正友算法标定RGB-D相机,获取其内外参数;其次,采用...
  • 大盘点|基于RGB图像下的机器人抓取

    千次阅读 2020-02-23 08:00:00
    点击上方“3D视觉工坊”,选择“星标”干货第一时间送达前言近期读取了一些最新基于RGB图像下的机器人抓取论文,在这里分享下思路。1、Optimizing Correlated Grasp...
  • 作者:Tom Hardy ...本文综述了基于视觉的机器人抓取技术,总结了机器人抓取过程中的四个关键任务:目标定位、姿态估计、抓取检测和运动规划。具体来说,目标定位包括目标检测和分割方法,姿态估计包括基...
  • 基于视觉引导搬运机器人多目标识别及抓取姿态研究 硕士毕业论文 第2章基于视觉引导搬运机器人系统设计 本文档是CAJ格式,请下载对应免费软件阅读。只有第2章精华部分。不是全部论文,请下载朋友注意下。想看...
  • 抓取论文汇总,作者总结得很全面,也一直在更新。 地址:https://github.com/GeorgeDu/vision-based-robotic-grasping 以下是目录
  • 本文综述了基于视觉的机器人抓取技术,总结了机器人抓取过程中的四个关键任务:目标定位、姿态估计、抓取检测和运动规划。具体来说,目标定位包括目标检测和分割方法,姿态估计包括基于RGB和R...
  • 采用本文提出机器人手- 眼视觉与超声波测距相结合检测装置, 以及融合二维图像信息与深度信息进行工件识别与抓取的方法,具有算法简单、计算量小、可靠性高等特点
  • 视觉传感器能直观反映物体外部信息,但单个摄像头只能获得物体二维图像,立体视觉虽能提供三维信息,但对于外形相同,仅深度有差别物体难以识别(如有孔物体、阶梯状物等) ,且对环境光线有一定要求. 由于超声...
  • 点击上方“3D视觉工坊”,选择“星标”干货第一时间送达摘要抓取物体堆叠和重叠场景中特定目标是实现机器人抓取的必要和具有挑战性任务。在本文中,我们提出了一种基于感兴趣区域(RoI)机...
  • 本文转自雷克世界(ID:raicworld)编译 | 嗯~阿童木呀在本文中,我们探讨了用于基于视觉的机器人抓取操作的深度强化学习算法。无模型深度强化学习(RL)已经在一系列具有挑战性的环境中得到了成功应用,但算法的...
  • 为提高机器人的抓取能力,实现非结构化环境中物体定位、识别和抓取,提出了一种定位识别算法,旨在使用机器人已观察到结果来预测未经测试的抓取的工作过程。为验证算法精确性,利用重复抓取动作时实际数据...

空空如也

空空如也

1 2 3 4 5
收藏数 87
精华内容 34
关键字:

基于视觉的机器人抓取