Paper Walkthroughs

CVPR 2017 · NeurIPS 20173D Object Detection

PointNet & PointNet++: learning directly on point sets

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation · PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

How do you feed an unordered set of 3D points to a neural network? PointNet's answer — a shared MLP and a max pool — became the building block inside almost every LiDAR detector that followed.

CVPR 20173D Object Detection

MV3D: flatten the LiDAR, then fuse with the camera

Multi-View 3D Object Detection Network for Autonomous Driving

Before detectors learned on raw points, MV3D turned the LiDAR scan into bird's-eye and front-view images, proposed 3D boxes with a 2D CNN and fused them with the camera — the image-style baseline every later LiDAR detector had to beat.

CVPR 20183D Object Detection

VoxelNet: a mini-PointNet inside every voxel

VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection

VoxelNet replaced hand-crafted bird's-eye-view features with a small PointNet run inside each voxel, followed by 3D convolutions and an RPN — the first end-to-end LiDAR detector and the template for every voxel-based method after it.

Sensors 2018, 18(10):33373D Object Detection

SECOND: skip the empty voxels

SECOND: Sparsely Embedded Convolutional Detection

SECOND keeps VoxelNet's design but computes convolutions only where there are points, fixes the angle loss with a sine, and pastes ground-truth objects into training scenes — three changes that made voxel detectors fast and became defaults.

CVPR 20193D Object Detection

PointPillars: drop the height axis

PointPillars: Fast Encoders for Object Detection from Point Clouds

PointPillars makes every voxel as tall as the scene, encodes each pillar with a one-layer PointNet and hands a 2D pseudo-image to a plain 2D CNN — no 3D convolution anywhere, and 3D detection at 62 Hz.

CVPR 20193D Object Detection

PointRCNN: proposals grown from segmented points

PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud

What if a detector never builds a grid at all? PointRCNN segments the foreground points of a raw LiDAR scan, grows a 3D proposal from each of them, and refines every proposal in its own canonical frame — a two-stage detector on a PointNet++ backbone.

CVPR 20203D Object Detection

PV-RCNN: sparse voxels for proposals, point sets for refinement

PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

Voxels or points? PV-RCNN stops choosing: a SECOND-style sparse voxel CNN finds the proposals, a small set of keypoints summarises its features with PointNet++ set abstraction, and a grid of points inside each box pulls them back for refinement.

CVPR 20213D Object Detection

CenterPoint: objects as points in bird's-eye view

Center-based 3D Object Detection and Tracking

Anchor boxes never fit a car that is turning. CenterPoint drops them: a heatmap finds object centres in BEV, a few small heads read off size, heading and velocity at each peak, and tracking becomes closest-point matching.

AAAI 20213D Object Detection

Voxel R-CNN: refinement from voxels alone

Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection

Do we really need raw points to refine a box? Voxel R-CNN pools RoI features straight from the sparse 3D voxel volumes, with a voxel query in place of the ball query — PV-RCNN accuracy at about three times the speed.

Reading roadmap · 3D Object Detection

  1. 01

    PointNet & PointNet++ CVPR 2017 · NeurIPS 2017

    How do you feed an unordered set of points to a neural network? The building block behind every point-based detector.

  2. 02

    MV3D CVPR 2017

    Before learning on raw points — project the LiDAR to a bird's-eye view, treat it like an image, and fuse it with the camera.

  3. 03

    VoxelNet CVPR 2018

    End-to-end detection from raw LiDAR: split space into voxels, learn a feature per voxel, then convolve. The origin of voxel-based detectors.

  4. 04

    SECOND Sensors 2018

    Most voxels are empty — so skip them. Sparse 3D convolution makes VoxelNet fast, plus the ground-truth sampling trick everyone still uses.

  5. 05

    PointPillars CVPR 2019

    Drop the height axis entirely. Vertical pillars + a 2D CNN give real-time 3D detection.

  6. 06

    PointRCNN CVPR 2019

    The point-based answer: segment foreground points first, propose boxes from them, then refine — a two-stage detector without voxels.

  7. 07

    PV-RCNN CVPR 2020

    Why choose? Voxels for efficient proposals, points for precise refinement — the point–voxel fusion that topped the benchmarks.

  8. 08

    CenterPoint CVPR 2021

    Objects as points in BEV: predict a heatmap of centers instead of anchor boxes — simpler, rotation-friendly and great for tracking.

  9. 09

    Voxel R-CNN AAAI 2021

    Do we really need raw points for refinement? Voxel RoI pooling keeps PV-RCNN's accuracy at a fraction of the cost.