PointWorld

Distributed 3D world model pre-training for robotic manipulation (NVIDIA + Stanford)

View on GitHub

Overview

PointWorld (NVIDIA + Stanford) is a large pre-trained 3D world model for robotic manipulation. It forecasts full-scene 3D point flow (per-point 3D displacements over ~1 second) from one or a few RGB-D images plus a sequence of robot actions.

This test case demonstrates distributed pre-training and evaluation on AWS GPU clusters orchestrated by Amazon EKS / SageMaker HyperPod EKS.

Key Features

  • 3D point flow prediction — Forecasts per-point displacements from RGB-D + actions
  • Embodiment-agnostic — Robot action represented as 3D point flow (not joint angles), enabling cross-embodiment learning
  • Multi-embodiment — Single model trains jointly across single-arm Franka and bimanual humanoid
  • Large PTv3 variant — Pre-trained on BEHAVIOR domain with PyTorch DDP
  • Validated on 2× p5en.48xlarge (16× NVIDIA H200)

Quick Start

# Default: 2-node smoke run on BEHAVIOR subset
# Scale up via env_vars knobs to more nodes and full dataset
kubectl apply -f pointworld-training.yaml