Learning algorithms

Learning algorithms allow to train humanoids to perform different skills such as manipulation or locomotion. Below is an overview of general approaches to training machine learning models for humanoid robots with example applications. Typically you need a simulator, training framework and machine learning method to train end to end behaviors.

Physics engines

Physics engines are software libraries designed to simulate physical systems in a virtual environment. They are crucial in a variety of fields such as video games, animation, robotics, and engineering simulations. These engines handle the mathematics involved in simulating physical processes like motion, collisions, and fluid dynamics. For a much more comprehensive overview see Simulately.

PhysX

PhysX is a physics engine by NVIDIA used primarily for video games and real-time simulations. It supports rigid body dynamics, cloth simulation, and particle effects, enhancing realism and interactivity in 3D environments.

MuJoCo

MuJoCo (Multi-Joint dynamics with Contact) is a physics engine designed for research in robotics and biomechanics. It's known for its speed, accuracy, and ease of use, making it popular for simulating complex systems with robotics and articulated structures.

Bullet

Bullet is a physics engine supporting real-time collision detection and multi-physics simulation for VR, games, visual effects, robotics, machine learning

Simulators

Isaac Sim

Isaac Sim is NVIDIA’s simulation platform for robotics development. It’s part of their Isaac Robotics platform and uses advanced graphics and AI to create realistic simulations.

Isaac Sim Features

• Advanced Physics Simulation: Includes PhysX and Flex for detailed simulations of physical interactions like rigid bodies, soft bodies, and fluids.
• Photorealistic Rendering: Uses NVIDIA RTX technology to make environments and objects look incredibly realistic, which is great for tasks that need vision-based learning.
• Scalability: Can simulate multiple robots and environments at the same time, thanks to GPU acceleration, making it handle complex simulations efficiently.
• Interoperability: Works with machine learning frameworks like TensorFlow and PyTorch and supports ROS, so you can easily move from simulation to real-world deployment.
• Customizable Environments: Lets you create and customize simulation environments, including importing 3D models and designing different terrains.
• Real-Time Feedback: Provides real-time monitoring and analytics, giving you insights on how tasks are performing and resource usage.

Isaac Sim Applications

• Robotics Research: Used in academia and industry to develop and test new algorithms for robot perception, control, and planning.
• Autonomous Navigation: Helps simulate and test navigation algorithms for mobile robots and drones, improving path planning and obstacle avoidance.
• Manipulation Tasks: Supports developing robotic skills like object grasping and assembly tasks, making robots more dexterous and precise.
• Industrial Automation: Helps companies design and validate automation solutions for manufacturing and logistics, boosting efficiency and cutting down on downtime.
• Education and Training: A great educational tool that offers hands-on experience in robotics and AI without the risks and costs of physical experiments.

Isaac Sim Integration with Isaac Gym

Note: Isaac Gym is now deprecated. NVIDIA now forwards users to their improved toolkit, Isaac Lab, also built on top of Isaac Sim.

Isaac Sim works alongside Isaac Gym, NVIDIA’s tool for large-scale training with reinforcement learning. While Isaac Sim focuses on detailed simulations, Isaac Gym is great for efficient training. Together, they offer a comprehensive solution for developing and improving robotics applications.

Training frameworks

Popular training frameworks are listed here with example applications.

Isaac Gym

Isaac Gym is NVIDIA's robotics simulation tool, part of the Isaac SDK. It leverages GPU acceleration to enable the simulation of thousands of robot bodies simultaneously, making it highly efficient for training machine learning models in robotics. It's designed to streamline robotics applications, focusing on reinforcement learning in a virtual environment.

Gymnasium

Gymnasium is an open-source toolkit for developing and comparing reinforcement learning algorithms. Originally developed by OpenAI as "Gym," it provides a standardized set of environments (like Atari games, robotic simulations, etc.) to test and benchmark AI algorithms. It's widely used in the AI research community to foster innovation and replication in RL studies.

Applications

Over the last decade several advancements have been made in learning locomotion and manipulation skills with simulations. See non-comprehensive list here.

Training methods

Imitation learning

Imitation Learning is a technique where models learn to perform tasks by mimicking expert behaviors. This approach is often used when defining explicit reward functions is challenging. It accelerates learning by using pre-collected datasets of expert demonstrations, reducing the need for trial-and-error in initial learning phases.

Reinforcement Learning

Reinforcement Learning involves agents learning to make decisions by interacting with an environment to maximize cumulative rewards. It's foundational in fields where sequential decision-making is crucial, like gaming, autonomous vehicles, and robotics. RL uses methods like Q-learning and policy gradient to iteratively improve agent performance based on feedback from the environment.