Learning algorithms

Learning algorithms allow to train humanoids to perform different skills such as manipulation or locomotion. Below is an overview of general approaches to training machine learning models for humanoid robots with example applications. Typically you need a simulator, training framework and machine learning method to train end to end behaviors.

Physics engines

Physics engines are software libraries designed to simulate physical systems in a virtual environment. They are crucial in a variety of fields such as video games, animation, robotics, and engineering simulations. These engines handle the mathematics involved in simulating physical processes like motion, collisions, and fluid dynamics. For a much more comprehensive overview see Simulately.

PhysX

PhysX is a physics engine by NVIDIA used primarily for video games and real-time simulations. It supports rigid body dynamics, cloth simulation, and particle effects, enhancing realism and interactivity in 3D environments.

Mujoco

MuJoCo (Multi-Joint dynamics with Contact) is a physics engine designed for research in robotics and biomechanics. It's known for its speed, accuracy, and ease of use, making it popular for simulating complex systems with robotics and articulated structures.

Bullet

Bullet is a physics engine supporting real-time collision detection and multi-physics simulation for VR, games, visual effects, robotics, machine learning

Simulators

Isaac Sim

ManiSkill

VSim

Contents

Training frameworks

Popular training frameworks are listed here with example applications.

Isaac Gym

Isaac Gym is NVIDIA's robotics simulation tool, part of the Isaac SDK. It leverages GPU acceleration to enable the simulation of thousands of robot bodies simultaneously, making it highly efficient for training machine learning models in robotics. It's designed to streamline robotics applications, focusing on reinforcement learning in a virtual environment.

Gymnasium

Gymnasium is an open-source toolkit for developing and comparing reinforcement learning algorithms. Originally developed by OpenAI as "Gym," it provides a standardized set of environments (like Atari games, robotic simulations, etc.) to test and benchmark AI algorithms. It's widely used in the AI research community to foster innovation and replication in RL studies.

Applications

Over the last decade several advancements have been made in learning locomotion and manipulation skills with simulations. See non-comprehensive list here.

Training methods

Imitation learning

Imitation Learning is a technique where models learn to perform tasks by mimicking expert behaviors. This approach is often used when defining explicit reward functions is challenging. It accelerates learning by using pre-collected datasets of expert demonstrations, reducing the need for trial-and-error in initial learning phases.

Reinforcement Learning

Reinforcement Learning involves agents learning to make decisions by interacting with an environment to maximize cumulative rewards. It's foundational in fields where sequential decision-making is crucial, like gaming, autonomous vehicles, and robotics. RL uses methods like Q-learning and policy gradient to iteratively improve agent performance based on feedback from the environment.