K-Scale Teleop

A minimal implementation of a bi-manual remote robotic teleoperation system using VR hand tracking and camera streaming.

Feature	Status
VR and browser visualization	✔ Completed
Bi-manual hand gesture control	✔ Completed
Camera streaming (mono + stereo)	✔ Completed
Inverse kinematics	✔ Completed
Meta Quest Pro HMD + NVIDIA® Jetson AGX Orin™ Developer Kit	✔ Completed
.urdf robot model	✔ Completed
3dof end effector control	✔ Completed
Debug 6dof end effector control	⬜ Pending
Resets to various default poses	⬜ Pending
Tested on real world robot	⬜ Pending
Record & playback trajectories	⬜ Pending

Setup

git clone https://github.com/kscalelabs/teleop.git && cd teleop
conda create -y -n teleop python=3.8 && conda activate teleop
pip install -r requirements.txt

Usage

• Start the server on the robot computer:

python demo_hands_stereo_ik3dof.py

• Start ngrok on the robot computer:

ngrok http 8012

• Open the browser app on the HMD and go to the ngrok URL.

Dependencies

• Vuer is used for visualization
• PyBullet is used for inverse kinematics
• ngrok is used for networking

Citation

@misc{teleop-2024,
  title={Bi-Manual Remote Robotic Teleoperation},
  author={Hugo Ponte},
  year={2024},
  url={https://github.com/kscalelabs/teleop}
}