G1: Week 2
tl;dr
- Created up-to-date G1 models for Isaac Lab and Mujoco Playground and got them wiggling around -- TBD how accurate they are when going sim2real
- Created TPU booties for my robot (fresh off the 3D printer - haven't tried walking with them yet!)
Learnings
- Both Isaac and Mujoco had G1 models but they were the 29DOF version with articulated hands (not my 23DOF with rubber hands), so spent most of my robot time tinkering with MJCF (xml) files for Mujoco or USD for Isaac
- Figured out how to roll a new environment/gym/task from scratch with those new models on both platforms
- As a newcomer to the field, I'm surprised how much information about the robot lives outside of the URDF/MJCF/USD that is imported via script (e.g. motor properties)
- I suspect there is a opportunity around improving the real2sim process - feels like it could/should take under 5 minutes if you have the robot on hand along with its CAD files
Isaac Lab learnings
- I really wish more of the environment was experienceable without running the sim (or with a sim that is already running) so every little change didn't require the temporal startup cost
- The conversion to USD is quite opaque (I still don't understand why the USD is split across multiple files, all of which seem to have non-relative path references that break if you move the files to a different location?)
- My wariness of USD binaries toned down a bit once I realized you could export the files to human readable .usda using usd-core
- Once the sim is running, it feels way faster than MJX and the default to an interactive viewer is great
- The cfg/mdp mindset continues to grow on me
Mujoco learnings
- I love have everything in .ipynb notebooks with human-readable XML files
- I don't love how long it takes to jit compile the simulation
- Training seems to go way slower than Isaac (might be my perception or ineptitude?)
- Devs responsive on github
Potential next steps
In no particular order...
- Train and deploy a very, very simple policy for G1 (e.g. raise arms)
Setup mujoco sim so we can experiment without turning on the robot(done)- Swap out AVP for Quest in their teleop repo
Design and print some booties for the G1's feet so I can walk it around outside without tracking dirt back in the office(testing)- Programmatically access LIDAR and RealSense and record dataset for offline exploration of current SLAM / splatting techniques
Addendum (01 April)
A colleague asked what the goal was and I'm transcribing response here for future-me and others who ask
I think my 3-month goal is to be able to go from idea to deployed policy in under a day (existent proof) with the added challenge (or follow-on goals) of(a) utilizing the lidar/rgbd inputs as part of it(b) incorporating 3D printed augmentations (per the booties in the log)
I'm guessing if I reach that point, I'll have (a) new next steps/goals and (b) a sense for how I can integrate this into the framework of capitalism (or techno-feudalism if you're a yanis fan).