Spectacles → SO-101 Teleop

Pinch a hand in the air wearing Snap Spectacles, and the gripper of a physical LeRobot SO-101 robot arm responds live: the hand-tracking is streamed over a relay server and polled by the control scripts, while a DH-model inverse-kinematics solver turns x/y/z targets into servo commands for the arm's joints. All built over a single weekend at a hackathon.

Technical Highlights

Hand-tracking relay — the Snap Spectacles Lens streams hand pose and pinch data over WebSocket to a Node relay server, which throttles it to ~20 samples/second into a shared status file that the robot control scripts poll.
Inverse kinematics — the inverse kinematics were solved, given a 5-DOF DH model of the LeRobot SO-101.
Closed-loop gripper control — a pinch on the Spectacles triggers movement on the robot arm: the tracked grip state is polled and converted into serial commands for the servo's on the robot arm.
Servo control — solved joint angles are converted to 12-bit absolute-encoder ticks and packaged into servo commands, sent via UART.

The system consists of three main parts: the Spectacles Lens does hand-tracking and streams it to a Node.js relay server (tunneled via ngrok) and adds the latest sample to a status file. The latest sample is parsed, the required inverse kinematics are solved, and converted into servo commands. The status is updated at a 5 Hz frequency, and converted into UART commands sent to the servos on the LeRobot SO-101. The arm position path (i.e. moving the end effector to an x/y/z target), as well as the servo commands are generated on the host.

Snap Spectacles Lens                 (hand/arm pose + pinch, on-device)
        │  WebSocket / HTTP  (via ngrok tunnel)
        ▼
Node relay  (saving_json_input.js)
        │  writes latest sample, throttled ~20/s
        ▼
status.json
        │
        ├─ polled @ 5 Hz ────────► control_with_json.py ──► Feetech serial ──► gripper (servo ID 6)
        │                                                                      [closed loop, automatic]
        │
        └─ (no live reader yet) ─► so100_control.py --x --y --z ──► so100_ik.py (DH + damped
                                    [manual / CLI-triggered]          least-squares IK) ──► Feetech
                                                                       serial ──► arm joints 1–5

Data flow from the Spectacles Lens to the physical arm — reproduced from the repo's own README.

Given a 5-DOF DH model of the LeRobot SO-101, the inverse kinematics solver calculates what angle each of the arm's five joints should be to reach a target position. It breaks the reach into a series of small steps and re-solves at each one (i.e. Cartesian path interpolation). Thus, ensuring stability for the robot arm. Once the solver has found a solution, the required joint angles are converted into servo commands for the LeRobot arm. Then the final commands are sent to the arm via UART.

The aim of this project is to demonstrate the range of abilities to make this system to work as a whole: AR hand-tracking, a networked relay, numerical inverse kinematics, and embedded serial control are each their own challenge. Getting a person's hand movement to convert to motion in a robot arm means reliably connecting these four aspects. Over the course of the CalHacks 12 weekend, we produced a working system spanning: spanning perception, networking, control theory, and hardware.