humanoid robotic hand integration

i spent a couple weeks integrating a humanoid robotic hand into the assistive robotic arm stack, to investigate a potential replacement for our current two-finger gripper.

there are pros and cons to a humanoid hand; while it can provide more dexterity, its complexity can be a drawback if control is not up to par. my high-level objective was to determine the usefulness of a couple different humanoid hands with different characteristics with "perfect" control. this provides a couple significant insights:

•firstly, how useful is a humanoid hand, quantified by a delta in the number of tasks that are possible, and their qualitative ease.

•secondly, what is our ideal hand? for example, tendon-driven or linkages? actuated wrist? how many dofs?

a very open-ended question, but a rewarding one to answer.

with so many hands on the market, i needed to identify a couple to physically test. the methodology was to go as broad as possible, and get two hands on opposite ends of the spectrum as possible. one hand was well-known on the market, with a dual focus on prosthetics and robotics, that is linkage-driven and robust. the other hand was not as known, with a larger focus on dexterous manipulation research, but it was tendon-driven with two additional dofs of wrist rotation.

neither of the hands had off-the-shelf mechanical integration with the kinova gen3. i first designed mounts for both hands, designed to be cnc machined. compactness was a key consideration as i wanted to minimize the moment arm as to not have the kinova be the limiting factor on payload.

they were 3d printed in the interest of speed. one hand had a control breakout module which was packaged within the wrist, as if it wasn't there. the other hand had an external module so mounting was super simple.

this was a 0 -> 1 project and so the control stack was essentially nonexistent. we had previously used vive controllers for tele-operation, but a new controller was needed for per-finger control. i used a manus glove in combination with a vive tracker for 6-dof motion and finger tracking. i was responsible for getting the manus glove to talk to each of hands.

one particularly difficult challenge was thumb mapping. each of these hands had two actuated dofs in the thumb, whereas the human thumb has 5 dofs (or 6, depending who you're talking to). an additional confounding factor is that the manus streams 4 dofs, all of which are interpolated from the thumb tip position. i tried a few methods to fix this. a promising one was to run the user through a few different calibration poses, then interpolate between them to get the target thumb pose. it worked fairly well in the middle regions, but towards the edges of the range of motion, the robot thumb was incredibly jerky and it was impossible to securely hold anything.

what i landed on was to use the joint readings from the manus glove to calcualate my thumb pose using forward kinematics, then using that target value to get the "closest" position with inverse kinematics with the robot hand. this was the best balance of precision in the center of the range of motion and smoothness at the edges.

another challenge i ran into - recall that one of the hands has an actuated wrist, making the entire system overactuated. there are now multiple ways to pitch and yaw the end effector. to implement this, i wrote a simple controller to prioritize hand motion over the arm, and to only pitch the kinova once the hand extents were reached. the same is true in reverse - we would "unwind" the wrist to bring the hand back to the neutral position first.

i then used the teleoperation stack to evaluate the two hands on a few different tasks, to assess both strength and dexterity in a variety of real-world scenarios, and artificial benchmarks which we previously used with the two-finger gripper to get a baseline comparison.

results were mixed - in our simple pick-and-place benchmark, the humanoid hands were worse, as accidental thumb movements would consistently make grasping during fine manipulation less secure. however, the hand did expand the range of possible tasks. for example, operating a power drill was impossible with a gripper - but on the other hand, operating power tools isn't something we'd encourage for the users we work with. another interesting unlock is the ability to natively use a fork or spoon. we'd previously enabled this with a custom end effector interface for a dedicated utensil, but the hand could unlock more flexiblity and real-world usability by not requiring a custom tool.

the ultimate takeaway was that at the moment, bci control would be a limiting factor in usefulness. down the road, with advancements in dexterous hands (which i'm super excited about), this could be a game-changer in restoring autonomy as physical beings, for those who have lost it. i'm looking forward to the future ;)