Skip to content
← All writing

Electromyo-Sleeve: moving a cursor with muscle signals

  • TensorFlow
  • Embedded
  • Flutter
  • Dart
  • Python
  • Accessibility

Source →

Where this came from

If you can’t use a mouse, the cursor is the problem.

Every alternative input method is an attempt to find some signal a person can still control reliably and map it onto the same two coordinates and two buttons. Eyes are one answer. Muscles are another — and muscle activity has a property eyes don’t: it’s still there even when the movement isn’t. A muscle can fire without producing usable motion, and that electrical activity is measurable.

That’s what electromyography reads. The idea behind the sleeve was to put those sensors somewhere a person would tolerate wearing all day, and turn the signal into cursor control.

How it works

The chain runs from skin to screen, and every link is a different discipline:

  1. EMG sensors on the sleeve pick up electrical activity from the muscle
  2. A transmitter sends it wirelessly, so the wearer isn’t tethered to the machine
  3. The computer receives the stream and feeds it through a custom TensorFlow model
  4. The model outputs coordinates
  5. PyAutoGUI applies them to the cursor

Dart and Flutter handle the app side, Python the model and the cursor.

The interesting problem is step 3. Raw EMG is noisy, drifts as the wearer moves or sweats, and varies enormously between people and between placements of the same sensor. Getting from that to a stable coordinate is not signal processing you can write down as a formula, which is why it’s a learned model rather than a threshold.

The feature that changed what it is

The sleeve also watches for abnormal EMG activity, and alerts the wearer or a designated caregiver when it sees it — seizure detection.

That addition is small in code and large in consequence, because it inverts who the device is for. Cursor control is something the wearer operates deliberately. Seizure detection runs when the wearer is, by definition, not in a position to operate anything, and its output goes to somebody else entirely.

Once you’re already collecting continuous muscle-activity data from someone with a motor condition, the data needed to notice a seizure is data you have. Not using it would have been the strange choice. It does mean the device carries an implicit promise — that someone will be told — and a device making that promise has to be right about it far more carefully than a cursor does.