I want to go out without my laptop and still be able to check on my coding agents. See what happened, give them another instruction, leave them running. That’s usually all I need.
But writing that instruction on my phone is annoying enough that I start thinking about bringing the laptop. Which rather defeats the point.
With smart glasses for a screen and a computer running somewhere else, I can imagine leaving most of the hardware at home. The part I got stuck on was typing. I’ve spent years learning to do it without looking at the keys. It seems a shame to throw away that muscle memory.
So I started trying to teach a camera to read my fingers.
I did look at the other options
Tap Strap 2 lets you type with finger combinations, but you have to learn its alphabet. I don’t particularly want to learn to type again. I want to keep doing the thing my hands already know.
Meta and ETH’s TouchInsight is closer: camera-based typing on a surface, using a mixed-reality headset. Their study reported 37 words per minute with 2.9% uncorrected error. It also describes the difficulties of fingers obscuring each other and uncertain hand tracking, and leaves out capitalization and keys such as Shift.
Samsung showed a phone-camera concept called SelfieType for CES 2020. TypeNet studied typing on flat surfaces with one camera, too. So much for my original title, “the world’s first keyboardless keyboard.”
My version starts with one phone above a desk. I’d like it to learn how I type, including my habits and the words I use at work. So far, I’ve been testing it on myself.
Getting both hands in the picture
This began as a project for hand gestures and computer shortcuts. Then I got distracted by the possibility of removing the keyboard altogether.
The plan was to record myself typing on a real keyboard, use the keypresses as labels, and then try the same movements on a bare desk. One camera. I didn’t want a second phone or a microphone listening for taps.
Before any of that, I spent far too long getting a usable camera feed.
On the iPhone itself, the wide-angle view showed both hands. On the computer, I could only see part of them. In my setup, Continuity Camera wasn’t exposing the ultra-wide lens, and Desk View cropped the wrong part of the scene.
That led to WideCam, a small iPhone app streaming the ultra-wide camera over Wi-Fi. The capture test gave us 1280 × 720 at about 60 fps, with timestamps. Both hands were finally visible.
I had set out to make a keyboard and had made a camera app instead. Anyway, on to typing.
The tap detector thought I was typing when I wasn’t
The first approach was to find each tap, identify the finger, and work out the key.
It would sometimes register taps while my hands were resting. When I moved a hand, several fingers appeared to press at once. The finger moving most wasn’t necessarily the one typing, either. It might already be reaching for the next key.
More recordings and better motion features helped, but a better tap detector still didn’t give me readable sentences on the desk.
I also made one recording where I just typed whatever came into my head. Afterwards, there was no reliable transcript to compare the predictions with. We had footage, but no way to tell how well it worked. For the next tests, I used known phrases or supplied what I’d typed.
I’ve been working on this with Claude Code and Codex. They’ve helped build the tools and run a lot of experiments. Some of those experiments were useful; quite a few just ruled things out.
The change that helped most was reading the movement as a sequence, instead of needing to find every tap first.
The current model follows hand landmarks—the estimated positions of the finger joints—over time. It predicts possible characters using CTC, a method that can learn from a whole sequence and its transcript without a label for the exact frame of every letter. A decoder then uses those predictions, a language model and my vocabulary to choose words.
Every gap is a tiny finger movement
In the September 15 recording above, I typed:
every tap is a tiny finger movement
It came back as:
every gap is a tiny finger movement
I’m pleased it got that close. I’m also fairly sure I would get tired of correcting it.
The t and g keys are close together, so similar finger movements are an obvious suspect. I haven’t yet tested whether that’s what caused this particular error.
But look at what happens after the wrong letter: “gap” is a word. Spellcheck won’t complain. It fits grammatically, even though “tap” makes more sense here. With another sentence, both words might make sense.
This is the problem with relying on context to finish the job. It can give me readable text that isn’t what I wrote.
The clip uses actual model output. The overlays were added afterwards, and the fingertip highlights are approximate illustrations of the character predictions. They aren’t measurements of which finger pressed each key. This also isn’t a live speed demonstration.
I kept asking whether a bigger model would fix it
If it gets most of the letters, surely a language model can work out the rest? Especially if it knows what I’m working on?
That was my thinking. We tried larger local language models, vocabulary-guided search, more candidate words and swipe-inspired motion matching. Later experiments added image crops and optical flow to the hand landmarks.
Those later attempts haven’t given us a convincing improvement. In one image experiment, shuffling the images gave similar results to using them in the right order. The score had moved a little, but that didn’t mean the model was reading the movement better.
I still think context is useful. I just no longer expect it to clean up everything the camera misses.
The words I actually use
A company name or a code identifier is often the part of an instruction that has to be exact. If I type USE_SPECIFIC_WORD_HERE, I don’t want autocorrect to replace it with something more normal.
The decoder already has a local personal vocabulary. It can consider terms from my own writing, rather than just common English words. The keyboard test transcripts are excluded when building that vocabulary, so it isn’t simply being given those answers.
For onboarding, I’d like to try a week of selected everyday typing. Or let someone supply relevant context from Notion, emails or Slack. If a project name appears throughout the document I’m working on, the decoder ought to consider it.
I’d accept sharing that context to have less to correct. I’d want to keep the useful terms and their frequencies locally, rather than retain everything I typed. I wouldn’t call that anonymous, though: names and unusual vocabulary can still identify someone.
I suspect a fairly small vocabulary covers a lot of my daily work. I haven’t measured how much. The question is whether that familiarity helps with new sentences, especially when I say something unexpected.
Spelling, punctuation, Groß- und Kleinschreibung
For prose, I’d let the language model suggest spelling, capitals and punctuation. My vocabulary should help it leave names alone. That’s a tradeoff I’m comfortable exploring for messages to my coding agents.
Actual code is less forgiving. userID and userId might refer to different things. An underscore or quotation mark can change whether something works. I need a way to enter those literally, pick an identifier from the project, or confirm an uncertain character.
That part isn’t built yet. The current character set is too limited for reliable code entry. There’s an expanded alphabet ready for training, but having more symbols in a model doesn’t mean it has learned to recognize them.
What I have now
A phone recording my hands, a recognizer that produces text, and enough of it working that I want to keep going.
On the older development set, the full decoder made 25 word edits across 250 reference words from three recordings: a 10% word error rate. Those recordings were reused during development, and the scoring ignores case, punctuation, digits and some spelling differences. It tells me about progress on that data, not what a new user should expect. The new demo above is a separate recording.
Speed still needs work. One full-decoder run took about five minutes to process two minutes of footage. Streaming experiments have been faster, but less accurate.
Next I want to freeze the pipeline and try fresh recordings, including unfamiliar terms and a different camera placement. Then get text appearing while I type, with an easy way to fix the uncertain bits.
For now, I’m still at my desk. I’d like the next version to get me a little closer to leaving the laptop there.