I'm building a keyboardless keyboard

TL;DR: One phone above a bare desk, no keyboard, no headset. A small sequence model reads my hand movements and a local language model turns them into words. On sentences it had never seen, it got 73% and 82% of words right in two blind tests. Along the way I learned that my camera sits at the one angle nearly every published system avoids, and that most of the ideas I tried stopped working once I ran the right control. So far it only works on me, and near the end there’s a section on everything that’s wrong with the approach. The code is open source.

I want to go out without my laptop and still be able to check on my coding agents. See what happened, give them another instruction, leave them running. That’s usually all I need.

But writing that instruction on my phone is annoying enough that I start thinking about bringing the laptop. Which rather defeats the point.

With smart glasses for a screen and a computer running somewhere else, I can imagine leaving most of the hardware at home. The part I got stuck on was typing. I’ve spent years learning to do it without looking at the keys. It seems a shame to throw away that muscle memory.

So I started trying to teach a camera to read my fingers.

I did look at the other options

Tap Strap 2 lets you type with finger combinations, but you have to learn its alphabet. I don’t particularly want to learn to type again. I want to keep doing the thing my hands already know.

Samsung showed a phone-camera concept called SelfieType for CES 2020. TypeNet studied typing on flat surfaces with one camera, too. Neither became something you can use today: SelfieType stayed a concept, and TypeNet a research paper. Still, so much for my original title, “the world’s first keyboardless keyboard.”

The serious one is Meta and ETH’s TouchInsight, which shipped as an experimental “Surface Keyboard” on the Quest 3. It is much better than mine: 37 words per minute with 2.9% uncorrected error. It is also a very different kind of system.

The last row surprised me most. The model design is almost the same. Meta’s advantage isn’t a cleverer network, it’s four cameras that can measure how high a finger is, and millions of labelled touches. It’s not released either: no code, no weights, no data.

What I want is the simpler version. No headset to wear, nothing strapped to your hands: the phone you already carry acts as the keyboard’s sensor. That constraint is the whole project, and it’s also what makes it hard.

Getting both hands in the picture

This began as a project for hand gestures and computer shortcuts. Then I got distracted by the possibility of removing the keyboard altogether.

The plan was to record myself typing on a real keyboard, use the keypresses as labels, and then try the same movements on a bare desk. One camera. I didn’t want a second phone or a microphone listening for taps.

Before any of that, I spent far too long getting a usable camera feed.

On the iPhone itself, the wide-angle view showed both hands. On the computer, I could only see part of them. In my setup, Continuity Camera wasn’t exposing the ultra-wide lens, and Desk View cropped the wrong part of the scene.

That led to WideCam, a small iPhone app streaming the ultra-wide camera over Wi-Fi. The capture test gave us 1280 × 720 at about 60 fps, with timestamps. Both hands were finally visible.

I had set out to make a keyboard and had made a camera app instead. Anyway, on to typing.

The tap detector thought I was typing when I wasn’t

The first approach was to find each tap, identify the finger, and work out the key.

It would sometimes register taps while my hands were resting. When I moved a hand, several fingers appeared to press at once. The finger moving most wasn’t necessarily the one typing, either. It might already be reaching for the next key.

More recordings and better motion features helped, but a better tap detector still didn’t give me readable sentences on the desk.

I also made one recording where I just typed whatever came into my head. Afterwards, there was no reliable transcript to compare the predictions with. We had footage, but no way to tell how well it worked. For the next tests, I used known phrases or supplied what I’d typed.

I’ve been working on this with Claude Code and Codex. They’ve helped build the tools and run a lot of experiments. Some of those experiments were useful; quite a few just ruled things out.

Reading the movement as a sequence

The change that helped most was reading the movement as a sequence, instead of needing to find every tap first.

The current model follows hand landmarks—the estimated positions of the finger joints, from Google’s MediaPipe—over time. It predicts possible characters using CTC, a method that can learn from a whole sequence and its transcript without a label for the exact frame of every letter. That matters here, because on a bare desk nobody knows the exact frame a finger “pressed”.

The concrete version:

• Model: convolutions feeding a small Transformer, about 1.2 million parameters. Six of them are averaged: three trained only on keyboard typing, three also fine-tuned on desk phrases.
• Data: 29 minutes of my own typing on a real keyboard, 4,698 keystrokes, labelled by a keylogger. Plus 31 short phrases typed on the bare desk, labelled by me.
• Pretraining: How We Type, a public motion-capture dataset of 30 people typing. Mixing it into training was one of the biggest single gains: word accuracy on the desk, with no desk training at all, went from about 52% to about 75%.
• Decoder: a local language model (Qwen3 8B) proposes candidate sentences, and each is checked against the hand-motion evidence and a vocabulary built from my own writing.
• Hardware: everything trains and runs on a Mac mini. Training one model takes a few minutes.

Every gap is a tiny finger movement

In the September 15 recording above, I typed:

every tap is a tiny finger movement

It came back as:

every gap is a tiny finger movement

I’m pleased it got that close. I’m also fairly sure I would get tired of correcting it.

The t and g keys are close together, so similar finger movements are an obvious suspect. I haven’t yet tested whether that’s what caused this particular error.

But look at what happens after the wrong letter: “gap” is a word. Spellcheck won’t complain. It fits grammatically, even though “tap” makes more sense here. With another sentence, both words might make sense.

This is the problem with relying on context to finish the job. It can give me readable text that isn’t what I wrote.

The clip uses actual model output. The overlays were added afterwards, and the fingertip highlights are approximate illustrations of the character predictions. They aren’t measurements of which finger pressed each key. This also isn’t a live speed demonstration.

Blind tests

A development score is easy to fool yourself with, so I ran blind tests. I typed sentences of my own on the bare desk, the system decoded them, and only then did I reveal what I’d written.

Those are small samples: the second one’s plausible range is roughly 70–98%. Three of the eleven sentences came back perfect. The misses were mostly names and unusual words (“bifi”, “claude”, “div”) and whole look-alike words, like “deceived” for “achieved”.

I kept asking whether a bigger model would fix it

If it gets most of the letters, surely a language model can work out the rest? Especially if it knows what I’m working on?

That was my thinking. We tried larger local language models, vocabulary-guided search, more candidate words and swipe-inspired motion matching.

The decoder did help. Between the two blind tests I added a larger local model, my personal vocabulary and camera models fine-tuned on desk phrases; run on the second test’s recording, the old pipeline scored 63% against the new one’s 82%. But then we measured something more useful than another decoder: for each held-out sentence, was the right answer anywhere in the list of candidates the decoder chooses from?

For 12 of 17 sentences, the exact sentence wasn’t in any list. The best any decoder could possibly have done was about 88% of words, and it was already picking about 85%. No language model can choose a word the camera never proposed.

One useful thing did come out of that: the step where the large model invents candidate sentences turned out to be unnecessary. A plain dictionary search over the camera’s letter guesses covers exactly the same correct words, which cuts decoding time by about two thirds.

The ideas that didn’t survive a control

Most of what I tried looked like it worked, until I compared it against a version that shouldn’t have worked.

A contact model from Meta. PressureVision++ estimates, from an ordinary photo, where a hand is pressing on a surface. Its authors used it to build a tabletop keyboard, typing with the index finger, and reached 25.8 words per minute against 14.4 for a pose-based baseline. That’s exactly my missing piece, so I ran it over my keyboard recordings, where the keylogger says exactly when each key went down.

It scored 0.66 at telling presses from hovers, where 0.5 is a coin flip. Then the control: the same model run on a single averaged image of the whole session, with every trace of per-frame appearance removed, scored 0.69. It wasn’t seeing contact. It was reacting to where the fingertips were. That’s not a knock on the model: it was trained for a camera at 45 degrees on someone else’s rig, and mine points straight down. A small model trained on my own keyboard footage scored 0.90.

Pixels. Adding image crops of the hands gave a small improvement. Shuffling those images out of time order gave the same improvement.

A better hand tracker. RTMPose tracks fingertips with up to 40% less jitter than MediaPipe on my footage. Retraining the whole thing on its landmarks changed the error rate by two words out of 250.

Each of these would have made a nice paragraph. None of them made the text more accurate.

Did I put the camera in the wrong place?

The pattern behind all of this looked like geometry.

When I checked how other systems film hands, almost none of them point straight down. PressureVision’s camera is at 45 degrees. TouchInsight looks down and forward from a headset. From directly overhead, a finger pressing a few millimetres moves almost entirely along the camera’s line of sight. On my setup that’s a change of well under a pixel. The one thing that separates a press from a hover is the thing a top-down camera sees worst.

So we simulated it. Using EgoPressure, a public dataset of 3D hand meshes with real touch measurements, we re-projected the same hands through a virtual camera at every angle and measured how well each view separates presses from hovers:

Tilting helps with contact but makes it much harder to tell which key was hit: keys are about 19 mm apart. Top-down is the best angle for position and the worst for contact.

Those were press gestures, though, not typing. So we ran it again on How We Type, 30 people’s real ten-finger typing captured in 3D, filmed through a virtual camera at different angles. This measures an easier task than the table above, so compare the columns, not the two tables:

At 45 degrees far fewer fingers are hidden, and the text gets no better. On real footage of three people from Meta’s PressureVision dataset, filmed from several angles, the contact signal did improve, from 0.85 to 0.90 at 40 degrees, but nothing suggests that’s worth re-recording everything for. So the answer to the heading is: probably not. Every single angle trades one problem for another. That’s why Meta uses four cameras. Extra hardware would help me too, but the whole point is to need nothing except the phone.

The words I actually use

A company name or a code identifier is often the part of an instruction that has to be exact. If I type USE_SPECIFIC_WORD_HERE, I don’t want autocorrect to replace it with something more normal.

The decoder already has a local personal vocabulary. It can consider terms from my own writing, rather than just common English words. The keyboard test transcripts are excluded when building that vocabulary, so it isn’t simply being given those answers.

For onboarding, I’d like to try a week of selected everyday typing. Or let someone supply relevant context from Notion, emails or Slack. If a project name appears throughout the document I’m working on, the decoder ought to consider it.

I’d accept sharing that context to have less to correct. I’d want to keep the useful terms and their frequencies locally, rather than retain everything I typed. I wouldn’t call that anonymous, though: names and unusual vocabulary can still identify someone.

I suspect a fairly small vocabulary covers a lot of my daily work. I haven’t measured how much. The question is whether that familiarity helps with new sentences, especially when I say something unexpected.

Spelling, punctuation and capitals

For prose, I’d let the language model suggest spelling, capitals and punctuation. My vocabulary should help it leave names alone. That’s a tradeoff I’m comfortable exploring for messages to my coding agents.

Actual code is less forgiving. userID and userId might refer to different things. An underscore or quotation mark can change whether something works. I need a way to enter those literally, pick an identifier from the project, or confirm an uncertain character.

That part isn’t built yet. The current character set is too limited for reliable code entry. There’s an expanded alphabet ready for training, but having more symbols in a model doesn’t mean it has learned to recognize them.

Can this read your password?

Reading keystrokes from video is a known attack; this doesn’t invent it. But it’s a narrow version of it. It needs a camera directly above your hands, about 23 cm away, and a model trained on your labelled typing. Nobody knows how well it works on a stranger; the closest published comparison, reading typing from wrist signals, went from 7% character error for a user’s own model to 52% on people it hadn’t trained on.

It’s also weakest exactly where passwords live. Most of the accuracy comes from the dictionary and the language model. On a random string there’s nothing for them to work with, and before that step roughly a third of characters come out wrong. Everything runs locally, and the camera only ever sees your hands.

What’s wrong with my approach

It’s easy to write a post like this and leave the weak spots out, so here they are.

• It’s one person. Every model here is trained and tested on my hands, my desk and my phone. The closest published comparison, reading typing from wrist sensors, went from 7% character error on a user’s own model to 52% on people it hadn’t seen. I have no reason to think mine would do better.
• The tests are small. The two blind tests add up to 11 sentences. One wrong word moves the score by several points.
• I tested on the same recordings too often. AI coding agents made it cheap to run dozens of experiments, and most of them ran against the same few desk sessions. Some of the gains I kept are probably luck, and the development numbers are optimistic. Only the blind tests are clean.
• The sentences were easy. Short, lowercase, typed without corrections. Real typing has backspace, capitals, punctuation and code.
• The language model does a lot of the work. That’s how “tap” became “gap”: the output reads well whether or not it’s right. For 12 of 17 held-out sentences, the exact sentence wasn’t among the candidates at all.
• The setup is fragile. When the phone was mounted slightly differently one day, key accuracy fell to 53%; an automatic re-alignment step brought it back to 70%. A different desk, different light or a different phone hasn’t been tried.
• The best version isn’t live. The accurate decoder needs about half a minute per phrase.
• A lot of the camera analysis is simulated, on other people’s hands from research datasets that also come with non-commercial licences.
• I chose top-down for convenience, not evidence. Most of what I learned since is about why it was the hardest angle to start from.

What I have now

A phone recording my hands, a recognizer that produces text, and enough of it working that I want to keep going.

There’s a live version too: raw letters appear as I type, and each phrase is decoded two to five seconds after I pause. To be that fast it uses a small language model, which gets roughly 67% of words right on the same phrases where the large one gets about 88%. The large one takes around half a minute per phrase. Closing that gap is the next job, and the dictionary-search result above suggests it’s doable.

Next: capital letters, punctuation and code, and more people than just me.

Code

Everything is on GitHub: pxshaaa/keyboardless-keyboard. That includes the WideCam capture app, the training and decoding pipeline, and the write-ups of every experiment, including the ones that didn’t work. My own recordings aren’t included, since they contain real typing.

If you’re working on camera typing or hand tracking, the README lists the open problems. The biggest are seeing contact from one camera and making it work for more than one person. Every dead end here is written up, so you don’t have to repeat them.

For now, I’m still at my desk. I’d like the next version to get me a little closer to leaving the laptop there.