Reading Pen Gesture Recognition Algorithm: How Swipes and Taps Become UI Commands

Audience: This article is for firmware engineers, UX designers, and OEM product managers who need to understand the gesture recognition algorithm running on the B1, B2, F8, K7, KF1, L5, P2, Q6, Q8, T2 and Y6 reading pens. We assume you are designing a custom menu UI or evaluating how to add gestures to an existing dictionary application, and you need to know what recognition rate, latency, and CPU load to expect.

A reading pen gesture is a sequence of touch samples captured over 30 ms to 500 ms. The firmware converts those samples into a discrete gesture code (TAP, SWIPE_LEFT, SWIPE_RIGHT, SWIPE_UP, SWIPE_DOWN, LONG_PRESS, DOUBLE_TAP, CIRCLE) that the application layer maps to a UI action. The conversion is small enough to run on any of the 11 models but tuned differently per model because the sensor resolution, sample rate, and host CPU budget all vary. This article covers the algorithm, the tuning numbers, and the failure modes buyers should plan around.

1. The Gesture Vocabulary

Across the 11 models we ship, the gesture vocabulary is consistent:

The simple-zone models (B1, B2, F8, P2, Q6, Y6) do not run gestures at all because they have no 2D touch sensor. They substitute 4 to 8 discrete button zones. This article focuses on the matrix models.

2. The Algorithm Pipeline

The gesture recognition algorithm runs in four stages. Each stage has a hard CPU budget; if any stage blows its budget the gesture is dropped (no command issued) rather than mis-recognized.

Stage 1: Sample acquisition. The touch controller reports coordinates at 80 Hz to 120 Hz. The host MCU reads them via I2C at 100 kHz to 400 kHz on K7/KF1/L5, and via SPI at 10 MHz on Q8/T2. Each sample is a tuple (x, y, t).

Stage 2: Debounce and outlier rejection. Two samples within 4 mm and 20 ms are merged into one. A sample that deviates more than 12 mm from the running centroid is dropped as an outlier. This stage takes 0.4 ms per sample on the L5.

Stage 3: Feature extraction. From the cleaned sample stream we compute 7 features: total displacement, total path length, mean velocity, peak velocity, dominant direction angle, curvature, and duration. These are the inputs to the classifier.

Stage 4: Classification. We use a template-matching scorer for the 6 directional swipes and a state-machine for the timing-based gestures (TAP, LONG_PRESS, DOUBLE_TAP). The state machine and template matcher together consume 1.6 ms per gesture on the L5.

Total budget per gesture on the L5: 0.4 + 1.6 + stage 4 classifier = 2.4 ms, plus interrupt handling overhead, leaves a 6 ms margin before the 8.3 ms inter-frame budget is exceeded.

3. The Template Matcher for Swipes

Each of the 6 directional swipes (left, right, up, down, plus the two diagonals on T2) has a template. The template is a 16-point polyline of the expected sample trajectory, normalized to a 12 mm length and centered at the origin. During recognition, we normalize the input sample stream the same way (translate centroid to origin, scale to 12 mm, resample to 16 points) and compute the mean Euclidean distance between input and template at each point.

The match score is 1 - (mean_distance / 12). A score of 1.0 is a perfect match; a score of 0.0 is orthogonal. The classifier returns the highest-scoring template, provided that score exceeds 0.78. Anything below 0.78 is classified as "unknown" and ignored.

The threshold 0.78 was tuned on a 1,200-gesture dataset collected from 30 children aged 4 to 9 across our two facilities. We measured a true-positive rate of 96.4% and a false-positive rate of 0.6% on the L5 with this threshold. Tightening to 0.82 drops false-positive to 0.2% but drops true-positive to 91.7%; loosening to 0.72 raises false-positive to 2.4% and true-positive to 98.6%.

4. The State Machine for Time-Based Gestures

TAP, LONG_PRESS, and DOUBLE_TAP are timing-dependent and cannot be classified from sample shape alone. We use a finite state machine with 5 states:

• IDLE: no touch present.
• TOUCH_DOWN: first sample received, transition to PRESS.
• PRESS: touch held, start the long-press timer.
• RELEASE: touch released, evaluate TAP vs LONG_PRESS vs DOUBLE_TAP.
• WAIT_DOUBLE: released, waiting up to 300 ms for a second tap.

Transitions are governed by 4 thresholds:

• LONG_PRESS_TIME: 600 ms (configurable 400 to 2000 ms).
• DOUBLE_TAP_GAP: 300 ms.
• TAP_MAX_DURATION: 250 ms.
• TAP_MAX_DISPLACEMENT: 4 mm.

The state machine is implemented as a switch in the touch ISR, taking 0.06 ms per transition on the L5. Total recognition latency for a TAP is the tap duration plus one inter-sample interval, which is 280 ms to 320 ms in the worst case but typically 130 ms to 180 ms.

5. CPU and Memory Cost

The full gesture pipeline (template matcher + state machine + feature extraction) on the L5 consumes 1.9 MIPS of CPU at a gesture rate of 3 gestures per second. The L5's host SoC has 240 MIPS of headroom after audio and OCR, so the gesture engine is essentially free.

Memory cost is 4.2 KB of code (in flash) and 1.1 KB of working RAM (for the 16-point template and the sample buffer). On the K7 and KF1 the same pipeline costs 5.1 KB of code and 1.4 KB of RAM because the lower-tier MCU compiles the templates with less optimization.

The Q8 and T2 add a CIRCLE gesture classifier using a 24-point Hough-transform-like loop. This adds 3.4 KB of code and 0.8 KB of RAM, and runs at 4.1 MIPS during classification. CIRCLE is a premium feature and is disabled by default on the Q8 and only enabled on the T2 unless the OEM buyer requests it on Q8.

6. Latency: Tap-to-Action and Swipe-to-Action

End-to-end latency from physical gesture completion to host action issued:

These numbers include the touch controller's report rate, the I2C/SPI transfer time, the algorithm execution, and the host application's response time.

7. Tuning for Different User Populations

The default thresholds are tuned for children aged 4 to 9. For adult users (teachers, parents) the gestures are larger and faster, which actually improves recognition accuracy to 98.5% true-positive at the same threshold. For children younger than 4, the OEM buyer can request a softer gesture profile via firmware configuration:

• LONG_PRESS_TIME raised from 600 ms to 900 ms.
• TAP_MAX_DISPLACEMENT raised from 4 mm to 6 mm.
• SWIPE_MIN_DISPLACEMENT lowered from 8 mm to 5 mm.

This profile is called KIDS_PROFILE_V1 and is available on all 11 models via a firmware build flag. An OEM program targeting the 2-to-4 age group should request this profile.

8. Acceptance Test for OEM Buyers

We recommend the following 5-test acceptance suite for gesture recognition:

1. SWIPE_LEFT recognition rate ≥ 96% on 100 trials by 3 different testers.
2. SWIPE_RIGHT recognition rate ≥ 96% on 100 trials.
3. TAP false-positive rate ≤ 1% over 1,000 non-touch seconds.
4. LONG_PRESS recognition latency between 600 and 800 ms.
5. DOUBLE_TAP recognition rate ≥ 92% on 100 trials.

Testers should be drawn from the target age group. A sample tested only by adult engineers will report inflated accuracy because adult gestures are more consistent. Our internal validation uses 30 children aged 4 to 9 plus 10 adults, and the published 96.4% true-positive rate is the children-only number.

9. Common Failure Modes

In OEM programs we have seen the following gesture integration failures:

1. The host application polls for gestures in a 50 ms loop instead of using interrupts, adding 50 ms latency. Always use the gesture interrupt, not polling.
2. The application enables DOUBLE_TAP but forgets to set DOUBLE_TAP_GAP to 300 ms, causing every tap to be classified as a double-tap.
3. The application maps CIRCLE to "next page" but the children's swipes are too small (5 mm instead of 12 mm) and the algorithm classifies them as noise. Lower CIRCLE_MIN_CIRCUMFERENCE from 12 mm to 8 mm if targeting younger users.
4. The application disables LONG_PRESS but leaves the timer running, wasting 0.04 MIPS of CPU continuously. Set LONG_PRESS_ENABLE = 0 to skip the timer.

For more on the touch sensor that feeds the gesture engine, see Reading Pen Touch Accuracy and Jitter Test. For how gestures are mapped to UI actions in the reference dictionary application, see Reading Pen Dictionary File Format and Edition. For how the host SoC schedules the gesture task against OCR and audio, see Reading Pen Firmware Architecture Explained. For OEM spec language and acceptance criteria, see Reading Pen OEM and ODM Specification Template. To request a gesture test report with 5-tester histograms from your sample batch, see /request-a-quote.

10. The Spec Language to Put in an OEM Contract

A defensible OEM specification for gesture recognition includes:

• Gesture vocabulary list with model-by-model availability.
• Template matcher threshold (default 0.78, configurable 0.70 to 0.85).
• LONG_PRESS_TIME (default 600 ms).
• DOUBLE_TAP_GAP (default 300 ms).
• TRUE_POSITIVE_RATE ≥ 96% on the 1,200-gesture child-aged test set.
• FALSE_POSITIVE_RATE ≤ 1% on the 1,000-second no-touch test.
• Gesture-to-action latency table per gesture class.

A supplier who cannot commit to these numbers is not yet running the algorithm at production quality. We publish the per-model gesture test report with each production batch at no additional cost.

Related Reading

• Reading Pen Touch Accuracy and Jitter Test
• Reading Pen Touchscreen Controller and Gesture
• Reading Pen Firmware Architecture Explained
• Reading Pen Dictionary File Format and Edition
• Reading Pen Quality Control and AQL Standards
• Reading Pen Display Luminance Uniformity

Supplementary Reading: Gesture Combos and Multi-Finger Sequences

Beyond the single-gesture vocabulary listed in Section 1, the L5, Q8 and T2 firmware supports a small set of two-finger gesture combos for power users:

• TWO_FINGER_TAP: equivalent to a back button in the dictionary app.
• TWO_FINGER_SWIPE_LEFT: equivalent to a page-back command in long-content readers.
• PINCH: zoom in or out of dictionary text by 25% per pinch cycle.

These combos are not in the default vocabulary and are disabled by default. OEM buyers can enable them via a firmware build flag, at the cost of 0.8 KB of additional code and 0.3 KB of additional RAM.

The gesture engine also supports a low-power "tap to wake" mode on the L5, Q8 and T2. In this mode the touch controller polls at 8 Hz instead of 120 Hz, consuming only 0.2 mW. When a tap is detected, the controller switches to 120 Hz within 12 ms. This mode is enabled by default on the Q8 and T2 because they have larger batteries, and disabled on the L5 because the smaller battery cannot absorb the wake-up latency penalty on every dictionary lookup.

Across the 47,000+ units shipped since 2023, gesture-related field failures account for 1.1% of all RMAs. The dominant cause is a stuck touch zone from liquid ingress (38%), followed by overlay delamination at the sensor edge (24%), and firmware lockup (18%). The 1.1% rate is much lower than the 4.2% touch failure rate because gestures are only used on the premium models that have better sealing and stricter AQL on the touch sensor.