Training a high-quality gesture recognizer requires providing a large number of examples to enable good performance on unseen, future data. However, recruiting participants, data collection and labeling, etc. necessary for achieving this goal are usually time-consuming and expensive. Thus, it is important to investigate how to empower developers to quickly collect gesture samples for improving UI usage and user experience. In response to this need, we introduce Gestures à Go Go (G3), a web service plus an accompanying web application for bootstrapping stroke gesture samples based on the kinematic theory of rapid human movements. The user only has to provide a gesture example once, and G3 will create a model of that gesture. Then, by introducing local and global perturbations to the model parameters, G3 generates from tens to thousands of synthetic human-like samples. Through a comprehensive evaluation, we show that synthesized gestures perform equally similar to gestures generated by human users. Ultimately, this work informs our understanding of designing better user interfaces that are driven by gestures.
Try the G3 app online, no software required!
Among other things, you'll be able to create gestures, organize them in collections, export collections and even create a ready-to-use gesture recognizer. G3 incorporates 3 gesture recognizers of the so-called $-family. Currently, 14 combinations of recognizer and programming language are available.
Although we highly encourage using the G3 app, you can use the web service on your own.
Here we describe the main API methods, which allows to synthesize any number of gestures samples using only one gesture example as input.
The API is JSON REST, so all requests have Content-Type: application/json.
POST https://g3.prhlt.upv.es/synthesize
Deprecated!
Creates a number of synthetic gestures using one gesture example as input. Warning: This method may not be available in the future.
Gesture synthesis is now performed via 2 explicit steps: First, reconstruction, which creates a model of the input gesture. Then, generation, which uses the gesture model to introduce variations and produce different samples.
| gesture (required) | Array | Sample gesture as a sequence of strokes: [ [[x,y,t], ... ], ... ] |
| num_samples (optional, default: 10) | Integer | Number of desired synthetic samples, between 1 and 999. |
| variability (optional, default: 1.0) | Float | The degree of variability (max. 1.0) that synthesized gestures should have (0.0 means no variability). |
| rate (optional, default: 100) | Integer | Sampling rate, in Hz, between 50 and 500. |
| errors | Array | List of errors (if any) encountered while synthesizing gestures. |
| result | Array | Synthesized gestures. |
POST https://g3.prhlt.upv.es/reconstruct
Creates a model of a given gesture example. This method allows to inspect the gesture model prior to generating synthetic samples.
| strokes (required) | Array | Sample gesture as a sequence of strokes: [ [[x,y,t], ... ], ... ] |
| errors | Array | List of errors (if any) encountered while reconstructing the gesture. |
| result | Object | config (model parameters, Object[]) and snr (a quality measure, Float). |
POST https://g3.prhlt.upv.es/generate
Takes as input a reconstructed gesture (model parameters) and generates a number of gesture samples.
| reconstruction (required) | Object | Model parameters as received by the /reconstruct API method. |
| nsamples (optional, default: 10) | Integer | Number of desired synthetic samples. |
| shapevar (optional, default: 1.0) | Float | The degree of geometric variability that synthesized gestures should have (0.0 means no variability). |
| lengthvar (optional, default: 0.0) | Float | The degree of stroke length variability that synthesized gestures should have (0.0 means that all strokes have the same length). |
| sametimes (optional, default: false) | Boolean | Whether the synthesized gestures should have the same timestamps as the original gesture. |
| errors | Array | List of errors (if any) encountered while generating the gesture samples. |
| result | Array | Synthesized gestures: [ [ [[x,y,t], ... ], ... ], ... ] |
The following are the datasets we used on our experiments. All downloads are zipped CSV files, one file per gesture.
Each file contains 5 columns, space-separated.
| stroke_id | An integer identifying each gesture stroke, starting at 0. |
| x | X coordinate. |
| y | Y coordinate. |
| time | Relative timestamp, in milliseconds. The first timestamp of the first stroke starts at 0. |
| is_writing | An integer indicating if the stroke is a pen-down (1) or not (0). |
If you use the G3 app, the API, or any of the synthesized datasets in your work, please cite us:
LaTeX users can use the following BibTeX entry for citation:
@Article{g3,
author = {Luis A. Leiva and Daniel Martín-Albo and Réjean Plamondon},
title = {Gestures à Go Go: Authoring Synthetic Human-like Stroke Gestures Using the Kinematic Theory of Rapid Movements},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2015},
volume = {7},
number = {2},
}