IConDraw is a small Java drawing application that is totally disconnected from input events (except for the standard Swing components around it). It just describes the kind of data it expects to be controlled by the user, by declaring four 'Drawing tool' devices (everything in ICon is seen as a device). Those devices (freehand, new line, new rectangle, and eraser) can receive positional values and brush attributes.
01_lowpass.avi (18 Mb): A menu item in IConDraw opens the input editor and displays the default configuration, based on a mouse and a keyboard. This example shows the part that controls a global pointer. This pointer (a 'Cursor' device) controls all swing standard components together with IConDraw drawing tools (which are not shown here). Low-pass filters are inserted before the pointer in order to get a 'smooth drawing' technique. The new configuration can be tested immediately (video frame rate is a bit low here).
02_straight.avi (26 Mb): Here, 'smooth drawing' is replaced by a 'constrained drawing' technique, by using the same kind of low-level reconfiguration.
straight.avi (12 Mb): Using keyboard modifiers, we can now draw horizontal or vertical lines with the freehand tool.
03_tablet1.avi (19 Mb): The user wants to use its graphical tablet instead of a mouse. All available input devices are listed on the left pane. He just has to drag the device he wants to use, and connect it.
04_tablet2.avi (21 Mb): Its graphical tablet being connected, the user also wants to make use of its pressure sensing capabilities. In the default configuration, keyboard keys are used to increase or decrease size & color components of a shared brush device. Here, the stylus pressure is directly connected to brush size through a scaling device.
tablet2.avi (14 Mb): Stylus pressure in action.
05_dedicated.avi (52 Mb): Using multiple dedicated devices is quite straightforward with ICon. Drawing tools can be instanciated without limit. Here, a freehand and an eraser tool are added to the configuration, and are connected to tablet's stylus and eraser, through cursor devices. A cursor device simulates a mouse cursor by drawing itself on the top of Java window. Multi-pointer interaction is easily achieved by using more than one cursor device. Multiple pointers also work with standard Swing components.
dedicated.avi (14 Mb): Dedicated devices in action: the three tools are concurrently controlled by stylus tip, stylus eraser and a mouse.
06_bimanual.avi (12 Mb): We are back to the default configuration. The example shows the part that controls the 'new line' tool: one pointer is used to validate both ends of a line with the standard technique. If the user wants a continous control over the line being created, he just has to connect the global pointer (mouse) to one end, and its tablet device to the other one.
bimanual.avi (12 Mb): Lines are created using both hands.
.jpg)
07_speech(2).avi (22 Mb): In the default configuration, brush attributes are controlled by keyboard keys. Here, the user loads another configuration where the keyboard is replaced by speech command devices. The vocabularies are simply specified through property boxes.
speech.avi (26 Mb): The user changes brush size and color using spoken commands such as 'bigger', 'smaller', 'red', 'blue', etc.
08_toolglass.avi (18 Mb): ICon also provides full interaction techniques, such as the toolglass device. This device activates or deactivates tools, and renders itself on top of the application. The configuration in this example uses a toolglass interaction technique. Again, IConDraw absolutely does not know how it is controlled, and does not know about the toolglass technique in particular.
09_toolglass2.avi (4 Mb): Toolglass location and transparency are controlled by a stylus (left hand). The mouse selects and uses the tool (right hand).
toolglass.avi (4 Mb): Bimanual use of the toolglass.
floating_quikwriting.avi (1 Mb):
Existing Swing applications can be controlled with ICon through 'Swing' devices. The 'JComponent' device receives standard positional events, whereas subclasses such as 'JText' or 'JScrollbar' can be directely controlled through their model (e.g. internal data). Non-standard interaction, such as voice control, can be built using those specialized devices.
'Floating QuikWriting' is an ICon device based on Ken Perlin's text input method for PDAs. It's an 'Adapter' device which allows text input in java text components with a positional device.
The floating version of QuikWriting is well-adapted to proofreading tasks: using this technique, the user can perform small corrections while navigating in a text. A single gesture is enough to move the caret and insert or delete characters.
This sample shows text correction on a Swing text component, using a stylus.
In a past paper we talked about extending Swing toolkit for multi-device interaction, and presented several accessibility techniques for controlling a 'mouse' cursor with alternative input. Those techniques are now supported under ICon.
The video above shows the Speech Cursor technique we implemented at this time, which emulates direct manipulation with commands such as 'go up', 'go right', 'slower', 'stop', 'press' or 'release'. Note how a double cursor technique is used to adress recognition delay problems: when the user starts to speak, part of the cursor (C1) stops and the other part (C2) keeps moving. If a command is recognized, C2 joins C1 and the command is performed. If nothing is recognized (noise), C1 joins C2 which keeps moving.