Theories and Strategies for "Control Module" Interface Design
by Eric Medine aka MKultra

this is a work in progress--edit list follows
10/28/2003
5/11/07

This essay is designed to provide an outline and working strategy for designing software around a hardware control platform such as a remote control, game controller, or any HID (Human interface device). Particular attention has been paid to ensuring an interface that "feels right", that is, it is intuitive and easy to use-- control panels (whether hardware or software) should be as easy to use as a large stick, or a skull-sized rock.

For the purposes of this discussion, I have designed a step by step guide for designing and programming a software video mixer to interact with a control module (in this case, a Nintendo Wii-mote). I chose to use Isadora for the software platform because it is a fairly affordable, cross-platform module-based software, has a flexible interface, and can be used for audio and video applications. I chose a Wii-mote because it's trendy.

There are two parts to this guide:
1) Designing and programming a software video mixer-- the "interface"
2) Configuring a Wiimote to act as a control module for the software-- the "control module"

Parameters:
The end result should be an intuitive interface that anyone can use with a minimum of instruction. It should be simple, intuitive, and fun to use. All functions should be well documented, and self-evident-- any type of action taken on the controller should produce an immediate result on-screen.

About this tutorial
This is more of a strategy guide than a tutorial-- I assume a certain level of technical knowlege of both hardware and software design and programming conventions. I have designed this as more of a "why" than a "how-to" guide. Use this tutorial however you want, feel free to add to it, be sure to forward it to whomever and pass it along for free. If you charge some lazy bastard for the strategies spelled out in this tutorial, you suck. If you apply the material cited in a performance project or a job and you make money-- you rock! If you have some questions, advice, or praise, contact me at:
info(at)ericmedine(dot)com

What you will need
- A Wiimote, of course (approximately £20/35$, online or at your local video games shop).
- Isadora software
- A working knowlege of Photoshop, programming languages, and visual design conventions.

Other resources:
Using the wiimote with Resolume (very excellent walkthru, complete with midi scripts for RSLume that unfortunately I have not quite been able to get to work):
http://www.resolume.com/forum/viewthread.php?tid=3102
My wiimote to Midi tutorial
wiitomidi.htm

Two common methods of manipulating video: FX and combination (mixing) techniques:

1) Applying Effects (FX)
We've all seen this stuff: color changes, kalidescopes, stretch or reposition the screen. Ideally it should be possible to add an effect to a video file, adjust the amount of effect, and apply it either to the particular channel. This relates to the next method, Combining Video.
Secondary considerations:
- where will these effects be applied? Where is the video to be affected coming from-- one signal, or to multiple sources of video? If the FX is applied to one signal, then should it be able to be applied all at once or as a matter of degrees? Keep in mind that the possiblity of multiple streams of video requires the possibility of applying FX to one, two, or more streams to all of them, or none of them.

2) Combining Video
Secondary considerations --some of which are their own art form:
--need a library of videos
--need a selection method to interact with the library
--there should be styles of combining video; i.e.wipes, difference, fades, etc (this relates to the previous category of Applying Effects)
--there should be a way to control a timeline for each and any video on display, or in the library. It would be nice if these settings could be saved.

About the Wii
The Wiimote features 12 buttons or input types: Four of them are arranged into a directional pad, and the rest are spread over the controller. As well as force feedback and the integrated mini-speaker, the true innovation is the presence of a gravity and acceleration sensor (the ADXL330, used in the army or automobile safety, for airbags systems) inside the Wiimote which collects the movements and acceleration of the Wiimote on each of the 3 axis, X, Y et Z.

Basically the input types can be broken up into two categories: switches (on/off) and range or positioning (1-100).

1) Range and positioning (from now on, movement)
X-axis (up and down), Y-axis (side to side), and Z-axis (in and out, like a pool cue)

These controls would seem to be ideal for range values like a volume, position, or color control, but the fact that it is impossible to affect one of the axis without affecting all the others makes this function particularly limited. For the wii in particular, the movement conrol is suited to one range at a time-- only up or down, in or out, or left and right). Humans seem to have a hard time giving a separate value toward up and left, or down and right in real time with a real object-- do you swoop the control one way to the next? or twist it as you move it? This can also be referred to as a problem for micro-control, that is, an interface that is sensitive to a range of values that humans are unable to correctly input. As a side note, in and out movements (poking, like a pool cue) are seldom used-- humans seem to prefer waving to poking when using a remote control.

These exaggerated swooping movements do not exactly lend themselves to micro-controls, so the interface elements that require incremental range values ("go to next video, go to previous video") are better suited for the switches such as the D-pad (up down left right), the + and - buttons, the A and B, or the 1 and 2 switches. It would be appropriate to state that their placement on the controller implies their range of movement of values in the interface. The motion controls are best suited to a simple, limeted range of values-- more color/less color, more volume/less volume, or timeline control (video scratching), but this seems to work best if the controller is doing only one of these things at a time. Thus, any motion control must be coupled with an appropriate switch.

2) Switches
On, off, left right, up down, A button, B button, and trigger.

These are the most flexible of inputs (this is where micro-values come into play) since difference is a more tangible concept than value. As we may recall from step one, a library of selection is necessary for a robust interface, so whatever switches on the wii that feel as if they could precisely access videos in the video library would work well here-- we don't want to use a vague controller like motion as a selection device-- we want to feel as if we have moved precisely from one specific video to another.

I suggest the d-pad (up down left right) for this application. It has the advantage of having multiple switches in one location, so any interface elements that are similar (more on grouping interface items later) can be placed here with the result that the control module can have distinct zones of control. It has the added advantage of scope-ability, that is, its functions can change in range but not in activation methods. For instance:
If I have a folder with a hundred videos in it, I would use the D-pad to move up and down that list-- click up is previous video, click down is next video, as if a button click is turning a page in a book. The advantage of this is that once this protocol is established, the click left and right buttons can effect not a next and previous function, but a meta-movement-- for instance "click up" can go to the next video, but "click right" can jump to the next ten videos, etc. In this way, one controller with a similar programming scheme can control a range of movement. This is truly the most flexible of all.

We seem to have three main requirements: Select videos in a library, combine any one of them together, add effects to any video in the library, and add effects to any of them together.

1) Select videos
We have determined that there needs to be a way to scan through and select libraries of video. It is imperative that a range control be able to "land" on a specific video, so the vagueness and imprecision of the motion controllers make them out of the question. The + and - buttons might be suitable since they "look" like they should be previous and next, but we quickly run into the issue of what happens if we have multiple channels of video-- how to ensure that these buttons control one or another channel of video? Toggle between libraries? That would require a way to select a particular video from one or more video libraries, a way to show which library is selected at any particular time. This is possible, but an easier solution would be to use the D-pad-- it has four buttons in an x and y axis, which can be parsed as two sets of buttons: up and down can be "previous and next" for one video library, and left and right can be "previous and next" for another video library.

2) Select and apply different effects
I suggest the + and - controls-- we don't have so many effects (after doing all this interface work, I kinda don't feel like building a large effect library). Let's leave it as, + is apply the next effect, - is the previous. There should be a way to apply the effect or not-- let's save that for the trigger, which is best suited to "on" and "off" functions since it is isolated on the controller.

We have left the 1 and 2 buttons, and the A and B. Perhaps these can be another form of "meta-control", possibly a contextual menu. How would that work?