Strategies for "Control Module" Interface Design
by Eric Medine aka MKultra
this is a work in progress--edit list follows
This essay is designed to provide an outline and working strategy
for designing software around a hardware control platform such as
a remote control, game controller, or any HID (Human
interface device). Particular attention has been paid to ensuring
an interface that "feels right", that is, it is intuitive
and easy to use-- control panels (whether hardware or software)
should be as easy to use as a large stick, or a skull-sized rock.
For the purposes of this discussion, I have designed a step by
step guide for designing and programming a software video mixer
to interact with a control module (in this case, a Nintendo Wii-mote).
I chose to use Isadora for the software platform because it is a
fairly affordable, cross-platform module-based software, has a flexible
interface, and can be used for audio and video applications. I chose
a Wii-mote because it's trendy.
There are two parts to this guide:
1) Designing and programming a software video mixer-- the "interface"
2) Configuring a Wiimote to act as a control module for the software--
the "control module"
The end result should be an intuitive interface that anyone can
use with a minimum of instruction. It should be simple, intuitive,
and fun to use. All functions should be well documented, and self-evident--
any type of action taken on the controller should produce an immediate
About this tutorial
This is more of a strategy guide than a tutorial-- I assume a certain
level of technical knowlege of both hardware and software design
and programming conventions. I have designed this as more of a "why"
than a "how-to" guide. Use this tutorial however you want,
feel free to add to it, be sure to forward it to whomever and pass
it along for free. If you charge some lazy bastard for the strategies
spelled out in this tutorial, you suck. If you apply the material
cited in a performance project or a job and you make money-- you
rock! If you have some questions, advice, or praise, contact me
What you will need
- A Wiimote, of course (approximately £20/35$, online or at
your local video games shop).
- Isadora software
- A working knowlege of Photoshop, programming languages, and visual
Using the wiimote with Resolume (very excellent walkthru, complete
with midi scripts for RSLume that unfortunately I have not quite
been able to get to work):
My wiimote to Midi tutorial
1) What processes (effects, properties, etc) do you want to apply
to the video using the software interface?
2) What inputs (buttons, movements, etc) do you have available? How
can you get the most out of your control module?
||STEP ONE: What do you want to do with your video?
Two common methods of manipulating video:
FX and combination (mixing) techniques:
1) Applying Effects (FX)
We've all seen this stuff: color changes, kalidescopes, stretch
or reposition the screen. Ideally it should be possible to add an
effect to a video file, adjust the amount of effect, and apply it
either to the particular channel. This relates to the next method,
- where will these effects be applied? Where is the video to be
affected coming from-- one signal, or to multiple sources of video?
If the FX is applied to one signal, then should it be able to be
applied all at once or as a matter of degrees? Keep in mind that
the possiblity of multiple streams of video requires the possibility
of applying FX to one, two, or more streams to all of them, or none
2) Combining Video
Secondary considerations --some of which are their own art
--need a library of videos
--need a selection method to interact with the library
--there should be styles of combining video; i.e.wipes, difference,
fades, etc (this relates to the previous category of Applying Effects)
--there should be a way to control a timeline for each and any video
on display, or in the library. It would be nice if these settings
could be saved.
||STEP TWO: How do you want to set up the controller?
About the Wii
Wiimote features 12 buttons or input types: Four of
them are arranged into a directional pad, and the rest are spread
over the controller. As well as force feedback and the integrated
mini-speaker, the true innovation is the presence of a gravity and
acceleration sensor (the ADXL330, used in the army or automobile
safety, for airbags systems) inside the Wiimote which collects the
movements and acceleration of the Wiimote on each of the 3 axis,
X, Y et Z.
Basically the input types can be broken up into
two categories: switches (on/off) and range or positioning
1) Range and positioning (from now on, movement)
X-axis (up and down), Y-axis (side to side), and Z-axis (in and
out, like a pool cue)
These controls would seem to be ideal for range
values like a volume, position, or color control, but the fact that
it is impossible to affect one of the axis without affecting all
the others makes this function particularly limited. For the wii
in particular, the movement conrol is suited to one range at a time--
only up or down, in or out, or left and right). Humans seem to have
a hard time giving a separate value toward up and left, or
down and right in real time with a real object-- do you swoop
the control one way to the next? or twist it as you move it? This
can also be referred to as a problem for micro-control, that
is, an interface that is sensitive to a range of values that humans
are unable to correctly input. As a side note, in and out
movements (poking, like a pool cue) are seldom used-- humans seem
to prefer waving to poking when using a remote control.
These exaggerated swooping movements do not exactly
lend themselves to micro-controls, so the interface elements that
require incremental range values ("go to next video,
go to previous video") are better suited for the switches such
as the D-pad (up down left right), the + and - buttons, the A and
B, or the 1 and 2 switches. It would be appropriate to state that
their placement on the controller implies their range
of movement of values in the interface. The motion controls
are best suited to a simple, limeted range of values-- more color/less
color, more volume/less volume, or timeline control (video scratching),
but this seems to work best if the controller is doing only one
of these things at a time. Thus, any motion control must be
coupled with an appropriate switch.
On, off, left right, up down, A button, B button, and trigger.
These are the most flexible of inputs (this is where micro-values
come into play) since difference is a more tangible concept
than value. As we may recall from step one, a library
of selection is necessary for a robust interface, so whatever
switches on the wii that feel as if they could precisely
access videos in the video library would work well here-- we don't
want to use a vague controller like motion as a selection
device-- we want to feel as if we have moved precisely from one
specific video to another.
I suggest the d-pad (up down left right) for this
application. It has the advantage of having multiple switches in
one location, so any interface elements that are similar (more on
grouping interface items later) can be placed here with the result
that the control module can have distinct zones of control.
It has the added advantage of scope-ability, that is, its functions
can change in range but not in activation methods. For instance:
If I have a folder with a hundred videos in it, I would use the
D-pad to move up and down that list-- click up is previous video,
click down is next video, as if a button click is turning a page
in a book. The advantage of this is that once this protocol is established,
the click left and right buttons can effect not a next and previous
function, but a meta-movement-- for instance "click
up" can go to the next video, but "click right" can
jump to the next ten videos, etc. In this way, one controller
with a similar programming scheme can control a range of movement.
This is truly the most flexible of all.
||STEP THREE: conclusion--make it all work together
We seem to have three main requirements: Select
videos in a library, combine any one of them together, add effects
to any video in the library, and add effects to any of them together.
1) Select videos
We have determined that there needs to be a way to scan through
and select libraries of video. It is imperative that a range control
be able to "land" on a specific video, so the vagueness
and imprecision of the motion controllers make them out of the question.
The + and - buttons might be suitable since they "look"
like they should be previous and next, but we quickly run into the
issue of what happens if we have multiple channels of video-- how
to ensure that these buttons control one or another channel of video?
Toggle between libraries? That would require a way to select a particular
video from one or more video libraries, a way to show which library
is selected at any particular time. This is possible, but an easier
solution would be to use the D-pad-- it has four buttons in an x
and y axis, which can be parsed as two sets of buttons: up and down
can be "previous and next" for one video library, and
left and right can be "previous and next" for another
2) Select and apply different effects
I suggest the + and - controls-- we don't have so many effects (after
doing all this interface work, I kinda don't feel like building
a large effect library). Let's leave it as, + is apply the next
effect, - is the previous. There should be a way to apply the effect
or not-- let's save that for the trigger, which is best suited to
"on" and "off" functions since it is isolated
on the controller.
We have left the 1 and 2 buttons, and the A and
B. Perhaps these can be another form of "meta-control",
possibly a contextual menu. How would that work?