Generative Visuals using Processing for Android
by Eric Medine aka MKultra

This is a tutorial on how to compile generative visuals built in Processing (a Java based authoring environment) onto an Android mobile device. Discussion includes downloading and running the Android and Processing SPK and API, as well as issues such as canvas size, processor power, and external display capabilities of mobile devices. The mobile device used in this demo is an HTC EVO, with 1080x760 HDMI output mirroring using the Full HDMI application by SAM BERRO. Authoring environment is a MacBook pro with 4gigs of ram with 2.5 MHZ processor.

This video is a little processing experiment I modified from some other processing experiments on OpenProcessing.org.

About Processing
from Processing.org:
Processing is an open source programming language and environment for people who want to create images, animations, and interactions. Initially developed to serve as a software sketchbook and to teach fundamentals of computer programming within a visual context, Processing also has evolved into a tool for generating finished professional work. Today, there are tens of thousands of students, artists, designers, researchers, and hobbyists who use Processing for learning, prototyping, and production.

About Android
Android is an open-source software stack for mobile devices that includes an operating system, middleware and key applications. Google Inc. purchased the initial developer of the software, Android Inc., in 2005. Android's mobile operating system is based upon a modified version of the Linux kernel, and it feels quite similar to JAVA in terms of syntax and structure.

About this tutorial
"Processing for Android" involves configuring a set of libraries that can be bundled with the Processing programming environment. This allows you to port most of your processing experiments onto an Android device running OS Froyo or later...I think. It takes care of most of the configuring and compiling, and even can install it onto your phone or mobile device for you.

Use this tutorial however you want, feel free to add to it, be sure to forward it to whomever and give it away for free. If you charge some lazy bastard for this tutorial, you suck. If you use it for a performance and you make money-- you rock! If you have some questions, advice, or praise, contact me at:
info(at)ericmedine(dot)com

BEWARE! there's a section here that's about rooting your Android device in order to run an HDMI output-- basically you have to re-install HDMI drivers on your device. This is not for the faint-of-heart, and although it worked fine for me with no problems, don't blame me if you brick your gear!

What you will need
- Processing environment and compiler.
- Android SDK
- Mobile device running Android (Froyo or greater)
- Video display capable of HDMI input

Other resources:
Processing experiments:
http://www.openprocessing.org/
Publishing your Android application:
http://developer.android.com/guide/publishing/preparing.html
Wiki for Processing for Android:
http://wiki.processing.org/w/Android
Full HDMI by Sam Bero:
https://market.android.com/details?id=com.orrebmas.fullhdmi
My website:
http://www.ericmedine.com

STEPS:
1) Install Processing and the Android SDK
2) Build your Processing application
3) Preview on the emulator

4) Save your application to your mobile device
5) Show your application output on an external display

STEP ONE: Install Processing and the Android SDK

There's a lot that goes into preparing a workstation to handle software development. Some people like their dedicated programming environments, like Eclipse, some people don't. For this demo we're just going to be concentrating on setting up your workstation for the basics necessary to get you up and running.

First, you want to install Processing on your computer. Go here:http://wiki.processing.org/w/Android#Download and follow the instructions to install. This is not the standard download-- it includes libraries necessary for working with Android. Although you'll be doing all of your editing in Processing (I do, anyway) there are some libraries that Processing needs to include when it compiles to your emulator or actual device, so you'll need to actually install the Android SDK and download some stuff.

Go ahead and install the Android SDK here:http://developer.android.com/sdk/index.html

Once everything is installed, open up the Android SDK and AVD Manager (basically your library manager) and look at the “Available packages". Look for the check box for “SDK Platform Android 2.1, API 7” underneath “Android Repository”, check it, and install it. You do not need the “ADT Plugin for Eclipse” (unless you want to use the Eclipse environment for your Android apps). If you are a fancy software developer you probably have Eclipse already.

You can install other versions of the Android API as well, but you have to include version 7. You can't install version 8 without 7 just because it's newer. Also note that these revisions are for the individual "SDK Platforms" (which refer to Android OS versions), which is different from the revision numbers on the "Android SDK Tools". Confusing? Yes. Supposedly future versions of Processing won't need you to do this, for all I know the most current does not.

Anyway, at this point you should have everything you need-- Processing and the Android extras you need to compile to your device. Next step: build your application!


STEP TWO: Build your processing application

Alright, open it up and build something!

1) Setup our sketch! Since we're doing this on a phone you supposedly don't have to set the screen size, but I like to do it anyway-- the EVO is 800x450, so that's what we're going to make it.


void setup() {
size(480, 800);
smooth();
}

On the other hand.... maybe you are planning to roll this out to different devices. So, you can do this:

import android.util.DisplayMetrics;

Then, put this in your setup method. It should display your current size metrics:

void setup() {
DisplayMetrics dm = new DisplayMetrics();
getWindowManager().getDefaultDisplay().getMetrics(dm);
float density = dm.density;
int densityDpi = dm.densityDpi;
println("density is " + density);
println("densityDpi is " + densityDpi);
}

What's that? It's throwing an error? It doesn't know what Display Metrics are? That's because this is an android-specific library, and we haven't turned this into an android app yet. Be patient.

Speaking of android-specific libraries, one thing to keep in mind-- we don't want to be shifting the orientation when using the phone, right? If we're video mixing and someone bumps the phone, the signal will go black when it shifts from horizontal to vertical, or whatever. So let's lock it down:

void setup() {
size(480, 800);
smooth();
orientation(PORTRAIT); // the hamburger way
orientation(LANDSCAPE); // the hot dog way
}

That probably threw an error as well. Comment it out, and save it for later.

2) So.... here's a basic sketch that will totally work in Processing, and then we'll port it to Android. After that's done, you can put whatever android specific parameters in there and it won't throw any errors (probably).

void setup() {
size(480, 800);
smooth();
// orientation(PORTRAIT); // the hamburger way, commented out for later
// orientation(LANDSCAPE); // the hot dog way, commented out for later
}

void draw() {
if (mousePressed) {
fill(0);
} else {
fill(255);
}
background(0);
ellipse(mouseX, mouseY, 80, 80);
}

So basically we've written some code that will fill the screen with black if the mouse is pressed... then afterward, draw a white circle underneath the mouse. A circle that follows the mouse.... not exactly rocket science, but it works.

3) While we're at it, wouldn't it be nice to turn off incoming calls while using the sketch? Yes, it would, but it's not really possible right now. Mixing Processing libraries with regular Android libraries is apparently not recommended-- the drawing model is incompatible, and although it might work in some cases, it will be inconsistent across other devices and/or Android OS versions. Processing for Android is basically endabled by an Android "Activity" wrapper, which means that some basice OS functions aren't really accessible from Processing for now.

Don't get me started on using the camera, either...

STEP THREE: Preview on your emulator

If we press the run button (or hit Apple-R), Processing compiles our sketch to run as a temporary Java applet, which we see running in a separate window. This is the basic Processing behavior. I have a few other options for compiling my little sketch. I could select ‘Sketch > Present’ from the menubar to present the sketch in fullscreen, I could select ‘File > Export’ to compile my sketch as a Java applet to display in the browser, or I could select ‘File > Export Application’ to produce a stand-alone file to launch like a ‘real’ application. These three basic options for compiling (run, present, export) have slightly different functions in Android Mode. Wait. What? Android Mode?

Here we discover the genius of our new, super Processing – we can build sketches the same way that we are normally used to, then switch into Android Mode to preview and publish our droidified sketches. Let’s do that with our simple circle sketch. From the ‘Android’ menubar heading, select Android mode (CTRL-D, or Apple-D, or whatevs). Let’s first see what the sketch circle looks like in the emulator, by pressing the run button.

Holy cow, that's freakin slow to boot up. But now you know what the emulator looks like! Congratulations! You have made a Processing application for Android!

Keep in mind, the "android mode" is where the functionality that ONLY works while in Android mode appears-- orientation, import android.util.DisplayMetrics, etc. Makes debugging tough sometimes when you're in a hurry...

This brings me to other Processing subtleties-- anytime you want to use images, seems like you need to put them in the root level with your sketch. But in Android mode, there's another step or two. Let's turn off Android mode, then add an image to our sketch.

I've saved a PNG called "bground.png" the same size as the sketch (android likes the background images to be the same size as the canvas) and placed it in the root level with the sketch.

void setup() {
size(480, 800);
smooth();
img = loadImage("bground.png"); //// this loads an image
}

void draw() {
if (mousePressed) {
fill(0);
} else {
fill(255);
}
background(img); /// this places the image
ellipse(mouseX, mouseY, 80, 80); /// draws the circle on top of the image
}

 
 

Then I added the image into the Processing sketch with the code. Test it with Android mode turned off. Works great, right? Now let's turn Android mode back on (CTRL-D, or Apple-D, or whatevs), and run it in the emulator. Hey, it's throwing an error-- why? Null pointer exception? Do you need to set paths to some weird location?

 
No... have to do Sketch>Add file. Then, it knows where the image is. Subfolders work as well, so this is where you do asset management.

Try it now!

Other Android specific stuff... XML tricks! Your XML files hold app configuration tags, you can modify these to allow or deny functionality to your app. Be sure to modify the copy in your sketch folder, not the file created by using Export or Export to Application. Those will be replaced next time you use the Export function. In some cases, you may want to make tweaks to the exported folder, but be sure to move it to a new location first so that you don't inadvertently lose changes when using the Export command again.

In particular, this is where you do permissions! Look at this! Android> Sketch Permissions. See if you can find where to grant your app permission to brick your phone!

STEP FOUR: Export to your mobile device
Very Important: To get the app running on your device, you’ll first have to make sure that USB debugging is turned on on your mobile device. You can do this from the Settings>Applications>development menu on your device. If your workstation/pc is running on Windows, you might have to do some other setup, too. Connect your Android device, and then select ‘Sketch>Present’ from the menubar (or, press Shit-Apple-R on a mac). Takes a million years, but you can see the thing on your gear.

And.... here you are! Processing on your phone!

You'll notice that the icon on your phone is a pretty boring blue blob. Maybe you want to have something customized? It's easy! All sketches use a default icon unless you specify otherwise, so to create your own, save images as icon-36.png, icon-48.png, and icon-72.png. These should be 36x36, 48x48, and 72x72 pixel icons, respectively.

Place them in the sketch/root level folder (not the data folder or any other subfolder). The next time you build your app, Android will use these instead of the defaults as if by freakin magic! You don't have to build a path to them or anything.

STEP FIVE (but you should probably do this first): Using FULL HDMI to display your app with an HDMI port

So... now that we have stuff working on our devices, how the hell to we get a signal out?

Depending on the type of device you have, you may not be able to export video to a display! The HTC Desire, insight, and the EVO all have video outputs... but they are not capable of displaying anything but "approved" content. Basically, hardware manufacturers get paid extra by content providers (sprint, youtube, whatever shitty media company is trying to get in the streaming business) to play only their content. So although I can play YouTube and sprint nascar and sprint football and sprint TV (of course, I wouldn't know how well they work because I've never run the motherfucker) I can't just run my phone into my TV with my HDMI cable and surf the web widescreen style, or play a video game on my TV from my phone. My TV will ONLY get a signal from the device I've purchased if I'm playing "approved" content on an "approved" application. This is known as "defective by design".

Unless.... you install Full HDMI by Sam Bero! Keep in mind that on some devices, you may have to root them to get full functionality. Check the Full HDMI website for particulars on your device... I had to do it for my gear, and although I'm not going to go into the pros and cons of jailbreaking/rooting your android device, it's really easy. Here's a more in-depth tutorial, it's what I used to root my own phone: http://www.goodandevo.net/2010/09/how-to-flash-a-zip-on-your-rooted-htc-evo-4g.html. I even used some of the images from their site for this tute, thanks guys.

The point of rooting a smartphone is to give the owner full control and access over the device. The reason we need to do this is that the actual hardware drivers for the HDMI port will only output a signal if it's "signed" by the software. Since we want it to work on our terms, not when some crippleware application tells it to, we need to replace the actual HDMI drivers. This is a little more complicated than updating printer drivers for the PC, but it's not so bad.

First, you'll need to download the new "kernel"-- drivers and whatnot to enable access to the HDMI port.

Download the HDMI kernel here: http://www.mediafire.com/?m7iao05o1777nm1 (this is the one I used)
or....
Download the sense kernel here : http://tinyw.in/O2
Download the cm7 kernel here: http://tinyw.in/O1

Once you've downloaded, put the new kernel on your SD card (if you're on a MAC you can just copy the new kernel file to the SD card like copying a file to a drive). If you have a PC you'll have to install QuickBoot to put stuff on your SD card.

Next, you'll have to "root" your phone in order to mount the new drivers.

I used Unrevoked One Click to root my phone:http://www.unrevoked.com/. Download, then follow the directions to "root" the phone. Once that's done, you need to use the "root" menu to mount the driver. Power off your phone completely, then hold the Vol-Down button and power back on while keeping the Vol-Down button pressed. This will take you to the bootloader.

From there go to "recovery". It should take you to something with green text like a console (menu). You want to select something that mounts your SD card to your pc, something like "Toggle SD Card" or "MS-Something". Once your SD card is mounted to your PC, transfer the HDMI_Kernel.zip (or whatever it's called) to the scdard from your pc. Unmount your sdcard by hitting the power button on your phone once, this brings you back to the green text menu. From there select flash zip from sdcard option (or something similar). This should take you to a list of files on yoru sdcard, select the zip and flash it. If you get an error saying something about signature, then you have to toggle signature checking off in the menu (I think its right after you select to flash a zip from main menu).... walkthru adapted from late-nite text support by sam bero

Scary? No, it's fun! Naturally, I can't guarantee results-- your experience may vary! In fact, it is possible (in theory) to brick your phone doing this, but you'd have to be really trying or semi-drunk for that to happen. Don't drink and root, kids. If in doubt, there are a lot of walkthrus and tutorials on how to do this process.

Now that's done, get Full HDMI by Sam Bero and install it. This is what it looks like... now you're good to go!