The Stick Figure Dance Company

Objective Complete - Mini Debriefing

In this task, we defined a class for our stick figures. We moved the code that accesses the 3D coordinates and the drawing of the dancer to our class to simplify our drawDancer() method in the main sketch. The new class also enabled us to draw more than one dancer.

To prevent our dance company from looking like an army of clones, we added the randomize() method, which scales the stick figures a bit and chooses a random color for each one.

Mission Accomplished

In this project, we taught Processing how to see by connecting the Kinect and installing the

OpenNI framework.

We used the depth image provided by the SimpleOpenNI framework and located the user in the image using the tracking capabilities of OpenNI. For every tracked user, the framework provides us with a bitmap defining which pixels belong to the tracked user and which don't. For the last two tasks, we used the depth image with the colored pixels that we created in the second task as an HUD display to enable the player to see what is currently being tracked.

Starting with our third task, we used the skeleton tracker to access the 3D coordinates of the player's limbs and joints and used the information to draw a stick figure that followed the player's moves.

In our final task, we refactored the drawing code for our stick figure to a class and used this class to draw multiple dancers so that our stick figure had some company while dancing.

You Ready to go Gung HO? A Hotshot

Challenge

In this project, we only scratched the surface of the possibilities that Kinect offers. It's impossible to describe all of it in one project; you could write a whole book about it. So try to expand the knowledge you've gained in this project to try one of the following:

52

The Disco Dance Floor

Your Hotshot Objectives

Our project consists of four tasks, namely:

Mission Checklist

This mission is about creating awesome visuals for music, so we obviously need some tunes to play and a sound card. These pre-requisites shouldn't be too hard to find. If you want to complete the final task of this mission, you will also need the final sketch from our previous mission and a Kinect, but the first three tasks can be followed independent of our previous mission.

At the time of writing, Processing 2.0 Beta doesn't support the Linux audio framework PulseAudio out of the box. If you are using a Linux distribution that uses PulseAudio, like for example, Ubuntu, you need to add the Java PulseAudio libraries to Processing. You can find detailed instructions on how to install the necessary libraries in the Processing ticketing

system at http://code.google.com/p/processing/issues/detail?id=930.

Can you hear me?

In the first task for this mission, we will write a Processing sketch that plays an MP3 file and uses the Minim library to access the played samples. We will use this data to write an oscilloscope-like curve. We will then use the fft class of Minim to calculate the frequency spectrum of the currently played sound and generate a bar graph to visualize it.

Engage Thrusters

Let's give Processing ears:

1. Create a new Processing sketch and add a setup() and a draw() method.

void setup() {

}

void draw() {

}

2.Add the import statements for the Minim library by navigating to Sketch | Import Library ... | minim.

54

Project 3

3.Now we need to add an MP3 file to our sketch by navigating to Sketch | Add File ...

or by simply dropping it onto the sketch window.

4.Add an AudioPlayer object to your sketch and initialize it in the setup() method.

import ddf.minim.spi.*; import ddf.minim.signals.*; import ddf.minim.*;

import ddf.minim.analysis.*; import ddf.minim.ugens.*; import ddf.minim.effects.*;

Minim minim;

AudioPlayer player;

void setup() { size(600, 300 );

minim = new Minim( this );

player = minim.loadFile( "loop.mp3", 1024);

}

void draw() {

}

If you don't use the MP3 file provided with the example code, be sure to replace "loop.mp3" with the filename of the MP3 you want to use.

5.To start our player, we add a mousePressed() method that calls the loop() method if the player currently isn't running and pause() if otherwise.

void mousePressed() {

if (player.isPlaying()) { player.pause();

} else { player.loop();

}

}

6.Now we create an oscilloscope display by adding the following code to our draw() method:

void draw() { background(0); stroke(0, 255, 0);

for ( int i=0; i< player.bufferSize()-1; i++) {

55

The Disco Dance Floor

float x1 = map( i, 0, player.bufferSize(), 0, width );

float x2 = map( i+1, 0, player.bufferSize(), 0, width );

line( x1, 75 + player.mix.get(i)*75, x2, 75 + player.mix.get(i+1)*75 );

}

}

7.So far we have accessed the sample data of our audio data to generate visualizations. Another very useful tool for analyzing and visualizing audio data is the so called fast Fourier transformation—fft for short—which calculates the frequency spectrum of our sound at a given time. We use thelogAverages() method to reduce the number of frequency bands we get. We add a new variable to our sketch and initialize it in the setup() method.

Minim minim; AudioPlayer player;

FFT fft;

void setup() { size(600, 300 );

minim = new Minim( this );

player = minim.loadFile( "loop.mp3", 1024);

fft = new FFT( player.bufferSize(), player.sampleRate()); fft.logAverages(11, 1);

}

8.In our draw() method, we update the spectrum and iterate over the frequency bands the fft object has calculated for us by adding the following code:

void draw() { background(0); stroke(0, 255, 0);

for ( int i=0; i< player.bufferSize()-1; i++) { float x1 = map( i, 0, player.bufferSize(), 0, width );

float x2 = map( i+1, 0, player.bufferSize(), 0, width );

line( x1, 75 + player.mix.get(i)*75, x2, 75 + player.mix.get(i+1)*75 );

}

56

Project 3

noStroke(); fill(255, 255, 0);

fft.forward( player.mix ); float step = width/8;

for ( int i =0; i < 8; i++) { float value = fft.getAvg(i) * 3;

rect( step * i, height - value, step, value );

}

}

9.Now run the sketch and click inside the sketch window to start the music. The visualizers should look like in the following screenshot:

Objective Complete - Mini Debriefing

In this task, we have imported the Minim library into our sketch and added an MP3 loop. We used the sound player object to access and manage the starting and stopping of the music and used the mixer to access the samples and visualized them as an oscilloscope-like line.

In step 7, we added an fft object to get access to the frequency spectrum of the sample that's currently playing. In this sketch, we used the frequency data to generate a bar graph of the frequency spectrum. The number of frequency bands we can access using the fft object depends on the buffer size of our AudioPlayer. If you increase the buffer, you get a higher accuracy in the frequency domain, but you also need more samples to calculate the buffer, so your visualization will be less responsive. You also need more CPU power to calculate the spectrum if you increase the number of frequency bands. We need only eight bands for the visualizers we are creating in the next task, so we told Minim to calculate the average over several frequency bands.

57