Logfile Geo-visualizer
13.Run our sketch. The timeline of our sketch should look similar to the following screenshot:
Objective Complete - Mini Debriefing
The first task of our current mission was to read a web server's logfile and extract the IP address and the timestamp of every log entry. We created a class that stores our log row information. We then read the logfile to an array of strings using the loadStrings() method. In step 6, we used a regular expression that matches and extracts the parts of our logfile we are interested in and returns them as strings.
There are an incredible number of ways a date could be represented as a string, since every corner of the world seems to be in urgent need to use a very special one. To make it easier for our sketch to work with the timestamps, we convert them to the Unix timestamp format, which stores the seconds that have elapsed since 1970-01-01.
In our draw() method, we take the timestamps we extracted from the logfile and visualize them on a timeline.
Geocoding IP addresses
So far we have concentrated on the timestamps of our logfile, so this part of our mission is to convert the IP addresses into a pair of coordinates that we can use for our geo-visualization. We will download and install the hostIP database—a free database that stores the geographic coordinates and the name of the city this IP address is coming from. The hostIP database is created by volunteers, so while you're there, add your own IP address and city to improve the service. The accuracy we get from this database is on city level, so you shouldn't design a pizza delivery service based on the geo-coordinates you get, but it's more than sufficient for most visualization projects.
We will keep the hostIP database in memory for geocoding our IP addresses and converting the IP addresses we extracted in the previous task.
Engage Thrusters
Let's start to convert the addresses:
1.To convert the IP addresses, we first need the hostIP database. So, open a browser and go to http://www.hostip.info/. Click on Data and then select one of the
HTTP mirrors in the Daily updates to the database? section.
194
Project 8
2.Download the hip_ip4_city_lat_lng.csv file from the cvs folder.
3.Now, open the sketch we created for the previous task and add the file by dragging it onto the sketch window or by using the Sketch | Add File... menu.
4.Create a new tab and name it IpDB. We add a new class to store the IP address blocks and its latitude/longitude pair.
public class IPdb{ long ipblock; String lat; String lon;
public IPdb( long ipblock, String lat, String lon ) { this.ipblock = ipblock;
this.lat = lat; this.lon = lon;
}
}
5.To load the database, we create a new method that parses the .csv file and stores the records in our IpDB class. We create a HashMap data structure with the IP block as an index so we can access it faster when we geocode the logfile rows. HashMap is a Java data structure that maps a unique key to a different object and allows us to retrieve the stored object using the key.
HashMap<Long,IPdb> parseIpDatabase( String filename ) { String[] raw = loadStrings( filename );
HashMap<Long, IPdb> map= new HashMap<Long,IPdb>(total); for( int i=0; i<raw.length; i++) {
String[] parts = raw[i].split(","); int ipblock = int( parts[0] );
map.put( new Long(ipblock), new IPdb( ipblock, parts[2], parts[3] ));
}
return map;
}
6.In our setup() method, we add a call to the parseIpDatabase() method and HashMap for our database.
import java.text.SimpleDateFormat;
LogRow[] data;
HashMap<Long, IPdb> ipdatabase;
void setup() {
195
Logfile Geo-visualizer
size(700,350);
ipdatabase = parseIpDatabase( "hip_ip4_city_lat_lng.csv" ); String[] log = loadStrings( "access.log" );
data = parseLogfile( log" );
}
7.Now we can use the table object to search the geo-coordinates for any given IP address by searching the table, but first we need to extend our LogRow class to store the coordinates. Switch to the LogRow tab of our sketch and add the following properties:
class LogRow { String ip;
long timestamp;
String lat; String lon;
public LogRow( String ip, long timestamp ) { this.ip = ip;
this.timestamp = timestamp;
}
void setLatLon( String lat, String lon ) { this.lat = lat;
this.lon = lon;
}
}
8.Now, back in the main sketch, we add another method named geoCode(), which takes the array of records as a parameter.
void geoCode( LogRow[] data ) {
}
9.In our method, we iterate over all the rows and try to find the geo-coordinates for our IP address. If we are unable to find the latitude and longitude, we set them to null. The database doesn't use actual IP addresses as a key, but as a numeric representation of an IP block. To convert an IP address into this search key, we need to convert the IP address into a numeric value by splitting it at the dot and multiplying the numbers with powers of 256.
void geoCode( LogRow[] data ) {
for( int i=0; i< data.length; i++) {
String[] block = data[i].ip.split( "\\." );
196
Project 8
int ip = int( block[0] ) * 256 * 256 * 256 +
int( block[1] ) * 256 * 256 + int( block[2] ) * 256; IPdb r = ipdatabase.get(ip);
if ( r != null ) { data[i].setLatLon( r.lat, r.lon);
}
}
}
10.Our sketch parses the logfile and geocodes the IP addresses in our setup() method, and depending on the size of the logfile, this can take a while. To inform our user why our startup takes so unusually long, we are now going to move the parsing into a background thread.
void setup() { size(700,350);
Thread t = new Thread() { public void run() {
ipdatabase = parseIpDatabase
( "hip_ip4_city_lat_lng.csv" );
String[] log = loadStrings( "access.log" ); data = parseLogfile( log );
geoCode( data );
}
};
t.start();
}
11.Now, thesetup() method returns immediately, but there will be no rows to use in the draw() method until the parsing has finished. So we need to modify ourdraw() method and print a message for our users to inform them of what's going on.
void draw() { background(0);
if ( data == null ) {
text( "Loading ...", 35, 190 );
}else { noStroke(); fill(255);
rect(0, 150, 700, 50); stroke(0, 10);
long mints = data[0].timestamp;
long maxts = data[ data.length-1].timestamp;
197
Logfile Geo-visualizer
for ( int i=0; i<data.length; i++) {
float pos = map( data[i].timestamp, mints, maxts, 0, 700 ); line( pos, 150, pos, 200 );
}
}
}
12.This is much better, but our users still have no idea how long the parsing process might take, so let's add a progress bar indicating how much of the parsing or geocoding is already completed. We need to add two variables holding the total row count and the number of already processed records. We also add a state variable, which stores the parsing stage we are currently at.
int LOADING = 0; int LOGFILE = 1; int GEOCODING = 2; int DONE = 3;
int state = LOADING;
int count = 0; int total = 0;
HashMap<Long, IPdb> ipdatabase;
LogRow data[];
void setup() { size(700, 350);
Thread t = new Thread() { public void run() {
state = LOADING;
ipdatabase = parseIpDatabase( "hip_ip4_city_lat_lng.csv" ); state = LOGFILE;
String[] log = loadStrings( "access.log" ); data = parseLogfile( log );
state = GEOCODING; geoCode( data ); state = DONE;
}
};
t.start();
}
198
Project 8
13.In our parseIpDatabase() method, we set the total variable to the number of rows in the database and update the count variable in the for loop.
HashMap<Long, IPdb> parseIpDatabase( String filename ) { String[] raw = loadStrings( filename );
total = raw.length;
HashMap<Long, IPdb> map= new HashMap<Long, IPdb>(total); for ( int i=0; i<raw.length; i++) {
count = i;
String[] parts = raw[i].split(","); int ipblock = int( parts[0] );
map.put( new Long(ipblock), new IPdb( ipblock, parts[2], parts[3] ));
}
return map;
}
14.In our parseLogfile() method, we need to set the total variable to the number of rows in the logfile and also update the count variable in the loop.
LogRow[] parseLogfile( String[] rows ) {
total = rows.length;
SimpleDateFormat sdf = new SimpleDateFormat( "dd/MMM/ yyyy:HH:mm:ss Z");
LogRow[] res = new LogRow[ rows.length ]; for ( int i=0; i< rows.length; i++) {
count = i;
String[] m = match(
rows[i], "(\\d+\\.\\d+\\.\\d+\\.\\d+) - - \\[(.*)\\]"); if ( m != null ) {
try {
long time = sdf.parse( m[2] ).getTime(); res[i] = new LogRow( m[1], time );
}
catch( Exception e ) { e.printStackTrace();
}
}
}
return res;
}
199
Logfile Geo-visualizer
15.And finally, we need to update our geoCode() method to set the total variable to the number of parsed rows and set the count variable to the loop.
void geoCode( LogRow[] data ) { total = data.length;
for ( int i=0; i< data.length; i++) { count = i;
String[] block = data[i].ip.split( "\\." );
int ip = int( block[0] ) * 256 * 256 * 256 + int( block[1] ) * 256 * 256 + int( block[2] ) * 256;
IPdb r = ipdatabase.get(ip); if ( r != null ) {
data[i].setLatLon( r.lat, r.lon);
}
}
}
16.Now we can draw the progress bar in our draw() method and show the name of the stage.
void draw() { background(0);
if ( state < DONE ) { stroke( 0, 200 ); strokeWeight(2); fill( 200, 200 );
rect( 25, 150, 650, 50 ); noStroke();
fill( 255 );
rect( 30, 155, map( count, 0, total, 0, 640 ), 40 );
fill( 0 ); String msg="";
if ( state == LOADING ) {
msg = "loading database ...";
}
else if ( state == LOGFILE ) { msg = "parsing logfile ...";
}
else if ( state == GEOCODING ) { msg = "geocoding ip adresses ...";
}
text( msg, 35, 190 );
200