Expect lots more activity there in the future

Sorry for not writing to you of late, but this year has been a busy one.  I’ll make up for my lack of output with this mammoth post reviewing a small selection of interesting things I got up to this year…

Did far too much work

Feels like I’ve been busting my balls for the last two years, so I decided to stick all my invoices into Google Docs and graph the output. The answer is yes, I’ve been working like a full-time employee.

2009 was a good year!

I love consulting and Microsoft have a great working environment, but a rolling contract at a huge corporation means living out of a hotel for months at a time, which gets old rather fast. I learned a lot about BizTalk, Visual Studio Load Test and T-SQL, I’ll miss the new friends I’ve made, but I certainly won’t miss the travel. So I’m taking a couple of months off to doss around, learn some new technologies and work on some free software before looking for another contract.

Rediscovered Bitcoin during the bubble

I initially flirted with Bitcoin in late 2010 and was completely fascinated by the idea of a P2P currency, specially the proof-of-work method and the anonymity. There was no GPU mining software available and doing work units on the CPU had pretty low returns, but in December I decided to get a decent PC for experimenting with this new currency. After a bunch of cooling problems and losing my wallet.dat (containing a mere 5 BTC, worth about 30 US cents each; £1 total at the time) I fell out of love with Bitcoin and forgot all about it…

Fast forward 6 months and everyone is talking about this amazing new currency, demand outstrips supply, prices soar and the bubble finally bursts at $28 a piece. Why, oh why did I spend £2,000 on a top of the range computer then not even bother mining any BTC, rather than £1000 on BTC in December and sell out at £95,0000 in June?

What could have been

On top of the new PC the Bitcoin thing ended up costing me about £50 in electricity, and my ATI 5970 hasn’t been feeling well since. I did play online poker with Americans after they closed all the poker sites, and made about $200 trading on Mt Gox. Not enough BTC to have someone assassinated but the option to buy 5 wraps of heroin from Silk Road is always open (INSTANT ADDICTION, TORE FAMILY APART. A++++ WOULD BUY AGAIN)

Read Gödel, Escher, Bach

With all the travel I’ve had a fair bit of time for reading, which is nice. Along with The Design of Everyday Things and Zen and the art of Motorcycle Maintenance, Godel, Escher, Bach (that’s Girdle, Esher, Bark to us infidels) changed the way that I think about a lot of things.

This year I’m going to try to set more time aside for reading.

Got into home brewing

A few months back I decided to start brewing mead, yeah, before Skyrim came out. I’m now seven batches in and I think I’m getting the hang of it, my latest batches taste a bit less like sweaty feet and a bit more like something you might want to drink. I started out using a 5L bottle of water with 3 jars of honey, some raisins and bread yeast added to it and a balloon stuck over the top, and now I’ve progressed to demijohns with real air-locks and exotic ingredients like.. uh.. orange flesh, cloves and vanilla pods. I even made some labels:

How's the abdomen?

I figure that this whole home brewing thing is a long game, it’s worth starting while I’m in my 30s if I’m going to actually master it before I’m old and grey. That is if the state of the kitchen doesn’t cause me to die of death by angry woman first.

Replaced my MBW-150

After the LEDs burned out on my Sony Ericsson MBW-150 I replaced it with a shiny new Metawatch. Apparently it looks like a cheap, tacky Casio (so some random guy in a bar tells me anyway), but it’s not really supposed to be bling; the firmware and Android software is open source, making the device infinitely programmable.

The down-side is that the firmware is too large to be compiled using gcc for MSP430, so there’s currently no free development environment. I sketched out an analogue face for it in C, but if I wanted to spend £2000 on an analogue watch face then I would have got a Rolex instead!

Neglected Free Software

I’ve been lazy when it comes to developing this year. I’ve done no work on my Android apps, pushed a single commit to Irrlicht and wrote about 100 lines of my web-based VNC client. I feel like I’ve cheated on the world by spending most of the year writing T-SQL and C# for Microsoft, so my new year’s resolution is to get off my backside and write some code.

And blog more. Yeah, I’ll do that too.

I’m kind of torn about adding Flattr links to my projects, on one hand I want to encourage the uptake by being another site which supports Flattr, on the other hand I don’t want to whore myself out for a few pence.

So at some point I’ll add Flattr links to the free Android apps which I’ve made, which will feature in the menu at the top shortly.

The meat of this is JavaScript stolen from another theme and some PHP code gratuitously pasted into my header, this isn’t IP theft, and not just because it’s the season of sharing! This is what’s great about free software licenses.

It makes you glad that America has freedom of the press, even if Britain doesn’t.

I’m not too bothered about popular music, the singles chart or who gets the Xmas #1 slot, but last week I was invited to join a growing group on Facebook who are campaigning to knock the X Factor winner from the top spot by mass purchasing Rage Against The Machine’s classic track Killing in the Name. The sound of rebellion to conquer the airwaves, political rap metal on future Christmas compilation albums, all for the princely sum of 79p? I don’t usually buy digital downloads but this time you can count me in!

According to Sky News the group had 43,000 members sometime on Friday, but by the time I got home on Saturday night there were 180,000 members and rising. As the media coverage increases so do the new members, which made me interested: how does a phenomenon like this evolve, how will it turn out next Sunday? What happens when the UK Charts people decide that it’s against the rules and disqualify the single?

So I decided to log and graph the group’s membership, every fifteen minutes I grab the page using wget, I extract the number of users and dump that into a text file along with the current date and time. Then I cut through it using a couple of awk and sed one liners, dump the results into an HTML file, graph it using Google Charts and upload the output to my file dump.

Update: These graphs are no longer live! Click for the live versions which are updated much more often using a different script

Here’s the scraping script:

#!/bin/bash
 
cd /home/gaz/ratm/
 
# get the timestamp
timestamp=`date "+20%y/%m/%d %H:%M:%S"`
 
# get the file
wget --max-redirect 2 -O temp.html http://www.facebook.com/group.php?gid=2228594104 --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5"
 
# extract user count from the file
usercount=`sed -n -e "s/.* of \(.*\) members.*/\1/p" temp.html`
 
# remove any commas from the string
usercount=${usercount//[,]/}
 
# it must have a length, or it will cause problems when Facebook is having problems!
# in this case, we just give a -1 (not good practice from a stats PoV, but it keeps it simple) 
if [ "${#usercount}" -eq "0" ]
then
    usercount="-1"
fi
 
# remove the temporary file
rm temp.html
 
# write the output in CSV format
echo "$timestamp,$usercount" >> data.dat
 
# next I run the graph generating script

And this one (no longer in use) creates the two above charts from the data:

#!/bin/bash
 
# gets a column from a line of a CSV file. The first index is 1, not 0.
getElement() {
    RESULT=0
    local p=`echo "$1"p`
    RESULT=$(echo $2 | sed 's/,/\n/g' | sed -n $p)
}
 
# get the start and end times
 
getElement 1 "$(tail -1 data.dat)"
end=$RESULT
getElement 1 "$(sed -n '1p' data.dat)"
start=$RESULT
 
# get the current minimum and maximum values
min=$(cat minval)
max=$(cat maxval)
 
# get the last value
getElement 2 "$(tail -1 data.dat)"
lastval=$RESULT
 
# set new max value
 
if [ "$lastval" -gt "$max" ]
then
    echo "$lastval" > maxval
    maxval=$lastval
    echo New maximum, $lastval
fi
 
# and the new min value
 
if [ "$lastval" -gt 0 ]
then
    if [ "$lastval" -lt "$min" ]
    then
        echo "$lastval" > minval
        min=$lastval
    fi
fi
 
# get values for the Y axis
quart=$((($max - $min) / 4))
q1=$(($min + $quart * 1))
q2=$(($min + $quart * 2))
q3=$(($min + $quart * 3))
 
# extract the data using regexp:
# 1. get every 4th line of the file, meaning hourly
# 2. take all the values from the file
# 3. remove the trailing comma
 
data=$(awk 'NR%4==0' data.dat | sed -n -e "s/.*,\([0-9]*\)/\1/p" | tr "\n" "," | sed -e "s/\(.*\),/\1/")
 
# build the URL to the total members chart
total_members="http://chart.apis.google.com/chart?chtt=Total+Members&chs=600x300&cht=ls&chxt=x,y&chxl=0:|$start|$end|1:|$min|$q1|$q2|$q3|$max&chds=$min,$max&chd=t:$data"
 
# now let's do members per hour
 
lastval=$min
min=0
max=0
data=""
inputList=$(awk 'NR%4==0' data.dat | sed -n -e "s/.*,\([0-9]*\)/\1/p")
while read line; do
    if [ "$line" -gt "0" ]
    then 
        val=$(($line - $lastval))
        lastval=$line
    else
        val=0
    fi
 
    if [ "$val" -gt "$max" ]
    then
        max=$val
    fi
 
    data="$data,$val"
done <<< "$inputList"
 
# remove comma prefix
data=$(echo "$data" | sed -e "s/,\(.*\)/\1/g")
 
# build the per hour chart
members_per_hr="http://chart.apis.google.com/chart?chtt=Members+per+hr&chs=600x300&cht=ls&chxt=x,y&chxl=0:|$start|$end|1:|$min|$max&chds=$min,$max&chd=t:$data"
 
# I then create an HTML file from some templates and upload everything to my dump

Worked fine for me on two different machines, my only problem is that the battery charge profile on my laptop was lost.

Some caveats:

• The file list currently tracks directory views (via the template) but not yet file downloads, as that requires a Piwik plugin that isn’t final so I haven’t installed it, adding it should be trivial though.
• As requests are passed through PHP using mod_rewrite it will result in long running scripts on your server, which may upset your shared web hosting provider.
• Scripts are not yet excluded by rewrite rules! You won’t want to keep other scripts containing sensitive information in dirs under the path.
• Access to files and folders beginning with “.” will be blocked.
• It only works on Unices, not Windows servers as they don’t have the “file” command to get the MIME type.
• You’ll obviously need an Apache installation with mod_rewrite.

To install it, just copy index.php, .htaccess and .filelist/ to the directory where you store your files. You can get the script (without icons) from SVN.

bitplane also makes sailors blush, 0.3% of the time.

Oh, shiiii…

Getting it working in Gallery2 was a bit of a pain, so I updated their wiki for them, but other than that it’s about the same as installing Google Analytics.

Subscribers are welcome to see my laughable stats