Why did it suck, and how did everyone get around it?

Knowing that their IIS 7.5 servers (both of them) couldn’t hold up under the load of 2 million people hitting F5 every few seconds, See did a “clever” thing with their DNS servers and set them to periodically return a 192.168 address for their booking page. This IP address gets saved on the user’s DNS resolver cache and effectively bans that computer from contacting the booking page until the DNS time-to-live has expired; you’re attempting to contact a machine on your own local network and no matter how furiously you hit refresh you aren’t causing See any damage. While you’re requesting a connection to a non-existent 192.168.202.201/202 the real servers at 194.168.202.201/202 (sneaky eh?) are not collapsing under the load. Crude, but effective.

There were complaints in 2010 that the system wasn’t fair, people who got in once could go back and register more people. The lucky winners with the right address in their cache could gorge themselves on orders until all the tickets ran out, larger groups who were more likely to have a winner got tickets while the smaller groups lost out. This also meant that if your company’s corporate proxy servers won the DNS lottery then everyone in your company could get a ticket. Like every year, much butthurt was caused by this unfair selection process.

This year See took the infinitely wise decision to “fix” this by decreasing their DNS TTL to 60 seconds. So to get a ticket legitimately you’d have to either complete the entire three-stage booking process in under 60 seconds, or win the DNS lottery several times in a row. Good luck with that, humans!

Those in the know just edited their hosts file to point glastonbury.seetickets.com to the correct IP, and as soon as eFestivals announced this to the world ’201 slowed to a crawl as thousands of queue-bargers found their way over the barriers and bought around 60,000 tickets in under 20 minutes. Once again, it sold out in record time.

Can’t wait to see what they come up with for the resale!

After four multiple-choice “which menu is this option in?” questions I’d cancelled the test in a rage and was on the phone to the agency explaining that I was genuinely offended that they’d rank their applicants using quantitative measures of memory when the required skill-set is essentially qualitative and down to problem solving. Ten minutes later I was sat pink-bottomed, my neck wound in, reluctantly plodding through the rest of the questions.

You’re a Windows expert right? Without looking, what’s the second item up after pressing the start button on a Windows XP machine? Okay, Linux, what’s the “-s” in “ls -s”?

I think I counted two questions about systems diagnostics, one on paging and one on bandwidth, which would actually judge a candidate’s ability to solve problems, but the rest were about installation, process names and menu items. Along with the menu questions I was expected to know whether the first parameter to sprintf was the formatting or the destination string, the difference between “%09ld” and ”%9ld”, that LoadRunner uses strftime and which token means abbreviated month name. Surely a smart person would press the F1 key every time rather than learning C’s escape sequence mess off-by-heart.

How a score of 76 out of 100,  a B+ at best, puts someone over two standard deviations from the mean is anyone’s guess. Either most people are rubbish or the test is measuring the wrong things. I suspect it’s a combination of both, but more of the latter than the former.

Good job it didn’t measure punctuality and work ethic or I’d have been jobless in the bottom quartile, spending tomorrow morning in my dressing gown with a beard instead of suited and booted in a bank. Do razors come with an instruction manual? It’s been a while…

So I’ve moved it over to GitHub where anyone can play with it, hopefully it will be more useful there.

Enjoy!

Time for some fun!

The first thing I did was break the plastic housing off the mic and exposed the sensor, I snipped off the headphone socket and taped up the rest. It’s not the prettiest bit of electrical tape you’re likely to see, but it does the job:

I then taped the mic to the airlock roughly at the area where the bubbles will be, then taped the whole thing up to make it a bit more water-resistant.

Okay, now onto the fun bit: writing the software

The easiest way to get sound from the mic port is to use ALSA‘s arecord, which comes as part of alsa-utils and should already be installed on your distro. Run amixer from the command line to find the correct input and unmute it if required, but in most cases just running arecord will work. I set my volume levels to be 100% (unamplified) as I want to be the only person messing with the data.

I’m after a raw stream of data without any of that WAV header nonsense, so throwing a stream of raw longs to stdout looks a little like this:

gaz@beast:~$ arecord --format=S32_LE --file-type=raw

The next task is detecting the bubbles in this raw data. The mic is touching the airlock so the bubbles are very loud and high pitched, so plugging in a high pass filter might help get rid of any background noise. I found that using Audacity’s high pass filter managed to keep the bubble track intact while filtering out most of the the sound of a vacuum cleaner, which is pretty impressive. I didn’t write my own high pass filter as it seems to work okay without one, maybe I’ll make an FFT-based one in the future.

For the moment though, I simply defined a bubble-part as a volume reading that is two standard deviations away from the mean, where the mean is taken by summing the distance of each sample from the base line; waves always add up to 0, so we need an absolute measurement from the base line or the average will always be 0. If this bubble part is more than t milliseconds from the last one, then we have a new bubble.

My limited testing shows that the bubble count appears to be roughly right even when I’m playing music in the background, but it will probably need some tweaks before it works in all cases. So far it’s only been tested with my 25 litre plastic wine fermenter, so I have no idea how it will perform on demijohns. I can think of a number of things that are wrong with the algorithm too.

For a start it’s adaptive, so if you simply unplug the cable then the odd bubble will be detected. A minimum bubble length may help here. Secondly, the averages are taken from the last listening period so it’s all dependent on the length of this window, moving averages might be better, but would be more processor intensive (as if Python wasn’t processor intensive enough) and I have no idea what that does statistically, but I imagine it’s not good from a purist perspective. Finally, it’s using average volume of each |sample value| but it actually uses peak levels to detect bubbles, so I guess that it ought to be using the maximum |sample value| over N samples where N is equivalent to some minimum frequency.

Ignoring all that it appears to work okay. Here’s a graph that shows it in action along with my thermometer, the dip is caused by me dismantling it while taking the photos above.

You can download or fork the the code over github.

Happy brewing!

Thankfully, a search for the device ID (0c45:7401 Microdia) finds ruby bindings for Temper.c by Michitaka Ohno, which conveniently includes the reverse-engineered USB protocol. So I’ve forked this on GitHub, removed the Ruby bits, edited the unit test wrapped it up in a little script that logs the time and temperature to a CSV file. It currently only supports one device, which will change if I ever get another one of these devices (spoiler: I probably will)

To start logging your TEMPer1′s data in Ubuntu just

 # install dependencies
 $ sudo apt-get install git build-essential libusb-0.1.4 libusb-0.1.4-dev
 # clone the logger
 $ git clone git://github.com/bitplane/temper.git
 # build it
 $ cd temper
 $ make clean
 $ make
 # start logging!
 $ sudo ./log.sh > temperature.log

Now I’m just waiting for my USB clip-on-mic to arrive. Once it does I can log the bubbles from the airlock on my beverages along with the temperature, and make some pretty graphs.

Watch this space!

My last experience with a printer in Ubuntu was when I attempted to use an old Lexmark Z-something, there were no Linux drivers and after much struggling I had to reboot into Windows to use it. The Windows drivers were close to a hundred megabytes, sluggish, always running and came complete with an intrusive systray app to further sap system resources. The app had a non-standard and ugly UI plus annoying American voice that begged me to “please insert paper in the auto sheet-feeder” each time I tried to print something. I remember Canon doing a similar thing with their Windows software too which is one reason why I’ve avoided printers for all these years. That sort of blatant disregard for system resources and desktop space makes the overhead of attaching a printer weigh in heavier than the reward it offers.

I don’t know whether it’s HP’s, Canonical’s or Debian’s idea but the experience with my OfficeJet 4500 in Ubuntu has been great. Rather than having to download drivers I just plugged it in. I installed gLabels from the Software Center and printed out some labels within minutes. Pressing meta and typing “print” takes me to Gnome’s Printing app which lists my printer and its fax machine counterpart (remember those?!), and I can right click and set quality settings and so on in a standard UI:

Keying “scan” into Unity’s search bar takes me to Canonical’s “Simple Scan” which really is as easy as π, it just scans. Before I could scan through Gimp I had to install XSane, which a bit of a mess. It opens four windows and somehow transfers to Gimp as JPEG by default, leaving 16×16 JPEG compression artefacts in my scanned images. These look bad even if I choose 100% quality in the menu, so it looks like the quality settings are ignored:

edit: A bit of investigation showed that this is due to Sane’s XP driver using 10% JPEG compression by default. This can be disabled via the Window -> Show advanced options dialog.

Expect lots more activity there in the future

Sorry for not writing to you of late, but this year has been a busy one.  I’ll make up for my lack of output with this mammoth post reviewing a small selection of interesting things I got up to this year…

Did far too much work

Feels like I’ve been busting my balls for the last two years, so I decided to stick all my invoices into Google Docs and graph the output. The answer is yes, I’ve been working like a full-time employee.

2009 was a good year!

I love consulting and Microsoft have a great working environment, but a rolling contract at a huge corporation means living out of a hotel for months at a time, which gets old rather fast. I learned a lot about BizTalk, Visual Studio Load Test and T-SQL, I’ll miss the new friends I’ve made, but I certainly won’t miss the travel. So I’m taking a couple of months off to doss around, learn some new technologies and work on some free software before looking for another contract.

Rediscovered Bitcoin during the bubble

I initially flirted with Bitcoin in late 2010 and was completely fascinated by the idea of a P2P currency, specially the proof-of-work method and the anonymity. There was no GPU mining software available and doing work units on the CPU had pretty low returns, but in December I decided to get a decent PC for experimenting with this new currency. After a bunch of cooling problems and losing my wallet.dat (containing a mere 5 BTC, worth about 30 US cents each; £1 total at the time) I fell out of love with Bitcoin and forgot all about it…

Fast forward 6 months and everyone is talking about this amazing new currency, demand outstrips supply, prices soar and the bubble finally bursts at $28 a piece. Why, oh why did I spend £2,000 on a top of the range computer then not even bother mining any BTC, rather than £1000 on BTC in December and sell out at £95,0000 in June?

What could have been

On top of the new PC the Bitcoin thing ended up costing me about £50 in electricity, and my ATI 5970 hasn’t been feeling well since. I did play online poker with Americans after they closed all the poker sites, and made about $200 trading on Mt Gox. Not enough BTC to have someone assassinated but the option to buy 5 wraps of heroin from Silk Road is always open (INSTANT ADDICTION, TORE FAMILY APART. A++++ WOULD BUY AGAIN)

Read Gödel, Escher, Bach

With all the travel I’ve had a fair bit of time for reading, which is nice. Along with The Design of Everyday Things and Zen and the art of Motorcycle Maintenance, Godel, Escher, Bach (that’s Girdle, Esher, Bark to us infidels) changed the way that I think about a lot of things.

This year I’m going to try to set more time aside for reading.

Got into home brewing

A few months back I decided to start brewing mead, yeah, before Skyrim came out. I’m now seven batches in and I think I’m getting the hang of it, my latest batches taste a bit less like sweaty feet and a bit more like something you might want to drink. I started out using a 5L bottle of water with 3 jars of honey, some raisins and bread yeast added to it and a balloon stuck over the top, and now I’ve progressed to demijohns with real air-locks and exotic ingredients like.. uh.. orange flesh, cloves and vanilla pods. I even made some labels:

How's the abdomen?

I figure that this whole home brewing thing is a long game, it’s worth starting while I’m in my 30s if I’m going to actually master it before I’m old and grey. That is if the state of the kitchen doesn’t cause me to die of death by angry woman first.

Replaced my MBW-150

After the LEDs burned out on my Sony Ericsson MBW-150 I replaced it with a shiny new Metawatch. Apparently it looks like a cheap, tacky Casio (so some random guy in a bar tells me anyway), but it’s not really supposed to be bling; the firmware and Android software is open source, making the device infinitely programmable.

The down-side is that the firmware is too large to be compiled using gcc for MSP430, so there’s currently no free development environment. I sketched out an analogue face for it in C, but if I wanted to spend £2000 on an analogue watch face then I would have got a Rolex instead!

Neglected Free Software

I’ve been lazy when it comes to developing this year. I’ve done no work on my Android apps, pushed a single commit to Irrlicht and wrote about 100 lines of my web-based VNC client. I feel like I’ve cheated on the world by spending most of the year writing T-SQL and C# for Microsoft, so my new year’s resolution is to get off my backside and write some code.

And blog more. Yeah, I’ll do that too.

If you fix any bugs, please send a pull request and I’ll do my best to merge them in, but keep in mind this is also my first attempt at using git; expect schoolboy errors there too!

import sys, time
 
sys.stdout.write("Date,Time,"
                 "Proc Run,Proc sleep,"
                 "Mem swap use,Mem free,Mem buffered,Mem Cache,"
                 "Swap in,Swap out,Blocks in,Blocks out,"
                 "Interrupts,Context switches,"
                 "CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen\n")
 
for line in iter(sys.stdin.readline, ""):
    if line[0] != 'p' and line[0:2] != ' r':
        t = time.strftime("%d/%m/%Y,%H:%M:%S", time.localtime(time.time()))
        line = ' '.join(line.split())
        line = line.replace(" ", ",")
        sys.stdout.write("%s,%s\n" % (t, line))
        sys.stdout.flush()

(Thanks to fabrizoM for the help!)

Run it from a shell script like so:

#!/bin/bash
vmstat 5 | python vmstat2csv.py >> servername-vmstat.log

Then start the job from the shell, press CTRL+Z and nohup it (so it doesn’t die if you disconnect) and tail it (so you can keep an eye on it) like so:

[1]+  Stopped                 ./log.sh
[gaz@box ~]$ bg
[1]+ ./log.sh &
[gaz@box ~]$ disown
[gaz@box ~]$ tail -f servername-vmstat.log
Date,Time,Proc Run,Proc sleep,Mem swap use,Mem free,Mem buffered,Mem Cache,Swap in,Swap out,Blocks in,Blocks out,Interrupts,Context switches,CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen
23/11/2010,12:59:47,1,0,0,1266452,465576,2074012,0,0,2,15,0,36,0,0,99,0,0
23/11/2010,13:00:11,0,0,0,1264460,465584,2074024,0,0,0,168,581,577,1,1,94,4,0

When you’re done, import it into the Analysis tool via the Tools -> External Monitors menu. Make sure you set the correct date for the custom format and remember to keep a close eye on the timezones. If you make a mistake while importing there’s no going back, so save a backup of your analysis files before you attempt this.