Yet another Glastonbury ticket fiasco

Posted by – 2012/10/07

Once again we have tickets for Glastonbury Festival, and once again there’s well-founded accusations of foul play as technical festivalgoers barged through SeeTickets’s queues to the booking pages, and once again I’m not complaining.

Why did it suck, and how did everyone get around it?

Knowing that their IIS 7.5 servers (both of them) couldn’t hold up under the load of 2 million people hitting F5 every few seconds, See did a “clever” thing with their DNS servers and set them to periodically return a 192.168 address for their booking page. This IP address gets saved on the user’s DNS resolver cache and effectively bans that computer from contacting the booking page until the DNS time-to-live has expired; you’re attempting to contact a machine on your own local network and no matter how furiously you hit refresh you aren’t causing See any damage. While you’re requesting a connection to a non-existent 192.168.202.201/202 the real servers at 194.168.202.201/202 (sneaky eh?) are not collapsing under the load. Crude, but effective.

There were complaints in 2010 that the system wasn’t fair, people who got in once could go back and register more people. The lucky winners with the right address in their cache could gorge themselves on orders until all the tickets ran out, larger groups who were more likely to have a winner got tickets while the smaller groups lost out. This also meant that if your company’s corporate proxy servers won the DNS lottery then everyone in your company could get a ticket. Like every year, much butthurt was caused by this unfair selection process.

This year See took the infinitely wise decision to “fix” this by decreasing their DNS TTL to 60 seconds. So to get a ticket legitimately you’d have to either complete the entire three-stage booking process in under 60 seconds, or win the DNS lottery several times in a row. Good luck with that, humans!

Those in the know just edited their hosts file to point glastonbury.seetickets.com to the correct IP, and as soon as eFestivals announced this to the world ’201 slowed to a crawl as thousands of queue-bargers found their way over the barriers and bought around 60,000 tickets in under 20 minutes. Once again, it sold out in record time.

Can’t wait to see what they come up with for the resale!

7 Comments on Yet another Glastonbury ticket fiasco

Respond | Trackback

  1. Rocker says:

    Interesting read, thanks. So are you suggesting (“There were complaints in 2010 that the system wasn’t fair”) that the same thing happened in 2010 (albeit without a 60 second TTL) ?

  2. Steve says:

    Except that the 194.168.202.192-255 addresses belonged to a junior school in Luton, and not Seetickets. Which makes it look as if any DNS monkey-business was something else entirely. In any case, there’s no real evidence (other than anecdotal) that changing the hosts file did any more than not. Of course, it’s comforting to think (if you didn’t get a ticket) that those who did had some secret advantage, but the numbers don’t really bear it out – the tickets didn’t even sell out any faster than after the last gap year back in 2006-7.

  3. Steve says:

    ‘Ang on – 2 million users? I’d heard estimates of a quarter of that . But if two million folk are going for less than 200,000 tickets, then *of course* 90% of them will be disappointed….

  4. Gaz Davidson says:

    You may be right, 2 million was 2010/2011 figures from memory.

  5. Gaz Davidson says:

    Those reverse lookup sites are out of date, those same IP addresses were used for the registrations website.
    Also the host file hack worked perfectly until everyone started doing it and the server died, at which time I switched IPs and continued to buy tickets.

    Glastonbury announced that they’d sold half the tickets, then 20 minutes later (after the host-file change went public) they sold out. By my reckoning that’s somewhere between 60,000 and 70,000 tickets in 20 minutes.

  6. eddie says:

    okay, so the seetickets dns servers were replying with a local machine ip in 2010, which gets stored in the cache, effectively leaving you banned as it says. I have a few questions:
    1. is this cache browser or machine specific?
    2. could you please explain in more detail the method they used in 2012? they reduced the TTL to 60 seconds, so does that mean they set their DNS servers to return a ‘faked’ ip a certain number of times until you got lucky and got the right ip? If the ttl of the dns was 60 secs and people were trying all day but didnt get in, that must mean See’s DNS dervers were returning the wrong ip all day? i thought the DNS servers were meant to give out the right ip to obviously connect you to the site?
    3. how did eFestivals, or whoever found this trick out first, find out the real ip was in fact 194.168.202.201/202 and not 192.168.201/202? i.e what method would you use to find out the real ip to add to your hosts file? did they just guess randoms ip’s in their hosts file until they got connected?!
    I’m a 15 year old boy who is interested in stuff like this and i just want to gain knowledge considering there’s noone to answer all these questions i have :( thanks, a reply would be fantastic !
    eddie

    [WORDPRESS HASHCASH] The poster sent us ’0 which is not a hashcash value.

  7. Gaz Davidson says:

    Hi Eddie

    1. Your DNS cache is part of your PC’s networking software. Your web browser will ask the TCP networking library (WinSock in Windows, Berkeley Sockets if you use Linux or a Mac) to “Open up a (TCP socket) connection to glastonbury.seetickets.com which will be listening on port 80″, and as far as it knows it’s got a connection to the website that it can read stuff from and write stuff to. Behind the scenes what will actually happen is WinSock will look in the DNS cache for glastonbury.seetickets.com and if it isn’t listed, or the listing is out of date, it will send a packet of information (a special IP packet on UDP port 53) to one of your internet service provider’s DNS servers (well, usually your router’s DNS server, which will ask your ISP’s DNS servers). This request will go from parent to child, from .com to ask for seetickets, to seetickets to ask for glastonbury, and eventually a DNS response packet will be relayed back to your machine. This response contains the location of the server on the network (its IP address) and the amount of time the address is valid for (the TTL; time to live), and this is saved to your DNS cache so you don’t have to ask for it again next time.

    Internet Protocol (IP) is basic delivery service that routes individual parcels of data (packets) between computers. It doesn’t have any concept of which packet is supposed to come first or which program wanted it, packets aren’t even guaranteed to arrive because they could get lost. This is where Transmission Control Protocol (TCP) comes in. TCP sockets are like a two-way conversation that’s built on top of this delivery network, inside the packets we have information like “this is the 5th packet of information”, “this is a delivery receipt for everything up to packet 58″, “hello? are you still there?”, but to a program like a web browser it looks like a solid pipe that can send and receive information.

    So to download a web page your browser has to open a TCP socket. This starts off with a “handshake” where the connector says “let’s talk” (SYN), the listener replies with “hey, I’m listening, did you really ask me to talk?” (ACK) and the connector says “yep that was me” (SYN-ACK).

    Okay, that’s enough background so you’ll understand my answer to your second question…

    2. They told seetickets.com’s DNS servers to return the correct IP address for glastonbury.seetickets.com randomly, to lucky winners, so most people were given the wrong directions. The address they’re given is a special address which isn’t on the public Internet, it’s on your own home network so your SYN “let’s talk” packets don’t even get a chance to clog up their pipes! So WinSock is sat around waiting for an ACKnowledgement and your browser is sat there waiting to connect until it gets bored and decides nobody’s home, the SYN packet never arrived because it was sent to the wrong place.

    Because they set the expiry time to 60 seconds plus each time you move forward a page your browser makes a fresh connection to the site, if 60 seconds have passed you will request a new IP address and probably be given the wrong one. So unless you’re a quick typist you’d have to “win” the right IP address multiple times to get through.

    The whole reason for this is because they have two weedy computers on a crummy network and wanted to serve hundreds of thousands of people, so they had to do some dirty trick to make sure everyone didn’t flood them, but they made a bad decision.

    3. Finding the proper IP is pretty simple, the clever part is figuring out that they gave you a bad address in the first place. Much to my shame I didn’t actually figure it out myself because I was in buying frenzy mode. To figure it out you’d need to record the network with a tool like Wireshark, to read the trace and realise that the IP address is wrong. You’d need a keen eye to spot that. Comparing a connection that fails with one that works will give you the correct IP address too, much easier.

    When the first server had crashed I found the second server by doing some Internet sleuthing, there are websites which record the IP addresses that have been used with different domain names and See had leaked this information (LOL). I was also extra cheeky towards the end, I guessed that their “www” server would be running the same software and put its IP address in my hosts file and managed to get to the booking page, but tickets had run out by then.

    If you’re really interested in this stuff I recommend reading up on the OSI Network Model and the Internet Protocol Suite, have a mess around with Wireshark and do a bit of network programming if you’re into that sort of thing. Also learn the basic command line tools and where your network settings are and what they all mean.

Respond

Comments

Comments