Category: Performance

Yet another Glastonbury ticket fiasco

Posted by – 2012/10/07

Once again we have tickets for Glastonbury Festival, and once again there’s well-founded accusations of foul play as technical festivalgoers barged through SeeTickets’s queues to the booking pages, and once again I’m not complaining.

Why did it suck, and how did everyone get around it?


Import vmstat logs into LoadRunner

Posted by – 2010/11/15

Here’s a little script to format vmstat’s output into something you can import into LoadRunner’s analysis tool:

import sys, time
                 "Proc Run,Proc sleep,"
                 "Mem swap use,Mem free,Mem buffered,Mem Cache,"
                 "Swap in,Swap out,Blocks in,Blocks out,"
                 "Interrupts,Context switches,"
                 "CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen\n")
for line in iter(sys.stdin.readline, ""):
    if line[0] != 'p' and line[0:2] != ' r':
        t = time.strftime("%d/%m/%Y,%H:%M:%S", time.localtime(time.time()))
        line = ' '.join(line.split())
        line = line.replace(" ", ",")
        sys.stdout.write("%s,%s\n" % (t, line))

(Thanks to fabrizoM for the help!)

Run it from a shell script like so:

vmstat 5 | python >> servername-vmstat.log

Then start the job from the shell, press CTRL+Z and nohup it (so it doesn’t die if you disconnect) and tail it (so you can keep an eye on it) like so:

[1]+  Stopped                 ./
[gaz@box ~]$ bg
[1]+ ./ &
[gaz@box ~]$ disown
[gaz@box ~]$ tail -f servername-vmstat.log
Date,Time,Proc Run,Proc sleep,Mem swap use,Mem free,Mem buffered,Mem Cache,Swap in,Swap out,Blocks in,Blocks out,Interrupts,Context switches,CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen

When you’re done, import it into the Analysis tool via the Tools -> External Monitors menu. Make sure you set the correct date for the custom format and remember to keep a close eye on the timezones. If you make a mistake while importing there’s no going back, so save a backup of your analysis files before you attempt this.

JPerfmeter crashes while starting

Posted by – 2010/05/17

I’ve been us JPerfmeter in work to keep an eye on Linux system counters served up over rstatd.

However, it has one very annoying bug:

If one of your servers goes offline, or it somehow lets you add a server which doesn’t exist JPerfmeter will crash and refuse to start again. The setting is saved somewhere on disk. Where? Google didn’t know (hence this blog post).

Half an hour of messing around in Procmon going through reams of logs and I find the settings in %userprofile%\JPerfmeter, just where you’d expect them to be. facepalm.jpg

LoadRunner Controller – Invalid Action Window

Posted by – 2009/11/27

LoadRunner 9.5′s VUGen sometimes gets its undergarments in a twist and messes up the actions in one of its INI files. You don’t notice this until you get the dreaded “Invalid Action” error message while running a test, which appears once for each virtual user and prevents other users from loading.

Thankfully, Sameh Abdelhamid found and documented a proper fix for this problem:

Find a file called default.usp and edit it in your favorite text editor. I use ScITE which comes with ruby. You will notice a parameter called [Profile Actions].
That will list your actions. If you cross check this with the actions in your script you will notice that in my example, I only have 3 actions, whereas, the has 4.

Which is great, but no good for me right now as it started happening half way into a test which I can’t cancel. If like me, you need that horrible, dirty, reactive solution right now, here’s a little VB Script to send a Y key to the messagebox every time it appears.

Dim Wsh
Set Wsh = Wscript.CreateObject("Wscript.Shell")
While True
  If Wsh.AppActivate("Invalid Action") Then
    Wsh.SendKeys "y"
    WScript.echo "go away!"
  End if
  WScript.Sleep 1000

Stripping snapshot files from LoadRunner Citrix scripts

Posted by – 2009/10/29

After several re-records of a Citrix script in LoadRunner, thousands of PNG snapshot files build up in the data dir. Over time they can build up to make your script eat up over a hundred megabytes of disk space, which is a little excessive given that the useful parts are only a few megs in total.

These snapshots are used to make the script tree-view useful, so you can’t delete all of them in case you need a reference image when inserting a new sync, keypress or some other event. So, I squirted out this little VB script which will remove unused stuff from the data directory, I’m sharing it with the Internet because I’m nice like that.

You can download it here.

To run it, just copy to the dir which contains your scripts (back them up first, I offer no warranty or compensation in case things go wrong) and click it. Be prepared to wait a long time, enumerating forty thousand bitmaps will always take some time.

When archiving your scripts you might also want to delete any results data, which also costs a few megs.

Citrix of the Trade

Posted by – 2009/10/23

I’ve been using LoadRunner for Citrix for about a week now, it’s a new protocol for me and I guess I’m starting to get the hang of it because I’ve created a list of tips and tricks to share with the Internet.

I’m assuming you’ve read the other guides, about using the keyboard where possible, making sure your colour settings and screen sizes are the same on each injector, using bitmap and text checks for synchronization, clicking on menus because mouse movements don’t work and so on.

Don’t trust LoadRunner to get your transactions right

As of version 9.5, LoadRunner records Citrix scripts which look like this:

lr_start_transaction("Receive bacon");
ctrx_sync_window("Bacon dispenser", ACTIVE, 100, 100, 200, 200, etc);
// push button
ctrx_mouse_click( 50, 50, 0, CTRX_LAST);
lr_end_transaction("Receive bacon", LR_AUTO);
lr_start_transaction("Enjoy bacon");
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);

In this case, at recording we put transactions around the time taken to push the button and get some bacon, not how long it takes to focus the bacon dispenser and then push the button. A transaction is all about timing something, which means you do something and then wait for a response; a transaction should never start with a sync and should almost always end with one. So, we need to edit the Receive bacon transaction to look something like this:

lr_start_transaction("Receive bacon");
// push button
ctrx_mouse_click(50, 50, 0, CTRX_LAST);
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);
lr_end_transaction("Receive bacon", LR_AUTO);

Don’t trust LoadRunner to insert think-times of the correct length, nor in the correct places.

This one is an even more painful schoolboy error, let’s use the bacon meme as an example again and a more realistic recording including recorded thinking time, LR would generate something like this:

lr_start_transaction("Receive bacon");
ctrx_sync_window("Bacon dispenser", ACTIVE, 100, 100, 200, 200, etc);
// push button
ctrx_mouse_click( 50, 50, 0, CTRX_LAST);
lr_end_transaction("Receive bacon", LR_AUTO);
lr_start_transaction("Enjoy bacon");
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);

Did you spot it? Firstly the think time is incredibly large because LoadRunner’s Citrix client recorder is obscenely slow, so you’ll have to adjust the value to the amount of time the user would actually spend marvelling at their delicious bacon before eating it. This would be far less than 15 seconds. Let’s also apply the fix from above:

lr_start_transaction("Receive bacon");
// push button
ctrx_mouse_click(50, 50, 0, CTRX_LAST);
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);
lr_end_transaction("Receive bacon", LR_AUTO);

Now the glaring error becomes obvious: The think-time and the sync are the wrong way around! If you don’t strip think times from the results, you can only report on timings of 3 seconds and over. That’s just sloppy, but much worse than that, if you do strip the think time then you’ll report timings 3 seconds shorter than the system actually takes to respond:

Yeah that means that if your scripters don’t know what they’re doing then you run the risk of reporting great timings for an abysmally slow system. So SLAs aren’t met, your company receives huge fines, or in the case of the system I’m working on lives may even be lost.

Ignore the replay guidelines published by Citrix.

Do not replay think-time when debugging scripts, this will force you to write robust scripts that measure rather than relying on the system response times. You won’t be tempted to resort to extending the think-times to fudge synchronisation when you get a huge number of users running.

When you can’t trust a bitmap, cheat with events

Sometimes you can’t trust a bitmap synchronization because the application has painted, but is still processing and will not respond to mouse input. Keyboard input isn’t so much of a problem as it will queue up, but mouse events may be discarded. Thankfully we know that MFC, VB and Java applications are event driven. This means we can use bitmaps synchronizations on things triggered by mouse movement and hover events, then do a click after that.

For example, move the mouse back and forth across a toolbar button, wait for a short period of time (use sleep(), not think time!), then do a check (not a sync) on the bevel which is drawn around it. The same applies to other mouse-over events, if it can take a mouse-over it can probably take a click event. Unfortunately LR can’t detect mouse cursor changes, which would be a great tool for this purpose.

Another similar idea is to use the text buffer; input a character or set the focus with the tab key, then wait for the change in state.

You can’t sync on a context menu

They don’t have a name so they get an obscure number at record time, which cannot be found at playback. Use bitmap checks on the screen area for these instead. Of course, you should really be using the keyboard where possible, but this isn’t always possible (accessawhatability?)

The pixel-sync trick

Record a 3×3 pixel bitmap check for white, grey and declare or #define them as functions like so:

#define Sync3x3(x,y,hash) ctrx_sync_on_bitmap(x, y, 3, 3, hash, CTRX_LAST)
#define WHITE(x,y) Sync3x3(x,y,"794f0585be94ed32b0fe3d42f8917eea")
#define GREY(x,y) Sync3x3(x,y,"2e32a138b3fee935c6efabeb24c05733")
#define BLUE(x,y) Sync3x3(x,y,"7c7ebab71cd7ebeab3f0aa710501fa07")

Then, providing you’ve been a good boy and remembered to maximize your window before recording, you can avoid re-recording when you need a new sync point. Just insert an object click via the GUI tree-view and replace it with one of the macros above in the code. Please note, your grey colour may be different to mine, it’s best to record your own pixels. Also don’t bother with a 1×1 pixel, LoadRunner always fails those checks. (I didn’t try a 2×2 check)

Remove those rogue mouse-up events

After drawing the rectangle for a bitmap or text synchronization point, you’ll often get a rogue mouse-up event. Remove those, they’re a confusing waste of processing time and won’t happen in the real world.


The industry standard load testing tool is simple enough for trained monkeys to use, but its default Citrix settings suck to the point of being dangerous. Handle with care!

That’s all I’ve learned this week, hopefully it’s enough for me me to help deliver this project on time. Expect more similar rants and tips over the coming months.

Loadrunner DEP closes iexplore.exe

Posted by – 2009/10/22

Search spam time!

When launching LoadRunner for the first time on a Windows XP or 2003 machine, VUGen may record no events and then complain that the Data Execution Prevention Service closed iexplore.exe.

In this case, go to Control Panel -> System -> Advanced -> Performance -> Settings -> Data Execution Prevention -> and select “Turn on DEP for essential windows programs and services only.”

Now reboot your computer/server.

It’s been a while since I used LoadRunner, as you can probably tell. Win2K desktops and XP machines before SP2 didn’t have this problem.

Be Internet rebranding fiasco

Posted by – 2009/07/22

My ISP, Be Internet just went live with their new website and even though I’m a tiny minority in thinking this, I quite like the new layout. It has REST-like URLs and social media bookmarks, it even uses the strict XHTML DTD, which is refreshing.

However, the design is the only thing I like! The new site has not only collapsed under the load, but users are reporting that they are seeing other people’s account details when they log in! Also their SSL certificate is invalid, so you’ll need to add an exception before you can even visit the site. Their nice new Digg account is apparently “A 14 year-old person from United Kindgom (GB)

Did Be actually do any testing before going live?

Leaky account details

Leaky account details

I’d be shocked and disgusted like the other users if I wasn’t a performance tester, screw ups like this are a wonderful advert for people like me!