Tag: performance testing

Import vmstat logs into LoadRunner

Posted by – 2010/11/15

Here’s a little script to format vmstat’s output into something you can import into LoadRunner’s analysis tool:

import sys, time
 
sys.stdout.write("Date,Time,"
                 "Proc Run,Proc sleep,"
                 "Mem swap use,Mem free,Mem buffered,Mem Cache,"
                 "Swap in,Swap out,Blocks in,Blocks out,"
                 "Interrupts,Context switches,"
                 "CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen\n")
 
for line in iter(sys.stdin.readline, ""):
    if line[0] != 'p' and line[0:2] != ' r':
        t = time.strftime("%d/%m/%Y,%H:%M:%S", time.localtime(time.time()))
        line = ' '.join(line.split())
        line = line.replace(" ", ",")
        sys.stdout.write("%s,%s\n" % (t, line))
        sys.stdout.flush()

(Thanks to fabrizoM for the help!)

Run it from a shell script like so:

#!/bin/bash
vmstat 5 | python vmstat2csv.py >> servername-vmstat.log

Then start the job from the shell, press CTRL+Z and nohup it (so it doesn’t die if you disconnect) and tail it (so you can keep an eye on it) like so:

[1]+  Stopped                 ./log.sh
[gaz@box ~]$ bg
[1]+ ./log.sh &
[gaz@box ~]$ disown
[gaz@box ~]$ tail -f servername-vmstat.log
Date,Time,Proc Run,Proc sleep,Mem swap use,Mem free,Mem buffered,Mem Cache,Swap in,Swap out,Blocks in,Blocks out,Interrupts,Context switches,CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen
23/11/2010,12:59:47,1,0,0,1266452,465576,2074012,0,0,2,15,0,36,0,0,99,0,0
23/11/2010,13:00:11,0,0,0,1264460,465584,2074024,0,0,0,168,581,577,1,1,94,4,0

When you’re done, import it into the Analysis tool via the Tools -> External Monitors menu. Make sure you set the correct date for the custom format and remember to keep a close eye on the timezones. If you make a mistake while importing there’s no going back, so save a backup of your analysis files before you attempt this.

Float.parseFloat is slow

Posted by – 2010/08/27

While profiling some loader code and expecting to see performance issues with my regular expressions, I found that ~60% of my time was actually spent converting String variables to floats. It turns out that the slow bit lives inside Float.parseFloat, where it calls String.toLowerCase every single time just in case this string is a hex float, a very rare type of float which I had never seen before. I logged this with Apache, but as I can’t ship a custom ROM with my SVG parser I decided to replace it.
Here’s the results from some tests of 3896 calls to parseFloat on my Nexus One:

  • a) Using Float.parseFloat(String): 1516.449ms
  • b) My own parseFloat(String): 654.882ms
  • c) …and my own isDigit(char): 449.622
  • d) …and isDigit written inline: 295.865ms
  • e) …copying the string into a char buffer and avoiding all string access and method calls: 254.527ms (112ms spent copying the characters!)

So in theory, if the method took a character buffer and length it could be as short as 135ms, over ten times faster than Float.parseFloat. I’m currently using option d as gives a nice performance gain without breaking my design, but when performance tuning the entire API later this may see some huge changes. Here’s the function I cooked up:

public static float parseFloat(String f) {
	final int len   = f.length();
	float     ret   = 0f;         // return value
	int       pos   = 0;          // read pointer position
	int       part  = 0;          // the current part (int, float and sci parts of the number)
	boolean   neg   = false;      // true if part is a negative number
 
	// find start
	while (pos < len && (f.charAt(pos) < '0' || f.charAt(pos) > '9') && f.charAt(pos) != '-' && f.charAt(pos) != '.')
		pos++;
 
	// sign
	if (f.charAt(pos) == '-') { 
		neg = true; 
		pos++; 
	}
 
	// integer part
	while (pos < len && !(f.charAt(pos) > '9' || f.charAt(pos) < '0'))
		part = part*10 + (f.charAt(pos++) - '0');
	ret = neg ? (float)(part*-1) : (float)part;
 
	// float part
	if (pos < len && f.charAt(pos) == '.') {
		pos++;
		int mul = 1;
		part = 0;
		while (pos < len && !(f.charAt(pos) > '9' || f.charAt(pos) < '0')) {
			part = part*10 + (f.charAt(pos) - '0'); 
			mul*=10; pos++;
		}
		ret = neg ? ret - (float)part / (float)mul : ret + (float)part / (float)mul;
	}
 
	// scientific part
	if (pos < len && (f.charAt(pos) == 'e' || f.charAt(pos) == 'E')) {
		pos++;
		neg = (f.charAt(pos) == '-'); pos++;
		part = 0;
		while (pos < len && !(f.charAt(pos) > '9' || f.charAt(pos) < '0')) {
			part = part*10 + (f.charAt(pos++) - '0'); 
		}
		if (neg)
			ret = ret / (float)Math.pow(10, part);
		else
			ret = ret * (float)Math.pow(10, part);
	}	
	return ret;
}

This is of course a very crude method, it requires a well-formed number and will return 0 or partial numbers instead of raising exceptions, it doesn’t work with hex floats or parse NaNs or Infinity and it doesn’t use doubles internally so the accuracy is not perfect. It does however work for my purposes, so I thought I’d share it with the Internet.

JPerfmeter crashes while starting

Posted by – 2010/05/17

I’ve been us JPerfmeter in work to keep an eye on Linux system counters served up over rstatd.

However, it has one very annoying bug:

If one of your servers goes offline, or it somehow lets you add a server which doesn’t exist JPerfmeter will crash and refuse to start again. The setting is saved somewhere on disk. Where? Google didn’t know (hence this blog post).

Half an hour of messing around in Procmon going through reams of logs and I find the settings in %userprofile%\JPerfmeter, just where you’d expect them to be. facepalm.jpg

Stripping snapshot files from LoadRunner Citrix scripts

Posted by – 2009/10/29

After several re-records of a Citrix script in LoadRunner, thousands of PNG snapshot files build up in the data dir. Over time they can build up to make your script eat up over a hundred megabytes of disk space, which is a little excessive given that the useful parts are only a few megs in total.

These snapshots are used to make the script tree-view useful, so you can’t delete all of them in case you need a reference image when inserting a new sync, keypress or some other event. So, I squirted out this little VB script which will remove unused stuff from the data directory, I’m sharing it with the Internet because I’m nice like that.

You can download it here.

To run it, just copy to the dir which contains your scripts (back them up first, I offer no warranty or compensation in case things go wrong) and click it. Be prepared to wait a long time, enumerating forty thousand bitmaps will always take some time.

When archiving your scripts you might also want to delete any results data, which also costs a few megs.

Citrix of the Trade

Posted by – 2009/10/23

I’ve been using LoadRunner for Citrix for about a week now, it’s a new protocol for me and I guess I’m starting to get the hang of it because I’ve created a list of tips and tricks to share with the Internet.

I’m assuming you’ve read the other guides, about using the keyboard where possible, making sure your colour settings and screen sizes are the same on each injector, using bitmap and text checks for synchronization, clicking on menus because mouse movements don’t work and so on.

Don’t trust LoadRunner to get your transactions right

As of version 9.5, LoadRunner records Citrix scripts which look like this:

lr_start_transaction("Receive bacon");
ctrx_sync_window("Bacon dispenser", ACTIVE, 100, 100, 200, 200, etc);
// push button
ctrx_mouse_click( 50, 50, 0, CTRX_LAST);
lr_end_transaction("Receive bacon", LR_AUTO);
 
lr_start_transaction("Enjoy bacon");
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);

In this case, at recording we put transactions around the time taken to push the button and get some bacon, not how long it takes to focus the bacon dispenser and then push the button. A transaction is all about timing something, which means you do something and then wait for a response; a transaction should never start with a sync and should almost always end with one. So, we need to edit the Receive bacon transaction to look something like this:

lr_start_transaction("Receive bacon");
// push button
ctrx_mouse_click(50, 50, 0, CTRX_LAST);
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);
lr_end_transaction("Receive bacon", LR_AUTO);

Don’t trust LoadRunner to insert think-times of the correct length, nor in the correct places.

This one is an even more painful schoolboy error, let’s use the bacon meme as an example again and a more realistic recording including recorded thinking time, LR would generate something like this:

lr_start_transaction("Receive bacon");
ctrx_sync_window("Bacon dispenser", ACTIVE, 100, 100, 200, 200, etc);
// push button
ctrx_mouse_click( 50, 50, 0, CTRX_LAST);
lr_think_time(15);
lr_end_transaction("Receive bacon", LR_AUTO);
 
lr_start_transaction("Enjoy bacon");
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);

Did you spot it? Firstly the think time is incredibly large because LoadRunner’s Citrix client recorder is obscenely slow, so you’ll have to adjust the value to the amount of time the user would actually spend marvelling at their delicious bacon before eating it. This would be far less than 15 seconds. Let’s also apply the fix from above:

lr_start_transaction("Receive bacon");
// push button
ctrx_mouse_click(50, 50, 0, CTRX_LAST);
lr_think_time(3);
ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);
lr_end_transaction("Receive bacon", LR_AUTO);

Now the glaring error becomes obvious: The think-time and the sync are the wrong way around! If you don’t strip think times from the results, you can only report on timings of 3 seconds and over. That’s just sloppy, but much worse than that, if you do strip the think time then you’ll report timings 3 seconds shorter than the system actually takes to respond:


Yeah that means that if your scripters don’t know what they’re doing then you run the risk of reporting great timings for an abysmally slow system. So SLAs aren’t met, your company receives huge fines, or in the case of the system I’m working on lives may even be lost.

Ignore the replay guidelines published by Citrix.

Do not replay think-time when debugging scripts, this will force you to write robust scripts that measure rather than relying on the system response times. You won’t be tempted to resort to extending the think-times to fudge synchronisation when you get a huge number of users running.

When you can’t trust a bitmap, cheat with events

Sometimes you can’t trust a bitmap synchronization because the application has painted, but is still processing and will not respond to mouse input. Keyboard input isn’t so much of a problem as it will queue up, but mouse events may be discarded. Thankfully we know that MFC, VB and Java applications are event driven. This means we can use bitmaps synchronizations on things triggered by mouse movement and hover events, then do a click after that.

For example, move the mouse back and forth across a toolbar button, wait for a short period of time (use sleep(), not think time!), then do a check (not a sync) on the bevel which is drawn around it. The same applies to other mouse-over events, if it can take a mouse-over it can probably take a click event. Unfortunately LR can’t detect mouse cursor changes, which would be a great tool for this purpose.

Another similar idea is to use the text buffer; input a character or set the focus with the tab key, then wait for the change in state.

You can’t sync on a context menu

They don’t have a name so they get an obscure number at record time, which cannot be found at playback. Use bitmap checks on the screen area for these instead. Of course, you should really be using the keyboard where possible, but this isn’t always possible (accessawhatability?)

The pixel-sync trick

Record a 3×3 pixel bitmap check for white, grey and declare or #define them as functions like so:

#define Sync3x3(x,y,hash) ctrx_sync_on_bitmap(x, y, 3, 3, hash, CTRX_LAST)
#define WHITE(x,y) Sync3x3(x,y,"794f0585be94ed32b0fe3d42f8917eea")
#define GREY(x,y) Sync3x3(x,y,"2e32a138b3fee935c6efabeb24c05733")
#define BLUE(x,y) Sync3x3(x,y,"7c7ebab71cd7ebeab3f0aa710501fa07")

Then, providing you’ve been a good boy and remembered to maximize your window before recording, you can avoid re-recording when you need a new sync point. Just insert an object click via the GUI tree-view and replace it with one of the macros above in the code. Please note, your grey colour may be different to mine, it’s best to record your own pixels. Also don’t bother with a 1×1 pixel, LoadRunner always fails those checks. (I didn’t try a 2×2 check)

Remove those rogue mouse-up events

After drawing the rectangle for a bitmap or text synchronization point, you’ll often get a rogue mouse-up event. Remove those, they’re a confusing waste of processing time and won’t happen in the real world.

Conclusion

The industry standard load testing tool is simple enough for trained monkeys to use, but its default Citrix settings suck to the point of being dangerous. Handle with care!

That’s all I’ve learned this week, hopefully it’s enough for me me to help deliver this project on time. Expect more similar rants and tips over the coming months.

Loadrunner DEP closes iexplore.exe

Posted by – 2009/10/22

Search spam time!

When launching LoadRunner for the first time on a Windows XP or 2003 machine, VUGen may record no events and then complain that the Data Execution Prevention Service closed iexplore.exe.

In this case, go to Control Panel -> System -> Advanced -> Performance -> Settings -> Data Execution Prevention -> and select “Turn on DEP for essential windows programs and services only.”

Now reboot your computer/server.

It’s been a while since I used LoadRunner, as you can probably tell. Win2K desktops and XP machines before SP2 didn’t have this problem.

Be Internet rebranding fiasco

Posted by – 2009/07/22

My ISP, Be Internet just went live with their new website and even though I’m a tiny minority in thinking this, I quite like the new layout. It has REST-like URLs and social media bookmarks, it even uses the strict XHTML DTD, which is refreshing.

However, the design is the only thing I like! The new site has not only collapsed under the load, but users are reporting that they are seeing other people’s account details when they log in! Also their SSL certificate is invalid, so you’ll need to add an exception before you can even visit the site. Their nice new Digg account is apparently “A 14 year-old person from United Kindgom (GB)

Did Be actually do any testing before going live?

Leaky account details

Leaky account details

I’d be shocked and disgusted like the other users if I wasn’t a performance tester, screw ups like this are a wonderful advert for people like me!