Last Monday I decided to pull out all the stops and actually get a job, meaning stop playing Minecraft, update my CV and pass it to a dozen or so agencies who appear to exist for the sole purpose of creaming money off the top of my daily rate. By Wednesday I had an interview lined up with a client, providing I pass a Loadrunner exam provided by IKM Europe. Sounds like fun… let’s do this!
Here’s a little script to format vmstat’s output into something you can import into LoadRunner’s analysis tool:
import sys, time sys.stdout.write("Date,Time," "Proc Run,Proc sleep," "Mem swap use,Mem free,Mem buffered,Mem Cache," "Swap in,Swap out,Blocks in,Blocks out," "Interrupts,Context switches," "CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen\n") for line in iter(sys.stdin.readline, ""): if line != 'p' and line[0:2] != ' r': t = time.strftime("%d/%m/%Y,%H:%M:%S", time.localtime(time.time())) line = ' '.join(line.split()) line = line.replace(" ", ",") sys.stdout.write("%s,%s\n" % (t, line)) sys.stdout.flush()
(Thanks to fabrizoM for the help!)
Run it from a shell script like so:
#!/bin/bash vmstat 5 | python vmstat2csv.py >> servername-vmstat.log
Then start the job from the shell, press CTRL+Z and nohup it (so it doesn’t die if you disconnect) and tail it (so you can keep an eye on it) like so:
+ Stopped ./log.sh [gaz@box ~]$ bg + ./log.sh & [gaz@box ~]$ disown [gaz@box ~]$ tail -f servername-vmstat.log Date,Time,Proc Run,Proc sleep,Mem swap use,Mem free,Mem buffered,Mem Cache,Swap in,Swap out,Blocks in,Blocks out,Interrupts,Context switches,CPU %User,CPU %System,CPU %Idle,CPU %Wait,CPU %Stolen 23/11/2010,12:59:47,1,0,0,1266452,465576,2074012,0,0,2,15,0,36,0,0,99,0,0 23/11/2010,13:00:11,0,0,0,1264460,465584,2074024,0,0,0,168,581,577,1,1,94,4,0
When you’re done, import it into the Analysis tool via the Tools -> External Monitors menu. Make sure you set the correct date for the custom format and remember to keep a close eye on the timezones. If you make a mistake while importing there’s no going back, so save a backup of your analysis files before you attempt this.
LoadRunner 9.5′s VUGen sometimes gets its undergarments in a twist and messes up the actions in one of its INI files. You don’t notice this until you get the dreaded “Invalid Action” error message while running a test, which appears once for each virtual user and prevents other users from loading.
Thankfully, Sameh Abdelhamid found and documented a proper fix for this problem:
Find a file called default.usp and edit it in your favorite text editor. I use ScITE which comes with ruby. You will notice a parameter called [Profile Actions].
That will list your actions. If you cross check this with the actions in your script you will notice that in my example, I only have 3 actions, whereas, the default.ups has 4.
Which is great, but no good for me right now as it started happening half way into a test which I can’t cancel. If like me, you need that horrible, dirty, reactive solution right now, here’s a little VB Script to send a Y key to the messagebox every time it appears.
Dim Wsh Set Wsh = Wscript.CreateObject("Wscript.Shell") While True If Wsh.AppActivate("Invalid Action") Then Wsh.SendKeys "y" WScript.echo "go away!" End if WScript.Sleep 1000 Wend
After several re-records of a Citrix script in LoadRunner, thousands of PNG snapshot files build up in the data dir. Over time they can build up to make your script eat up over a hundred megabytes of disk space, which is a little excessive given that the useful parts are only a few megs in total.
These snapshots are used to make the script tree-view useful, so you can’t delete all of them in case you need a reference image when inserting a new sync, keypress or some other event. So, I squirted out this little VB script which will remove unused stuff from the data directory, I’m sharing it with the Internet because I’m nice like that.
You can download it here.
To run it, just copy to the dir which contains your scripts (back them up first, I offer no warranty or compensation in case things go wrong) and click it. Be prepared to wait a long time, enumerating forty thousand bitmaps will always take some time.
When archiving your scripts you might also want to delete any results data, which also costs a few megs.
I’ve been using LoadRunner for Citrix for about a week now, it’s a new protocol for me and I guess I’m starting to get the hang of it because I’ve created a list of tips and tricks to share with the Internet.
I’m assuming you’ve read the other guides, about using the keyboard where possible, making sure your colour settings and screen sizes are the same on each injector, using bitmap and text checks for synchronization, clicking on menus because mouse movements don’t work and so on.
Don’t trust LoadRunner to get your transactions right
As of version 9.5, LoadRunner records Citrix scripts which look like this:
lr_start_transaction("Receive bacon"); ctrx_sync_window("Bacon dispenser", ACTIVE, 100, 100, 200, 200, etc); // push button ctrx_mouse_click( 50, 50, 0, CTRX_LAST); lr_end_transaction("Receive bacon", LR_AUTO); lr_start_transaction("Enjoy bacon"); ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);
In this case, at recording we put transactions around the time taken to push the button and get some bacon, not how long it takes to focus the bacon dispenser and then push the button. A transaction is all about timing something, which means you do something and then wait for a response; a transaction should never start with a sync and should almost always end with one. So, we need to edit the Receive bacon transaction to look something like this:
lr_start_transaction("Receive bacon"); // push button ctrx_mouse_click(50, 50, 0, CTRX_LAST); ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc); lr_end_transaction("Receive bacon", LR_AUTO);
Don’t trust LoadRunner to insert think-times of the correct length, nor in the correct places.
This one is an even more painful schoolboy error, let’s use the bacon meme as an example again and a more realistic recording including recorded thinking time, LR would generate something like this:
lr_start_transaction("Receive bacon"); ctrx_sync_window("Bacon dispenser", ACTIVE, 100, 100, 200, 200, etc); // push button ctrx_mouse_click( 50, 50, 0, CTRX_LAST); lr_think_time(15); lr_end_transaction("Receive bacon", LR_AUTO); lr_start_transaction("Enjoy bacon"); ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc);
Did you spot it? Firstly the think time is incredibly large because LoadRunner’s Citrix client recorder is obscenely slow, so you’ll have to adjust the value to the amount of time the user would actually spend marvelling at their delicious bacon before eating it. This would be far less than 15 seconds. Let’s also apply the fix from above:
lr_start_transaction("Receive bacon"); // push button ctrx_mouse_click(50, 50, 0, CTRX_LAST); lr_think_time(3); ctrx_sync_window("Bacon", ACTIVE, 100, 100, 200, 200, etc); lr_end_transaction("Receive bacon", LR_AUTO);
Now the glaring error becomes obvious: The think-time and the sync are the wrong way around! If you don’t strip think times from the results, you can only report on timings of 3 seconds and over. That’s just sloppy, but much worse than that, if you do strip the think time then you’ll report timings 3 seconds shorter than the system actually takes to respond:
Yeah that means that if your scripters don’t know what they’re doing then you run the risk of reporting great timings for an abysmally slow system. So SLAs aren’t met, your company receives huge fines, or in the case of the system I’m working on lives may even be lost.
Ignore the replay guidelines published by Citrix.
Do not replay think-time when debugging scripts, this will force you to write robust scripts that measure rather than relying on the system response times. You won’t be tempted to resort to extending the think-times to fudge synchronisation when you get a huge number of users running.
When you can’t trust a bitmap, cheat with events
Sometimes you can’t trust a bitmap synchronization because the application has painted, but is still processing and will not respond to mouse input. Keyboard input isn’t so much of a problem as it will queue up, but mouse events may be discarded. Thankfully we know that MFC, VB and Java applications are event driven. This means we can use bitmaps synchronizations on things triggered by mouse movement and hover events, then do a click after that.
For example, move the mouse back and forth across a toolbar button, wait for a short period of time (use sleep(), not think time!), then do a check (not a sync) on the bevel which is drawn around it. The same applies to other mouse-over events, if it can take a mouse-over it can probably take a click event. Unfortunately LR can’t detect mouse cursor changes, which would be a great tool for this purpose.
Another similar idea is to use the text buffer; input a character or set the focus with the tab key, then wait for the change in state.
You can’t sync on a context menu
They don’t have a name so they get an obscure number at record time, which cannot be found at playback. Use bitmap checks on the screen area for these instead. Of course, you should really be using the keyboard where possible, but this isn’t always possible (accessawhatability?)
The pixel-sync trick
Record a 3×3 pixel bitmap check for white, grey and declare or #define them as functions like so:
#define Sync3x3(x,y,hash) ctrx_sync_on_bitmap(x, y, 3, 3, hash, CTRX_LAST) #define WHITE(x,y) Sync3x3(x,y,"794f0585be94ed32b0fe3d42f8917eea") #define GREY(x,y) Sync3x3(x,y,"2e32a138b3fee935c6efabeb24c05733") #define BLUE(x,y) Sync3x3(x,y,"7c7ebab71cd7ebeab3f0aa710501fa07")
Then, providing you’ve been a good boy and remembered to maximize your window before recording, you can avoid re-recording when you need a new sync point. Just insert an object click via the GUI tree-view and replace it with one of the macros above in the code. Please note, your grey colour may be different to mine, it’s best to record your own pixels. Also don’t bother with a 1×1 pixel, LoadRunner always fails those checks. (I didn’t try a 2×2 check)
Remove those rogue mouse-up events
After drawing the rectangle for a bitmap or text synchronization point, you’ll often get a rogue mouse-up event. Remove those, they’re a confusing waste of processing time and won’t happen in the real world.
The industry standard load testing tool is simple enough for trained monkeys to use, but its default Citrix settings suck to the point of being dangerous. Handle with care!
That’s all I’ve learned this week, hopefully it’s enough for me me to help deliver this project on time. Expect more similar rants and tips over the coming months.
Search spam time!
When launching LoadRunner for the first time on a Windows XP or 2003 machine, VUGen may record no events and then complain that the Data Execution Prevention Service closed iexplore.exe.
In this case, go to Control Panel -> System -> Advanced -> Performance -> Settings -> Data Execution Prevention -> and select “Turn on DEP for essential windows programs and services only.”
Now reboot your computer/server.
It’s been a while since I used LoadRunner, as you can probably tell. Win2K desktops and XP machines before SP2 didn’t have this problem.