Category: Development

Float.parseFloat is slow

Posted by – 2010/08/27

While profiling some loader code and expecting to see performance issues with my regular expressions, I found that ~60% of my time was actually spent converting String variables to floats. It turns out that the slow bit lives inside Float.parseFloat, where it calls String.toLowerCase every single time just in case this string is a hex float, a very rare type of float which I had never seen before. I logged this with Apache, but as I can’t ship a custom ROM with my SVG parser I decided to replace it.
Here’s the results from some tests of 3896 calls to parseFloat on my Nexus One:

  • a) Using Float.parseFloat(String): 1516.449ms
  • b) My own parseFloat(String): 654.882ms
  • c) …and my own isDigit(char): 449.622
  • d) …and isDigit written inline: 295.865ms
  • e) …copying the string into a char buffer and avoiding all string access and method calls: 254.527ms (112ms spent copying the characters!)

So in theory, if the method took a character buffer and length it could be as short as 135ms, over ten times faster than Float.parseFloat. I’m currently using option d as gives a nice performance gain without breaking my design, but when performance tuning the entire API later this may see some huge changes. Here’s the function I cooked up:

public static float parseFloat(String f) {
	final int len   = f.length();
	float     ret   = 0f;         // return value
	int       pos   = 0;          // read pointer position
	int       part  = 0;          // the current part (int, float and sci parts of the number)
	boolean   neg   = false;      // true if part is a negative number
 
	// find start
	while (pos < len && (f.charAt(pos) < '0' || f.charAt(pos) > '9') && f.charAt(pos) != '-' && f.charAt(pos) != '.')
		pos++;
 
	// sign
	if (f.charAt(pos) == '-') { 
		neg = true; 
		pos++; 
	}
 
	// integer part
	while (pos < len && !(f.charAt(pos) > '9' || f.charAt(pos) < '0'))
		part = part*10 + (f.charAt(pos++) - '0');
	ret = neg ? (float)(part*-1) : (float)part;
 
	// float part
	if (pos < len && f.charAt(pos) == '.') {
		pos++;
		int mul = 1;
		part = 0;
		while (pos < len && !(f.charAt(pos) > '9' || f.charAt(pos) < '0')) {
			part = part*10 + (f.charAt(pos) - '0'); 
			mul*=10; pos++;
		}
		ret = neg ? ret - (float)part / (float)mul : ret + (float)part / (float)mul;
	}
 
	// scientific part
	if (pos < len && (f.charAt(pos) == 'e' || f.charAt(pos) == 'E')) {
		pos++;
		neg = (f.charAt(pos) == '-'); pos++;
		part = 0;
		while (pos < len && !(f.charAt(pos) > '9' || f.charAt(pos) < '0')) {
			part = part*10 + (f.charAt(pos++) - '0'); 
		}
		if (neg)
			ret = ret / (float)Math.pow(10, part);
		else
			ret = ret * (float)Math.pow(10, part);
	}	
	return ret;
}

This is of course a very crude method, it requires a well-formed number and will return 0 or partial numbers instead of raising exceptions, it doesn’t work with hex floats or parse NaNs or Infinity and it doesn’t use doubles internally so the accuracy is not perfect. It does however work for my purposes, so I thought I’d share it with the Internet.

Microphone for Android

Posted by – 2010/08/22

So, my 4th Android app is out, a simple mic which you can plug into your computer. It’s got quite a slating on the Android Market because of its latency, the problem is that there’s quite large minimum values on the buffer size for accessing input and output buffers, there’s no way to reduce this latency below the ~300ms which it currently has. Hopefully AOSP will address this eventually, paving way for realtime audio applications.

Adventures in Android

Posted by – 2010/08/17

Since getting myself a Google Nexus One I’ve decided to do a bit of Android development. My first impressions with the platform weren’t really that good, but I’m beginning to get the hang of it now.

My main problem is that Java is a horribly verbose and restrictive language. To do the simplest things you need to implement at least three interfaces and extend a couple of abstract classes (after reading the documentation for each of them and their alternatives of course). Something which could be written in Python in three lines of code and two minutes takes fifty lines and twenty minutes in Java. Perhaps that’s just me, maybe I’ve been spoiled by dynamic languages and expect to be able to do things a hundred different ways by just typing whatever comes to mind, rather than carefully considering the available options and planning every step of my app. In my short journey through Java development I’ve suffered a lot of pain because I make assumptions and choose a design pattern which is incompatible some point later down the line, so I end up refactoring everything ten times before I even get a working prototype. I’m slowly learning not to fly by the seat of my pants (even if it is the most fun way to fly) and perhaps that’s a good thing.

My other gripe is the UI editor and XML format, their steep learning curve makes the design stage a huge time sink. I’ve now spent a fair bit of time banging my head off my keyboard while attempting to make the simplest of interface, which still don’t look like they do on paper, but I think I’m getting the hang of it.

Anyhow, so far I’ve managed to knock out 3 applications which are available in the Android marketplace:

Anonypic

This little app just adds an extra option to your share menu, and will upload an image to bayimg.com, the anonymous image sharing service by the creators of The Pirate Bay. Images have their EXIF data removed, so you can share images without leaving a trail of metadata leading back to your phone.

Rainwatch

Rainwatch is an alternative viewer for the BBC’s Maps Presenter, the Flash application available on their weather site. It’s far from complete but is good enough to predict the weather, so I’m quite happy with it (as are my 1500 active users)

MC Mail

MC Mail, or Missed Call Mailer is a little app which sends you an email when you get a missed call. The back-end is written in PHP and MySQL, the front-end is mostly just a service which monitors your call log. Useful if you’re a hospital or call centre worker who has access to email but can’t use a phone in work, or if you always forget your phone. It usually only works while the phone is in silent mode.

Next steps?

I’ll make a couple more apps and maybe a game, then learn the Native Development Kit and port Irrlicht using Christian’s OpenGL ES driver. I haven’t done any Irrlicht code in ages, so that’s certainly on the agenda.

Automatic favicon to PNG script

Posted by – 2010/05/16

Ever want to add a link to a site with the favourites icon in the page along with the link? I did, so I made this cool little toy:

http://favicon.bitplane.net/

read more…

code_swarm

Posted by – 2010/02/14

I’ve been playing with code_swarm and made this video of the Irrlicht Engine’s development history:

Since the video is generated from the SVN version history it starts at the point where Niko gave hybrid and I SVN access (June 2006), so misses out years of development leading up to the 1.0 release.

code_swarm is a really cool tool, one thing which is exciting is the fact it supports MediaWiki. It would be really nice to show a small wiki coming together.. or maybe one day Wikipedia’s entire changelog (we can dream!)

Interpolated list class for Python

Posted by – 2009/12/12

While messing with Python to output some better graphs for my Facebook group scraper I stumbled upon an interesting problem. What happens if you have missing samples, or want to change the sampling frequency half way through your log? Well, you could use a proper math library and have extra dependencies or generate the missing data manually, but the simplest answer I could think of was to write a list class which interpolates between missing values, so you can do something like this:

>>> a = InterpoList()
>>> a[0]   = 0
>>> a[100] = 200
>>> a[200] = 0
>>> a[50]
100.0
>>> a[125]
150.0

And then store the time values as seconds past the epoch. Now I can inject data from multiple sources taken at different frequencies and retrieve an interpolated value for any time, which means more nice graphs like this:

As usual I also decided to share it with the Internet, because I’m nice like that! You can get a copy of the InterpoList module here.

Irrlicht 1.6 Released

Posted by – 2009/09/30

So, we finally released Irrlicht 1.6. This release has lots of bug fixes and plenty of new features for you to sink your teeth into!

I’m sure I’ve missed out lots of stuff, but here’s a summary of the most interesting new features…

Video

Burning’s software rasterizer is now at version 0.44 and comes with sphere map, two layer reflection and transparent alpha ref (plant renderer) materials, plus lots more enhancements.

It’s now possible to specify an override material for the video driver. This could be used, for example, to draw the entire scene in wireframe. It is incredibly useful for deferred rendering where you need to render the entire scene using some specific shader.

New SMaterial settings. Materials now support LoD bias for mipmaps, antialias on a per-material basis, and a Swiss Army Knife of a feature: disabling writes for specific colour channels. Combined with the override material this makes stereoscopic anaglyph rendering really simple:

Luke demonstrates red-cyan anaglyph rendering in 12 lines of code per eye!

Floating point render pixel formats are now supported, currently only in render targets in OpenGL and D3D, but it still means HDR rendering! The colour converter doesn’t yet support these new formats, but in future this will mean floating point image loaders meaning native HDR support across the board:

FuzzYspo0n's reflections (BlindSide's method, I think) using SteelStyle's HDR image loader to load floating point textures

Direct3D8 finally supports 32-bit index buffers. The D3D drivers can now be used in the SDL device, and they now render in screen space in 2D mode (0,0 -> W,H).

Gary Conway donated an SGI image loader, which means Irrlicht can now open Silicon Graphics .rgb, .rgba, .sgi, .int and .inta textures, including RLE compressed ones.

The TGA loader can now load uncompressed 8-bit greyscale and indexed images.

Core

position2d has been removed, it is now a just a typedef for vector2d. This gets rid of the need for all those ugly vector to position conversions, even though they are essentially the same type.

Device

Irrlicht now has a console device. I added a simple ASCII render just for fun, but the console device has a serious use, you can use Irrlicht to do offline rendering without the need to ever create a window which means it doesn’t need an X server on Unices.

The console device can present images from the software rasterizers

Irrlicht can now also be compiled with more than one device type and select the best one at runtime. This does increase the size of the library, on OSX you have the option to compile with the console, Cocoa, SDL and X11 devices which may seem like overkill. The ability to select the device type at runtime allows the same version of the shared library to be used with a wider range of applications, for example my GNOME thumbnailer which runs in the background:

Nautilus showing thumbnails for Irrlicht supported meshes

Nautilus showing thumbnails for Irrlicht supported meshes

We now have an Xbox project too, apparently compiling with Direct3D8 on the official XDK, but OpenXDK support with SDL is planned.

The device now has get/setGammaRamp functions.

Scene

RogerBorg’s light manager is in at last! Using a light manager you can add hundreds of lights to the scene and activate the closest 8 to each surface:

RogerBorg's light manager made it in

RogerBorg's light manager made it in

Collision enhancements. You can now get the node which was selected in getCollisionPoint, making it easy to work with meta selectors. ICollisionCallback so you can be notified of collision events.

ISceneNodeAnimators now have an hasFinished method, so you know when they’re complete.

Irrlicht can now load and save PLY meshes, including massive ones. This is the file format used by the infamous Stanford Bunny but also by commercial 3D scanners in general:

Loading a massive PLY mesh from a 3D scanner. There are no UV coords here, this is all vertex alpha

Quake3 BSPs support is now greatly enhanced, supporting new shader types and lots of other stuff.

GUI

The GUI now comes with a tree, which is demonstrated in the new Quake Map Explorer example:

Quake Map Explorer

Tommi's Quake Explorer demonstrates (among other things) use of the tree view, donated by Reinhard Ostermeier

New skin constants for text offsets in GUI windows, allowing nice skinned titlebars like this:

Klaskers GUI skinner, with skinned by FuzzYspo0N

File System

Irrlicht’s IFileSystem now supports pluggable archive loaders via two new interfaces, IFileArchive and IArchiveLoader. This allows users to extend the VFS by adding new archive types, you can for example lock your data up in a proprietary archive format, store your meshes in a database, or add ways of extracting data from current archive formats. Currently archives are still read only, though this may change in future.

To celebrate this, Irrlicht can now open .tar.gz archives thanks to the new TAR reader and the fact that the ZIP reader can now open gzip archives.

I’m adding two examples to IrrExt. Firstly a Windows .ico and .cur loader which treats these files like an archive. This allows you to (for example) use Windows mouse cursors or favicon.ico files from web pages. Secondly, a Windows PE RSRC loader, which allows you to load resources from Windows binaries in a cross platform way. Combined with the .ico loader you can use this to show the icons of Windows executables from inside Irrlicht. The RSRC loader will be moved over to IrrExt in a few days, once I’ve updated it to work with Irrlicht 1.6.

There’s now a special type for holding file names and paths. io::path is currently a string type which may be wchar_t based in Windows. In future all string based path comparison, replacement and collapse methods will move to this class.

Songs in code

Posted by – 2009/09/15

I’m a bit late on commenting on this Twitter meme as it’s over a month old, like all memes it’s a variation of something as old as the Earth and just recently revived. The concept is simple, just write a song lyric or title as a code snippet, set it as your status and share with others by linking the #songsincode hashtag. That’s all.

Someone’s even made a database of all the posts where you can vote on the entries. Most are boring and just consist of lyrics padded out by punctuation, but there are some quite creative, non-obvious ones which use programming constructs to infer the lyrics. Here’s mine

(Hover over the links for spoilers)

class BillieJean : public girl
{ 
  int i; 
  relationship lover;
public: 
  BillieJean(relationship &me) 
  : lover(!me), i(1) {} 
};

spoil

body() { 
  print '<body>' . body() . '</body>'; 
} 
love() { 
  body();
}

spoil

/* mon = 255; tue=wed=0xaaaaaa; */ 
if (friday)
  love.insert(me);

spoil

GET /back.html HTTP/1.1
 
HTTP/1.1 301
Location: /you_once_belonged.html

spoil

Some of my favourites (with slight edits):

video$ kill -9 `ps -ef | grep radio*`

by brozow

int main() {
  return -1;
  get_Satisfaction();
}

by stevecooperorg

$live = ($you==true) ? false : false;

by o_Freud_o

$i = '<HTML><HEAD></HEAD></HTML>';

by whybird

class City : Rock, Roll {};

by alexmdac

try {
  get(WHAT_YOU_WANT);
} finally {
  get(WHAT_YOU_NEED);
}

by rickgrundy

/dev/null << shout();

by plikrg

PHP file list using mod_rewrite

Posted by – 2009/09/14

I’ve noticed that most of the traffic to this site is actually people visiting my permanent and temporary file dumps, which until today were literally a bunch of files dumped in a folder. DreamHost don’t keep my logs for over 30 days and I’d like to incorporate my file dumps into my Piwik stats, so I made a nice fancy file list in PHP. It has nice looking icons courtesy of famfamfam, though may move to stdicon in the future.

Some caveats:

  • The file list currently tracks directory views (via the template) but not yet file downloads, as that requires a Piwik plugin that isn’t final so I haven’t installed it, adding it should be trivial though.
  • As requests are passed through PHP using mod_rewrite it will result in long running scripts on your server, which may upset your shared web hosting provider.
  • Scripts are not yet excluded by rewrite rules! You won’t want to keep other scripts containing sensitive information in dirs under the path.
  • Access to files and folders beginning with “.” will be blocked.
  • It only works on Unices, not Windows servers as they don’t have the “file” command to get the MIME type.
  • You’ll obviously need an Apache installation with mod_rewrite.

To install it, just copy index.php, .htaccess and .filelist/ to the directory where you store your files. You can get the script (without icons) from SVN.

This [e107] page has moved permanently [to gallery2]

Posted by – 2009/09/06

If you do a Google search for some of my really old code, like my Python T9 Dictionary or XBMC IRC MP3 downloader, you’ll notice that my pagerank has been carried over to SVN and the gallery. This is because Google and other search engines take note of HTTP 301 response codes and update their links accordingly.

When I migrated from e107 to WordPress and Gallery2, I replaced download.php with a redirect script which redirects to the gallery, svn or file dumps. The script looks like this:

<?php
// lots excluded
$table['4']='http://svn.bitplane.net/misc/trunk/py/py9/readme.txt';
$address = 'http://bitplane.net/';
 
foreach ($_GET as $i => $value)
{
  $index = explode('_', $i);
  if ( isset($index[1]) && isset( $table[ $index[1] ]) )
  {
    $address = $table[ $index[1] ];
  }
}
 
// send 301 (permanent redirect)
header("HTTP/1.1 301 Moved Permanently");
header("Location: " . $address );
?>
<html><head></head><body>
This file has permanently moved <a href="<? echo $address; ?>">here</a>
</body></html>

So when you try to view PY9, you’re redirected to the new location. There’s no excuse for 404 errors when you move your site around, if you care about keeping the web up-to-date then please use a redirect script!