AM Stereo

Inc.com has a great article interviewing Alex Rigopulos, co-Founder and CEO of Harmonix (makers of the video games Rock Band, Guitar Hero, Frequency, etc.) about Harmonix’s long road to success. I’ve been a fan of their games ever since Frequency on the PS2 (4-player XLR8R!), and it’s great  — both as a game consumer and as a fellow software startup entrepreneur — to see them hit it big.

Alex also gave a great speech at the DICE 2007 conference, called “Living the Dream” — Coding Horror has a good writeup of it. I’ll give you a sneak peek of one slide, as it shows what life is like for many startups (and most don’t make it 10 years):

And now I’m feeling nostalgic… so here’s a song/level from what I think of as Harmonix’s first game:

I love that Austin’s mayor is the kind of guy who would lead a Thriller dance performance:

After trying some other load-testing tools, I found Tsung, a load-testing application written in Erlang to take advantage of that language’s concurrency support. It scales well (it’s been used to simulate tens of thousands of users), it supports forms and HTTP sessions, and includes some niceties like proxy recording, ‘thinktime’ support, and a choice of random or ordered traffic.

Not having a physical server handy to run Tsung isn’t a big obstacle; I just used an Amazon EC2 instance (an xlarge turned out to be more than enough, you could probably get 100s of users on a medium), which due to being rentable by the hour is great for a short-term need like occasional load-testing. I put one of Alestic’s Ubuntu images on the EC2 instance, and then it’s just a few short steps to install Tsung on the machine (we’ll call it ‘tsunghost’):

1. sudo apt-get install erlang
2. sudo apt-get install erlang-src
3. sudo apt-get install gnuplot-nox sudo apt-get install libtemplate-perl libhtml-template-perl libhtml-template-expr-perl
4. Download the latest from http://tsung.erlang-projects.org/dist/ and install it (the usual configure, make, make install). I prefixed it into /opt/tsung just to keep it a little cleaner, so the following will reflect that.

Update: Ben says in the comments, “Tsung now has Ubuntu packages, so you don’t need to compile. You can grab the latest deb at http://tsung.erlang-projects.org/dist/ubuntu/ .  Then just sudo dpkg -i on it.”

The first thing you’ll want to do is record a sample web session. Start Tsung in record mode with ‘/opt/bin/tsung -L 9000 recorder’ (the -L bit indicates the port number that the proxy will listen on). Now, you can set tsunghost:9000 as the HTTP proxy in your web browser and all the web activity will be recorded by Tsung for playback later.

With Tsung recording, exercise your web application — but keep in mind that all of your actions will be repeated later, so for example, don’t make comments on a web page that only allows a max of 10 comments, or else you’ll get a high percentage of errors when it fails every time it’s hit after the 10th.

When you’re done with your web session, stop Tsung with ‘/opt/bin/tsung stop_recorder’. If you look in your home directory, you should now have a .tsung directory with a log of your session, something like ~/.tsung/tsung_recorderYYYMMDD-HH:MM.xml. That file contains the log of all the activity you did while sending traffic through the Tsung proxy, and will be used to drive the simulated user sessions during the load testing.

With the session file in hand, we can create a very simplified file to drive Tsung’s testing. There are some examples that were installed for you at /opt/tsung/share/doc/tsung/examples, but even the simplest example was more complicated than what I ended up using for my testing.

The only thing you must change here is the path to the session file that you recorded earlier, and possibly the hostname — although in my case it appeared to be overridden by the host defined in the session file. Apart from that, you might want to configure some details of the load testing, and set a maxusers value that’s somewhat reasonable for the test you have in mind.

<?xml version="1.0"?>
  <!DOCTYPE tsung SYSTEM "/opt/tsung/share/tsung/tsung-1.0.dtd" [ <!ENTITY mysession1 SYSTEM "/root/.tsung/tsung_recorder.xml"> ] >
  
  <!-- set dumptraffic="true" to dump all received and sent packets --> 
  <!-- set loglevel="debug" for maximum verbosity -->
  <tsung loglevel="notice" dumptraffic="false" version="1.0"> 
    
    <!-- Client side setup -->
    <clients> 
      <!-- maxusers is the max number of simultaneous clients. Don't set it too high because you can run out of file descriptors. Remember that ssl_esock use 2 fds by connection. -->
      <client host="localhost" maxusers="100" use_controller_vm="true"></client>
    </clients>
    
    <!-- Server side setup -->
    <servers>
      <server host="example.com" port="80" type="tcp"/>
    </servers>
    
    <!-- several arrival phases can be set: for each phase, you can set the mean inter-arrival time between new clients and the phase duration -->
    <load>
      <arrivalphase phase="1" duration="10" unit="minute">
        <users interarrival="5" unit="second"></users>
      </arrivalphase>
    </load>
    
    <!-- Options. -->
    <options> <!-- Thinktime value overrides those set in the session nodes if override=true. -->
      <option name="thinktime" value="3" random="false" override="true"></option>
      
      <!-- HTTP parameters -->
      <option type="ts_http" name="user_agent">
        <user_agent probability="100">Mozilla/5.0 (Windows; U; Windows NT 5.2; fr-FR; rv:1.7.8) Gecko/20050511
          Firefox/1.0.4
        </user_agent>
      </option>
    </options>
    <!-- ********************************************** --> <!-- start your recorded session --> <!-- ********************************************** -->
    <sessions> &mysession1; </sessions>
  </tsung>

In this case, we’re using clients on the local machine only, with a new user arriving every 5 seconds for 10 minutes, and with each user pausing for 3 seconds between requests to simulate thinking time. All clients claim to be Firefox, but that didn’t really matter to me — you may want to change it depending on how much browser-specific code you have.

When you have your tsung.xml file the way you like it, start Tsung with /opt/tsung/bin/tsung start. It will start and run until the last user session has completed (so a test defined to run for 10 minutes might take much longer than 10 minutes to complete if the server was heavily loaded). When the test is completed, Tsung will exit and write out a log file in ~/.tsung/log with the timestamp included in the filename.

Tsung has a tool included to generate HTML formatted reports that summarize the test, but they have to be generated by running /opt/tsung/lib/tsung/bin/lib_stats.pl in that test run’s directory. It will generate several html files and charts in png and ps format. Open up report.html or graphs.html in a browser to view an array of data from the test run. The Tsung home page has some samples of what the report will look like.

One last thing — by default, the charts are pretty low-res PNG files, but if you want to, you can create higher-resolution images from the PS (PostScript) files that each chart links to using a tool like GSView.

As resources, both the Tsung documentation and this post by 21 croissants were very useful to me as I was getting started.

So, if you were going to do some load-testing on your Rails app, you might think, hey, Siege is pretty cool. It supports load-testing multiple URLs at once (either sequential or in random order), with a delay in-between, with lots of options for setting the duration and characteristics of the load. Plus, there’s a tool that works with Siege, Sproxy, that lets you generate that list of URLs automatically by using Sproxy as a web proxy, recording your actions on the site.

Siege doesn’t support sessions (of course, that’s not clear from the docs, but I assure you it’s the case), which at first might be a deal-breaker if your app handles security by authenticating the session… but then you remember that you can pass in the _session_id parameter, and think all will be well.

Complication #1: In recent Rails versions, the session id can only be pulled from the cookie. However, we remember that you can set

ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS[:cookie_only] = false

And allow the session id to be taken from the parameter as well. However, if you’re running Rails 2, this won’t work. See Complication #2.

Complication #2: Thanks to a fix intended to prevent session fixation attacks, but which had some unintended consequences, the ability to override the cookie_only setting is broken. There is a monkey patch attached as a comment to that bug, but that’s not something I want to do when load-testing code. I do feel bad for the people out there (mainly mobile browsers and other non-traditional clients) that depend on parameter-passed session IDs.

So, time to look for a new tool, initial candidates are httperf/autobench, Flood, ab, and Tsung.

Firefly - The Complete Series — $17 for the complete set of one of the best SF series ever on TV.

You now have no excuse to:

• Not own Firefly
• Not have given Firefly to a friend
• Not have given Firefly to complete stranger

That is all.

I used to think that “ostrich with its head in the sand” was just a colorful description of the Bush administration’s response to environmental concerns, but now I see that it’s actual policy.

That’s right: Think you’re going to get an EPA report that you don’t want to deal with? Just order your staffers not to open the email!

Next step: Not opening science textbooks…

Photo courtesy fiver13 on Flickr

Steve Yegge (aka stevey) has a worth-your-while post (and that’s saying something when it’s that long) about dynamic languages in general and Rhino (server-side JavaScript in the JVM) in particular — along with about 10 or 15 tangents along the way.

InfoQ has a great video interview with Randy Shoup, in charge of search at Ebay, where he talks about several things, including their heavy use of partitioning, the unique issues you run into with such a high volume site, and what’s coming up for them.

Btw, I love InfoQ’s interface for presenting not only the video of the interview, but the transcribed and video-linked text of the interview as well. Great for skipping ahead or when the audio quality drops out and you can’t make out a word.

“I was shot 14 times on my way to work today, including twice by police,” said one Algonquin-area resident. “That is unacceptable.” –from the Onion

The Gibberlings3 modding community has a Widescreen Mod to allow you to play Infinity Engine games (like one of my all-time favorites, Planescape: Torment) in any resolution that your monitor can support instead of the original 640×480. Combined with the community-supported FixPack, I’m looking forward to taking another run through the world of Planescape!