[Hidden-tech] Stress Testing Web Applications

[Mark Bucciarelli]

On Wed, Aug 30, 2006 at 11:11:11AM -0400, Tom Goldsmith wrote:

> Whatever tools you find, and however you
> generate load, be sure you also do some
> "collateral" testing with REAL USERS.

Amen.  Users enter the strangest data sometimes ... ;)

Siege was created in an attempt to partially address this issue, with
respect to variable sequence of URL requests.

    <quote ref="http://www.joedog.org/Siege/History">

    The number of simulated users exceeded the number of
    acceptance testers by many orders of magnitude, yet the
    acceptance testers were able to break the server, but the
    simulated users were not.  Torture was pounding the server
    one URL at a time, the acceptance testers were roaming
    through the content at will. That discrepancy, I concluded,
    explained how the code passed one test and failed in the
    other. I had to recreate the acceptance testing in a lab
    environment so that I could pinpoint the cause of failure.

    </quote>

> The trick is to get load up with whatever
> tools you can find.  Monitor your servers
> to see that the load is true and realistic,
> and then at the same time get in with a
> "human" user or two to see that you have
> full functionality when load is in place.
> Also, if any parts of your application do
> error logging, of course you should be
> prepared to look at those logs.  ...Real
> time with a "tail" function, if possible.

If you are on a shared resource, this will be tricky as your app's
performance will depend on what other apps are doing on the box.  This can
be mitigated somewhat by using virtual servers with resource limits.

Hopefully, you will have a colocated server so this won't be an issue.  If
you don't, get one.  A 1U slot should go for no more than $50 a month.
Matt (Croker)  or I (GAIA Host Collective) can provide this for you.
Sheldon (JagFly) or Rich (TnC, maybe?) may also offer this service.


> Be realistic about the amount of load you
> want to test.  Not an astronomical number
> of users at the start.  And see that a
> modest amount of load works well before
> you increase the intensity of your test.
> Do any really heavy, peak load testing,
> only after you've confirmed that your
> initial daily ambitions will be met with
> good service.

A few specific suggestions:

- learn how to tell the difference between a box constrained by CPU and one
  constrained by IO (If on a *nix box, learn to love vmstat.)  If CPU, is
  is system calls (probably your database) or user space?  If IO, is it
  disk reads or disk writes?  Are you starving the kernel of memory to
  hold things like network packets?  There are different fixes for each of
  these things.

- push it till you see at what point do you start swapping (available ram
  exhausted)

- watch for steadily increasing memory usage (a leak).

- load your database with a huge number of records.  script this so you can
  repeat exactly the same test after changes and monitor the difference.
  poorly written queries will work just fine with small datasets.  this won't
  bite you until your site is phenomenally successful (murphy's law).

- don't optimize what you can't measure.

m