A theory about testable web UI: Part I

[thespeedbump]

One of the things that I spend an awful lot of time thinking about at IMVU is testability. I am always interested in finding ways to write tests that are faster, simpler, and more reliable. Once your tests reach a certain size, the return on investment you get by improving the performance of your tests increases linearly, until it's just too valuable to pass up. You can push the problem off somewhat by investing in hardware, and you can save quite a lot of engineer-muscle by doing this, but there's eventually going to be a point where you just can't resist looking for some way to shave just a few percentage points off your top-line test time.

At IMVU, our tests fall into a few basic categories. Some tests are pure PHP logic. Some are PHP logic that also interacts with MySQL, Solr, or a number of other external services. Others still use Selenium to drive an actual instance of Internet Explorer. (we also have a whole slew of tests for our 3D client, but they run on a separate buildslave cluster)

One idea that I find particularly intriguing is sacrificing acceptance tests in favour of unit tests. Unit tests are small, reliable, fast, and, while they don't have the same kind of guaranteed real-world applicability as a full-system acceptance test, and you probably don't want to trust them to catch unexpected regressions, it turns out that you can get quite a lot of mileage out of a sufficiently large swarm of them. Real unit tests can also be run one per thread on a dualcore machine for twice the effective performance. (and really, who even has a single-core CPU these days?)

The set of constraints placed upon frontend web tests is formidable:

• The tests have to be fast
• The tests have to be race-free on a platform that is built entirely on stateless, asynchronous I/O
• The tests have to run on whatever browsers your customers use, because every browser has subtle quirks that will randomly break your code
• Most web developers aren't used to the idea of letting tests dictate the structure of their markup or scripts

Some of these issues are engineering culture problems. Some of these requirements are directly in conflict with each other, while others are, think, poorly understood and categorically dismissed as intractable problems that must be endured.

Just to pick one example, Internet Explorer 7 may be the world's most popular browser, but it also has one of the slowest current-generation JavaScript engines. In particular, there are certain Selenium operations that run hundreds of times slower in IE than they do in Firefox. On the other hand, IE7 does certain things very differently from Firefox, and failing to take those differences into account can have catastrophic results.

So it seems that the choice is to sacrifice realism by testing on a faster browser, or to sacrifice speed to actually catch issues that will impact customers.

So, with all this in in mind, I believe that it is perfectly reasonable to structure your web frontend as two major classes of tests:

• Frontend unit tests that do not touch the network. These tests can run on a fast JavaScript engine (like Firefox, or maybe even Chrome).
• Full-blown acceptance tests that exercise everything (including the backend). Since these guys are the last line of defense, I'm pretty sure we really do have no choice but to run them on IE7.

I have found no shortage of literature and experience surrounding unit testing in general, and similarly, nobody is terribly confused about how to write effective acceptance tests. Within the specific field of test driving HTML interfaces, however, I have found precious little. So, in my next few posts, I am going to do a little bit of research, and share it all with the you, my adoring reader(s), as I go.

Next, I will talk about what I am going to try to build and my testing setup. (squeeeee project time!)

* I am endlessly entertained by my vivacious vocabulary. Also, alliteration is awesome!