Playing in sandboxes can be a lot of fun. I just spent a little while playing in a software sandbox I set up last fall, and can only wish I had gotten around to setting it up a lot earlier.

Last October, for reasons I need not go into, I conceived a strong desire to generate a large-ish number of test cases for particular areas of XML Schema, for use in thinking about the current state of implementation of XSDL 1.0, and in thinking about what ought to happen in XSDL 1.1. So I spent some time wrapping my head around the relatively elaborate test suite framework the XML Schema Working Group adopted some years ago for our test suite. (I had looked at it when we adopted it, of course, but you look at a vocabulary in a different way when you are going to be generating data using it.) Eventually, my plan was (and still is) to generate test cases automatically from Alloy models, but as a first step I mocked some test cases up by hand.

But to make sure the test cases were actually testing what I wanted to test, I needed to run them, systematically, on different processors. So I spent an afternoon or two (or three) installing every schema processor I could conveniently get my hands on and persuade to run under MaC OS X, or under Windows XP (since I started using this Mac I have not had any machines running Linux [I say it with a bit of a sigh]), and writing some scripts to wrap around them so I don’t have to remember what order they want their command-line arguments in, or what options they want.

It’s always interesting to construct test cases to illustrate some question or other that arises, whether from a comment on the spec or an inquiry by email. And I have directories with scores of small test cases I have constructed over the last few years. But until I started this more systematic construction of a schema validation sandbox, I contented myself with checking those test cases with one or two processors.

But it turns out that having more processors to test is just a lot more fun. Here is a test case. OK, what does libxml say about it? MSV? Saxon? Xerces C? Xerces J? (And if I’m really energetic, move to the other machine and check MSXML 4, MSXML 6, and xsv. I ought to get a current version of XML Spy installed, too, but I haven’t been that energetic yet.) For substantially the same effort, to get five times as much information, just makes the whole thing more fun. (And the more fun we can make it to construct and run test cases, the better.)

Today’s effort was an attempt to answer a question raised by Xan Gregg in a comment on an open issue against XSDL 1.1 Structures. Two schema documents (one pretty much as Xan provided it, one modified to correct a possible oversight), and five instances, provide a simple test of how implementations have interpreted a rule in XSDL which Xan and I turn out to have read differently. (The test cases, and a catalog for this tiny collection of tests, are all on the W3C server at http://www.w3.org/XML/2008/xsdl-exx/. I plan to put every schema example I generate this year there, instead of hiding it on my hard disk. At least the interesting ones.) The process of making them and testing them was delayed for a bit of yak-shaving (quick, how do you embed selected XHTML modules into your DTD, in a way that you are willing to let other people see in public?), but I got them made eventually, and demonstrated to my own satisfaction that virtually all the implementors have agreed on the meaning of this bit of the spec. (Fortunately for me, they all read it the way I read it. But Xan is right that it could be taken in a different way; the wording should be changed to make it clearer.)

Maintaining a software sandbox with installed copies of all the software one wants to play with can be time consuming. And since in the usual case, you aren’t familiar with the software yet, you may not be able to make a strong case for making it an urgent task. Uncertain cost, uncertain benefit, low priority. Other things always seem more urgent. But having such a sandbox, and playing in it from time to time, are important tasks, even if not often urgent. It’s nice when you get a chance to do it.

Happy New Year.