Mistaeks I Hav Made
Good judgement is the result of experience ... Experience is the result of bad judgement.
—
Good judgement is the result of experience ... Experience is the result of bad judgement.
—
As I mentioned before, I've been writing Protest with Phil Dawes. Protest is a Python test framework and toolset for writing programmer tests and generating useful stuff from those tests. Currently the most interesting tool is the API documentation generator.
Writing API documentation by hand is a chore that offends my hard-learned distaste for duplication. When I've written tests that specify the API's behaviour I don't think I should to repeat that information in documentation. Even worse, the documentation cannot tell me when it's incorrect.
On the other hand, I really don't like Javadoc and similar tools for Python, such as Epydoc. Putting documentation into the code being documented ends up a maintenance nightmare. The code becomes hard to read and hard to navigate, lost in a mass of comments or docstrings.
So, Protest takes the same approach as Testdox and generates documentation from programmer tests that follow some straightforward conventions. I've tried to make it generate higher quality documentation than Testdox, which only creates a rudimentary overview of a test suite. The generated documentation has rich navigation links and a graphical visualisation of the module structure of the documented code. Having seen the importance of concrete examples in the Scrapheap Challenge workshop, the documentation includes the test code to show how to use every feature of the documented classes. Explanatory text can be to any example by giving the test a documentation string in reStructuredText format, keeping long comments out of the production code and in the tests where they are more useful.
A surprising side effect of the tool is how it really helps you improve your test code. I like to think I'm pretty good at writing tests. I designed the documentation generator to work with the way I like to make my tests self-explanatory. However, when I saw my test code in the documentation, rather than the IDE, I found lots of room for improvement: the test names needed tweaking to generate better titles, variables needed better names to identify the roles they played in the test, test data needed to be made self-describing, and so forth. I was pleased to find that the tool did more than just generate documentation, it gave me a fresh perspective on my test code that highlighted where it needed improvement.
I recently wrote a simple test framework to use on a project that is using Python on Series 60 phones. I began the project using PyUnit but found the API unintuitive. I wanted something more "pythonic", so I rolled my own.
I was initially puzzled about how to bootstrap the development of the test framework. How could I write the test framework itself test-first? Then I came up with a cunning plan: because I was also writing my project test-first, I could use the project as the test for the test framework that was being used to test the project. A simple, straightforward solution to my dilemma, as I'm sure you'll agree. Maybe.
The way I figured it, I'd write a test in my not-yet-existing test framework that would test my not-yet-existing project. When I first ran the test, it would not report that my project did not behave as expected. Intead it would report that the test framework did not behave as expected. So, I would keep writing the test framework and running the tests until it reported the expected failure of my project. I could then make the tests pass by writing project code, and start all over again until I had a working project tested by a working test framework. Pretty soon the test framework would be complete and I could focus just on the project.
It worked like a charm. I soon had a simple, flexible test framework and well tested project code. But of course, I was missing tests for the test framework itself. As I added features to the framework and built some useful development tools around it (more about those later) other developers started to get interested in the framework and wanted to use it on their projects too. This meant I had to extend the framework with features that my project didn't need and so I could no longer test the test framework against the project and vice versa.
I had to bite the bullet and write tests for the test framework. I hate testing after the fact — it's boring and error prone, but what else could I do? I should have written good tests in the first place instead of being too clever for my own good.