164 Speciication by Example

In this chapter, I explain how teams from my research handled these three problems.

Reducing unreliability

An unreliable validation process can undermine a team’s conidence in a product and the process of Speciication by Example. Investigating intermittent failures that aren’t caused by real problems is a huge waste of time. If that happens often, developers will have an excuse not to look at validation problems at all. That will allow real issues to pass undetected, defeating the whole point of continuous validation.

Legacy projects rarely support automated functional testing easily, so executable speciications might need to be automated through unreliable user interfaces or suffer from nondeterminism caused by asynchronous processes. This is especially problematic when developers need convincing to participate in the process and see it only as an improvement on functional testing (in other words, not their problem).

Clare McLennan faced this issue with her team. “Developers didn’t care about tests because they weren’t stable, but we needed their knowledge to make them stable,” she said. This presented a chicken-and-egg problem for her team. To get the developers to participate, she had to show them the value of executable speciications. But to do that, the executable speciications had to be reliable, which required developers to change the system design and make it easier to plug in automated tests.

To get the long-term beneits of Speciication by Example, many teams had to invest signiicant effort into making their validation processes reliable. In this section I present some good ideas for that.

Find the most annoying thing, ix it, and repeat

When: Working on a system with bad automated test support

One of the most important things to understand about making the system more reliable for automated testing is that this won’t happen overnight. Legacy systems aren’t easy to change; otherwise, they wouldn’t be legacy. When something was built without a testable design for years, it won’t suddenly become clean and testable.

Introducing too many major changes quickly would destabilize the system, especially if we still don’t have good functional test coverage. It would also severely interrupt the low of development.

For example, McLennan’s team realized that slow test data processing was causing tests to time out. Their database administrator improved the database performance, which