For the past 15 months my Test Team has been building up a set of Test Cases and Test Procedures for a software project following strict MIL-STD-498 standards. It’s a military project, I could tell you more about the design but I would have to kill you afterwards. However, I can continue to talk about my experiences within the last 15 months.

Today is quite a day, it is the last day before we enter Formal Acceptance Testing. We await officials from another country to arrive on Monday who will then inspect my Test Team while we run through our 15 months worth of Test Procedures. Earlier this week we got the formal approval of all our tests, which resulted in a small pat on our own backs as it marked the end of our long work designing the tests. Monday morning, we run the tests for real….

27 Test Case documents, containing between 2 and 40 Test Procedures each, each procedure having 2 to 200 steps each…. Estimated 120 man hours to run.

The interesting bit here is that it is the first time we, as a software team in whole, have worked with MIL-STD-498. As the Software Test Manager for the project I was required to get involved as soon as the formal requirements were approved. Before any code was written, it was required of us to produce a Verification Cross Reference Matrix - for each requirement we needed to describe how we would go about verifying it, not only what we would do but how would we do it. Would we need to simply analyze a design, demonstrate a function, or run a full blown test. This resulted in a possible test before the code was written, a little bit of Extreme Programming going on - the old ideas are always the best.

The task of designing the tests started as soon as we had a stable semi-approved design. We knew that it would be months (infact over a year in the end) before we would ever see the software we would be testing. Quite a task then as previous experience has shown the Software Design Descriptions tend to change and/or do not contain enough information. MIL-STD-498 with its strict procedures to follow, should overcome this - and over the last year we have put it to the test….

It took a long time to get a complete set of Test Case documents, and a couple of months ago we started our Test Dry Run (TDR) phase, the first time our tests would face the real software. Quite a moment, would our tests relate to the software in any way or would it be way out? Written from the design of both the software and the numerous Test Harnesses written for the purpose of both development and testing, it was an interesting moment. The result? I would say a good 80% of test steps were correct, maybe one or two Test Cases that needed a new approach due to misunderstanding, but nothing that 8-16 man hours of low risk edits would not solve.

That’s the interesting bit. We all know how software projects should be run, and we all know that in a commercial market sometimes bits get skipped. MIL-STD-498 does not allow anything to be skipped. Test steps document right down to the level of “press this button, press that button”, give a thick pile of Test Case print outs to someone on the street and over three weeks they would be able to rerun them and fully verify the project for the customer. MIL-STD-498 requires this, and it could possibly bankrupt a small company if used against any old project as the level of detail required is just way over the top for most. However, lessons can be learnt from MIL-STD-498 and testing procedures adopted that don’t fully strictly to the standard but do go through similar steps.

The proof of it working or not is our 80% rate of correct Test Cases when we ran them for the first time. A year’s worth of test design against a Design Specification, and the majority of this ran first tim with no problems when Tests met Code a year later. That is the special bit, given the level that the test steps go into too. After managing software testing teams for the past 10 years, I have quite enjoyed all this, and taken a lot from it too.

So now, we relax over the weekend, we start Formal Acceptence Testing next week, for three weeks….

Related Post