JTSR

If normal application code can fail and so requires testers, then what is stopping the code that testers develop to automate their testing from doing the same?  You cannot have a string of testers each testing each others code until you get to the end of the chain and finally to the development code!  It would certainly make a good comedy scetch, but laughter would be the only thing good you would ever get out of it.

It may be that the code written by the testers is simple and in easy to manage blocks, in which case seeing any errors would be a simple task and any errors may be quickly noticed, in theory.   It may however be that the code written by the testers is just as complex as the code being tested.  This will certainly be the case once you start to build up a large testing engine handling not only the full testing of the software but your own reporting and recording routines.    All practices of good software development will of course be used, so possibly an OO approach, small chunks of code to build up to a complete testing app.   With this, bugs and just things that don’t work as you would expect - what then to do?

I am a great fan of creating tests and automation way before the final build of developement software has been written.  Using specifications and requirements it should be enough to get a good understanding of what the sofware is going to do and so be able to not only write formal test cases but also create automation.  Certainly, these should be the same specs that the developers themselves will be using.

Create your tests and execute them from day one.   In true Extreme Programming style, see your tests fail in the early days due to the software under testing not working correctly or simply not yet being available.   Once you have seen your tests fail a number of times you understand more your test and the software under test.   By the time you get a formal test release of software you know how your test fails, now it is time to see it pass, and if it does then possibly bingo.  If it doesn’t, then it will either be your test code or the application under test that is at fault, but at this point you have indeed found a fault that must be fixed. If you were to run your automation against software not yet finished or not available and it passes… then either your tests are not complete or your test code is buggy.

It is then always  working with the “negative”, everything fails until something in the development code allows it to pass.   The number of test runs on pre-alpha releases allows you to “test” your own code before the formal release.     Any pass by that time should be a result of code testing code and complient software under test.

Related Post

• Testing Software to MIL-STD-498 standards

For the past 15 months my Test Team has been building up a set of Test Cases and Test Procedures for a software project following strict MIL-STD-498 standards. It’s a military project, I could tell you more about the design but I would have to kill you afterwards. However, I can continue to talk about my experiences within the last 15 months.

Today is quite a day, it is the last day before we enter Formal Acceptance Testing. We await officials from another country to arrive on Monday who will then inspect my Test Team while we run through our 15 months worth of Test Procedures. Earlier this week we got the formal approval of all our tests, which resulted in a small pat on our own backs as it marked the end of our long work designing the tests. Monday morning, we run the tests for real….

27 Test Case documents, containing between 2 and 40 Test Procedures each, each procedure having 2 to 200 steps each…. Estimated 120 man hours to run.

The interesting bit here is that it is the first time we, as a software team in whole, have worked with MIL-STD-498. As the Software Test Manager for the project I was required to get involved as soon as the formal requirements were approved. Before any code was written, it was required of us to produce a Verification Cross Reference Matrix - for each requirement we needed to describe how we would go about verifying it, not only what we would do but how would we do it. Would we need to simply analyze a design, demonstrate a function, or run a full blown test. This resulted in a possible test before the code was written, a little bit of Extreme Programming going on - the old ideas are always the best.

The task of designing the tests started as soon as we had a stable semi-approved design. We knew that it would be months (infact over a year in the end) before we would ever see the software we would be testing. Quite a task then as previous experience has shown the Software Design Descriptions tend to change and/or do not contain enough information. MIL-STD-498 with its strict procedures to follow, should overcome this - and over the last year we have put it to the test….

It took a long time to get a complete set of Test Case documents, and a couple of months ago we started our Test Dry Run (TDR) phase, the first time our tests would face the real software. Quite a moment, would our tests relate to the software in any way or would it be way out? Written from the design of both the software and the numerous Test Harnesses written for the purpose of both development and testing, it was an interesting moment. The result? I would say a good 80% of test steps were correct, maybe one or two Test Cases that needed a new approach due to misunderstanding, but nothing that 8-16 man hours of low risk edits would not solve.

That’s the interesting bit. We all know how software projects should be run, and we all know that in a commercial market sometimes bits get skipped. MIL-STD-498 does not allow anything to be skipped. Test steps document right down to the level of “press this button, press that button”, give a thick pile of Test Case print outs to someone on the street and over three weeks they would be able to rerun them and fully verify the project for the customer. MIL-STD-498 requires this, and it could possibly bankrupt a small company if used against any old project as the level of detail required is just way over the top for most. However, lessons can be learnt from MIL-STD-498 and testing procedures adopted that don’t fully strictly to the standard but do go through similar steps.

The proof of it working or not is our 80% rate of correct Test Cases when we ran them for the first time. A year’s worth of test design against a Design Specification, and the majority of this ran first tim with no problems when Tests met Code a year later. That is the special bit, given the level that the test steps go into too. After managing software testing teams for the past 10 years, I have quite enjoyed all this, and taken a lot from it too.

So now, we relax over the weekend, we start Formal Acceptence Testing next week, for three weeks….

Related Post

• Testing before you have the code