Looking for flaws

Or checking for correctness? When you test your code, do you look for all the wrong things that can happen, or do you make sure it’s working properly first?

The weak link by James Steidl @ iStockphoto

Of course, you should be doing both. But you’ve got to do one or the other first. You can’t do both at the same time, because they’re completely different. One is where you don’t know what will happen, like rolling your head on the keyboard to come up with random input. One is where you know exactly what has to happen, like a textbox accepting numeric input, and only between 10 and 23.

Correctness is defined here as “working according to what’s reasonably expected”. If you have specs, there you have it. Negatives can count as correct too. For example, “must not show information of other users” is a negative requirement. If the application indeed does not show other users’ information, then it’s working correctly according to the requirement.

The fastest way to complete a project is to code according to specs, test for correctness, then test for flaws. Because the specifications are known, there are thus well defined finite areas of correctness, which provide a basis for deadline estimation.

Flaws on the other hand, are practically infinite. You can always find something to change for the better. When you test for flaws first, you will find tons of minor irritating aspects of the application that you can change. None of which will significantly bring you any closer to what’s required.

You’ll only stop because the deadline has mysteriously gotten closer, so you decide it’s better to stop nitpicking and actually start testing for the requirements.

Let’s take the example given before, a textbox accepting numeric input, and only between 10 and 23. Reading through the specs, you find out that it’s for hours of operation, as in anything between 10 am and 11 pm. So the value obtained from the textbox is used as an integer.

So you test by punching in numbers and making sure only numbers 10 through to 23 are accepted. Numbers outside the range as input will give an error. You also make sure alphabets and symbols give errors too. This satisfies the correctness requirement.

Then you test for flaws. Things like 12.0 or +19 are accepted, but preferably not.

Alright, as a contrived example, the textbox input requirement seems rather lame. But I hope you got the point. Testing for correctness means you get a working application quickly. Then when you test for flaws, you’re just adding on to the correctness.

The thing is, some people must overcome whatever their misgivings about your application before they even check how cool your application is and how correct it is. These people are usually your users, or your managers, or whoever uses the application but don’t really give a rodent’s behind about your code.

I’ve met some of these people. And I understand their position and view point. They don’t quite like this colour, or they want that button somewhere else. I get it. There’s a reason why I understand this behaviour, and that I even expect it of them.

They already assume your application is working correctly. Hence they go nitpicking. They have a right to nitpick. In their minds, the application is working according to specifications, and they’re tuning it to better suit them.

Now, the authors of defect reports… I don’t understand them. Dedicated testers are supposed to give feedback on the application, which includes both correctness and flaws. And correctness should be a higher priority. They should be the safety net, catching the uncommon cases that escaped the programmer, but contribute to the correctness of the application.

Yet the testers I know seemingly focus exclusively on the flaws with zero regards to the correctness. Once, my colleague had to come up with a suitable image for a trashcan, for use as a delete button in a datagrid. The original image was done by me, bluish in theme. My colleague even apologised to me that he’s not using my image. It was fine. The image wasn’t artist-grade, but it serves its purpose.

He had to do it, or the testers refuse to test the intended web form at all. They don’t want to test if a new record can be correctly inserted into the database. They don’t want to test if they can update existing records correctly. They don’t want to do anything on the web form, refusing to understand how it works, until they’ve got their trashcan image.

One image after another was sent. They don’t like the look. They don’t like the colour. They don’t think it looks like a trashcan. Finally I suggested to my colleague, just ask them for an image instead.

They couldn’t. They don’t like the images submitted to them, and they couldn’t give us an image they approve of. After wasting many days over this trivial matter, my colleague came up with an image, digitally manipulated to fit what hopefully suits the testers’ perceived requirements.

It was ugly. It was dull red, like someone had a nosebleed and sneezed, and one of the blood drops happened to hit the floor in a vague squarish pattern. Then a digital photograph was taken of it, and the result had a slight yellowish-green cast to it. And because of the looming deadline, the testers finally went with it.

I am working hard with the users to bring the application to live status as soon as possible. The sooner it goes live, the sooner users get to benefit. The testers, on the other hand, seem bent on delaying the project. Their fears of causing calculation errors, of users using their approved interface design (but might possibly be perceived as unfriendly), of maybe tens of other irrational fears have stopped them from growing. And their fears have essentially stopped the company from growing.

It’s about mindset. Do you expect greatness and code and test for correctness? Or do you fear and nitpick on minutiae?

We should call them improvement reports

My career as a software developer has always been fraught with danger. I’ve been involved in several software projects now, and they all come with their challenges. The development cycle would involve the gathering of user requirement, planning, development and testing. The testing here refers to my testing. Where I work now, they have a whole department just to test the resulting applications of developers.

The gathering of user requirement is fairly simple. A couple of meetings, a few phone calls and some emails usually do the trick. Smiling helps. The simplicity of planning will vary, depending on deadlines (users will always say “as soon as possible”. Smile some more.), complexity of tasks involved and available manpower. For me, development is usually a breeze. I’ve shook my hand in the air (and mentally swore words that cannot be repeated here) whenever I do work with PowerBuilder, but that’s a different story. The internal tests are also easy. Enter a few values, make sure they are calculated and stored in the database correctly, and check that the reports are designed according to the user’s requirement.

The tests done by the dedicated testing department are, well, nerve-wrecking. I’ve had to duplicate the production environment onto the test environment. I’ve had to upload data, download data, change data and massage data from text to Excel and vice versa. I’ve got to refer to the finalised user requirement document to confirm (and defend) the veracity of my code.

There’ll be phone calls and emails to and fro, to confirm this, to help set up that, to ask why that number is supposed to be that number. I’ll probably be working on another project while smiling on the phone and scribbling notes on my already messy desk.

And I haven’t even gotten to the fun part.

At the end of the test period, I will be sent *drum roll*, defect reports. These evil documents contain what the testers had deemed unforgivable deviations from the righteousness of the user requirement document. They would identify the defect in my program, point their righteous fingers at me, and cackle “You made mistake. *dance around* Your program doesn’t work. *dance around some more* It’s all your fault”.

The thing about defect reports is that, well, they contain defects. When I pointed out that testers only focus on generating defect reports, I was faced with blank stares, with the implicit “Of course they look out for defects. That’s what they are supposed to do!”.

What you focus on, expands.

When the appraisal of a tester is determined in part by the number of defect reports the tester produces, what happens? That tester will produce tons of defect reports, on whatever bug or however slight the program misbehaviour was. This skews the perception of the accuracy of the program. Then when the defect report came to me, I would do whatever it took to downplay the report, to deny the existence of the purported defect, uh I mean inconsequential oversight, where I try to skew the perception back into balance.

This sparring of responsibility is damaging to the project, unhelpful to the user, and definitely a waste of the company’s resources. “Are you telling me you spent two hours typing that report about why there should be an ‘s’ for plural, when you could have been giving suggestions on how to improve the user experience?”. “Are you telling me you spent two hours going through every display message in the program code to make sure the ‘s’ was appended correctly, when you could have been implementing better features?”

Instead of keeping track of defects, we should be focusing on improvements. Yes, there are bugs. And yes, I will eliminate them. I also appreciate being told about them. I’m just a programmer. If you are a tester, you’ll be closer to being a user than I’ll ever be. Other than complaining about how lousy my program is, can you add anything to improve the user experience?