January 3, 2009 by mensming.

With the new year, it is time for me to write my annual bug. Like many products, ours includes a copyright statement of the format "© 2006–2008 Company Name" that is displayed to our users. Every year, I submit a bug to change the ending year.

The United States Copyright Office provides the following guidelines in the copyright basics PDF.

Form of Notice for Visually Perceptible Copies

The notice for visually perceptible copies should contain all the following three elements:

1. The symbol © (the letter C in a circle), or the word “Copyright,” or the abbreviation “Copr.”; and
2. The year of first publication of the work. In the case of compilations or derivative works incorporating previously published material, the year date of first publication of he compilation or derivative work is sufficient. The year date may be omitted where a pictorial, graphic, or sculp­tural work, with accompanying textual matter, if any, is reproduced in or on greeting cards, postcards, stationery, jewelry, dolls, toys, or any useful article; and
3. The name of the owner of copyright in the work, or an abbreviation by which the name can be recognized, or a generally known alternative designation of the owner.

By these guidelines, the second date is not required. However, sometimes it is better to go with the flow.

Posted in system testing, web testing | No Comments »

December 27, 2008 by mensming.

Over several months, Michael Hunter, in his blog on Dr. Dobb’s Portal, ran a series of posts with the title of "You Are Not Done Yet", each of which picked an area of a software deliverable and listed items that a tester needs to consider before considering themselves done testing it. This is a great series and I have referred numerous individuals to it time and again.

I recently wanted to refer someone to it and it took me a while to find the particular URL I needed. So, in order to save myself some time, here is the link to the table of contents to all of the articles in the series.

Posted in system testing, web testing | No Comments »

December 14, 2008 by mensming.

While it has been a while since I needed to do installer testing, I recently researched a product that may be helpul in this task. Sandboxie strives to isolate programs from making changes to the rest of the operating system. The Sandboxie website describes it as:

Sandboxie runs your programs in an isolated space which prevents them from making permanent changes to other programs and data in your computer.

and lists as a benefit:

Windows Stays Lean: Prevent wear-and-tear in Windows by installing software into an isolated sandbox.

Sandboxie adds a third approach to installer testing. The first approach, to install over and over on the same instance, is expediant but not necessarily the most realistic. The second approach is to re-image / restore the system. When I first started, this meant using a program such as Ghost, now owned by Symantec. This approach has been supplanted using virtual machines. Virtual machines have made this process much faster.

Using Sandboxie, it is possible to have the cleanliness afforded by virtual machine or images plus the expediency of installing over and over on the same system. On a single system, multiple sandboxes could be created - one for each install test. This approach seems reasonable to me. It is still necessary to do installer testing on native system, but this can be done after many bugs our found and resolved.

Pricing is very reasonable for a commercial license. Check it out.

Posted in test tools, system testing | No Comments »

April 30, 2008 by mensming.

A lot of developers I have worked with have been amazed to see that I expand a suite of tests they developed by a factor of four. This is almost always because I am stressing error handling which is often not tested at all. Unfortunately, these tests will often cause many warnings. Sometimes I want to verify the exact warning. Other times it just clutters the output being sent to the screen. This can be very dangerous - it may lead someone running the test to believe there is a newly introduced bug.

Luckily, perl has a well defined mechanism for overriding the warn behavior. If you want to suppress warnings from being displayed in a section of tests do the following:

Once the scoping ends, whatever warning handler was in effect before will be active again.

Posted in test tools, system testing, web testing | 1 Comment »

March 10, 2008 by mensming.

Scott Hanselman, on his Hanselminutes podcast interviewed Quetzal Bradley of Microsoft in show 103. I enjoyed the discussion because it covers two topics I find myself explaining to less technical managers time and time again:

• Why test coverage is a necessary but insufficient measure of when testing is done
• Why 100% automation is not necessarily a good thing

It is a quick listen - recommended.

Posted in test tools, system testing | No Comments »

February 18, 2008 by mensming.

I recently ran into a couple of bugs that we should have found in our development, test and staging environments. However, they slipped through into our production environment. These were not major issues, just a couple of helper utilities. We thought they were working fine. When we tested the utilities they behaved correctly.

This was just luck of the draw. We had done our testing with clean databases — databases in which the various primary key id all started at an auto increment value of 1. Since, all of the inserted data started at the same id value, the functionality accidentally worked.

We considered a couple of approaches to prevent this from happening again. We already have a copy of our production data that we use for some testing. This is not always feasible since the amount of production data results in a substantial setup time. Most of the time, we will use the same scripts we have always used to create a clean database. However, we added an additional step to the scripts where they set the auto increment starting values for each table at 10,000 value increments from each other. This will prevent us from “accidentally” thinking our application is working.

Posted in system testing | No Comments »

July 9, 2007 by mensming.

This week I received an email from a graduate student in Sweden. He was asking about the abstraction level of test cases. My response to the email is below:

A couple of comments on the decomposition of test cases. When I submitted the paper, the conference committee was very divided on the paper. Half of the reviewing committee was saying that the topic was obvious and not worth including in the conference program. The other half of the reviewers commented that this was a worthwhile topic because there was (and still is) very little written on the topic. When I first started out in the software testing field, this is what I discovered — there was much written on test case design but very little on how to organize the test cases.

The situation today is different but still there is very little written on organizing test cases. Some of the best work I think that has been done organizing testing is the work by James and Jon Bach on exploratory testing / session based testing. Jon has a Google TechTalk video. There is a Wikipedia article on session based testing that might give you some pointers to more information.

In general, the move towards agile and test-driven development has led to less documentation. However, in a lot of cases, I think this has led to less planning of testing. (Perhaps with better quality planning on some development projects.) Test driven development has focused more on organizing test cases around the structure of the source code — which evades the question of the best way to decompose the testing.

With respect to your second question about executing “sub test cases” and the percent that they cover the “parent test case”, I have not given much thought to this topic. I generally look at eliminating redundancy - so if I can eliminate any test case I generally will do so. However, the question I think you are asking points to the larger question of how deeply do you test a particular feature before moving on to some other portion of the project. For example, say the “parent test case” is test feature A - how thoroughly do you test feature A before you say you are done. Obviously, it depends on the importance of the feature, the risk involved if the feature, etc.

I am not sure if this is helpful. Best of luck on your thesis and continued studies.

Posted in system testing, web testing | No Comments »

March 19, 2007 by mensming.

Stories like this are examples of what gives quality assurance people a bad name…

Shark Tank: Well, it didn’t abort! - Computerworld Blogs

“The goal was for no errors in the execution or interruption to the program series flow. No consideration was given to what the program actually did!”

Posted in system testing | No Comments »

March 13, 2007 by mensming.

As I have mentioned before, I recently read The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt and David Thomas. One of the tips is “Use Saboteurs to Test Your Testing”. This reminded me of my first encounter with error seeding.

While working on my Masters, I took a software testing course. I had been working on test teams for several years so I felt confident coming into the course. We were asked to take a program we had written in a prior course to use as our case study. We were asked to create a suite of tests for our program. I did so and discovered several bugs I had not identified when I turned in the program. Overall, I felt very confident in the test suite I had created. Throughout the semester, we would apply the various concepts we were learning against the program we had selected.

When the concept of error seeding was introduced, the teaching assistant (TA) for the course took our programs and introduced some errors into them. She went through quite a bit of work. Some errors she introduced were simple to create - for example, changing <= to <. Others were more subtle, such as introducing a side effect into an existing method that had no side effects before. We were given new binaries and told to run our tests and submit the results. (We were not told how many errors were introduced into the program.) I ran my suite of tests -- all automated by now given the fact that this was an ongoing project. Sure enough, my tests discovered 3 changes in behavior. I felt good about the results. Then we were told that each program had 10 errors in them. I was humbled - and learned a lot by examining what my suite had not found.

Posted in system testing, web testing | No Comments »

March 8, 2007 by mensming.

As I have mentioned before, I recently read The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt and David Thomas. In the section on “Programming by Coincidence” I was reminded of a bug I found, it was fixed and it broke a customer’s implementation.

The product was a set of tools, services and application servers that a client could use to automate some processes in a particular vertical industry. As part of the product, there was a full programming language. I came on board to build the test team. As we moved forward with new functionality, we slowly worked through the functionality which had never been formally tested before. When we started testing the date arithmetic functionality, we found quite a few errors.

One of these errors was the “add” function would accept blatantly invalid dates. When this error was fixed, we verified and moved on. It ends up that the biggest customer used the “accident of implementation” of the add function to determine whether the current year was a leap year or not. They would add the date “February 30″ for the current year with the value 0. The system before the fix would return the last day of February (28 or 29 depending if it was a leap year or not) for the year in question. After the fix, the system would return an error. (Sometime before this bug was fixed, an isLeapYear function had been added to the system but that was after the customer implemented this code.) The customer’s developers had used this accident of implementation for some crucial logic in their application. This behavior was not documented - someone had “discovered” it.

Posted in system testing | No Comments »