June 4, 2009 by mensming.

A friend sent me a link to this graphic. Not sure who to attribute it to. The image is hosted on http://media.oldben.com.ar.

Posted in system testing, web testing | No Comments »

April 18, 2009 by mensming.

In a prior post, I described an example of a side effect of a feature that our users came to rely on as a feature in its own right.

While reading Raymond Chen’s The Old New Thing: Practical Development Throughout the Evolution of Windows, I came across what I consider the mother of such side effects. In an essay entitled “The hunt for a faster syscall trap”, Raymond describes how an Intel representative was perplexed when a Windows engineer requested speeding up the fault when an invalid instruction is executed. This would seem to imply that the Windows code was buggy in some sort of way. The story goes on to relate that executing an invalid instruction was intentional - as a way to get into the CPU kernel mode.

A version of the essay can be found online here.

Posted in software engineering, system testing | No Comments »

March 22, 2009 by mensming.

In show 146 of Hanselminutes, Scott interviews agile coach Scott Bellware.

It is an interesting conversation about whether test driven development is a misnomer, the misuse of the word testability, test smells, etc. A valuable conversation for testers to consider when discussing ways to improve the quality of a product. (Especially if someone asserts that additional testing is not required since "we use test driven development".)

Posted in software engineering, process improvement, system testing, web testing | No Comments »

February 18, 2009 by mensming.

On the last day of 2008 - a leap year - some models of the Zune were reported as not working. When the date rolled over to January 1, 2009, all was fine. Even though I was not impacted by this bug, I can relate.

At my second job out of college, our product was hit by a similar bug. The product used both Gregorian and Julian calendars (I do not recall why…). We had done all of the normal leap year tests around February 29 for the year in question. On New Years Eve and New Years Day, our support lines lit up. Some month end, quarter end and year end processes were failing. Sure enough, they did not expect 366 days in this year.

Ever since that time, I always wanted to have a set of systems that were running some number of months in the future so I could discover these misses before our customers did.

Posted in system testing | No Comments »

January 3, 2009 by mensming.

With the new year, it is time for me to write my annual bug. Like many products, ours includes a copyright statement of the format "© 2006–2008 Company Name" that is displayed to our users. Every year, I submit a bug to change the ending year.

The United States Copyright Office provides the following guidelines in the copyright basics PDF.

Form of Notice for Visually Perceptible Copies

The notice for visually perceptible copies should contain all the following three elements:

1. The symbol © (the letter C in a circle), or the word “Copyright,” or the abbreviation “Copr.”; and
2. The year of first publication of the work. In the case of compilations or derivative works incorporating previously published material, the year date of first publication of he compilation or derivative work is sufficient. The year date may be omitted where a pictorial, graphic, or sculp­tural work, with accompanying textual matter, if any, is reproduced in or on greeting cards, postcards, stationery, jewelry, dolls, toys, or any useful article; and
3. The name of the owner of copyright in the work, or an abbreviation by which the name can be recognized, or a generally known alternative designation of the owner.

By these guidelines, the second date is not required. However, sometimes it is better to go with the flow.

Posted in system testing, web testing | No Comments »

December 27, 2008 by mensming.

Over several months, Michael Hunter, in his blog on Dr. Dobb’s Portal, ran a series of posts with the title of "You Are Not Done Yet", each of which picked an area of a software deliverable and listed items that a tester needs to consider before considering themselves done testing it. This is a great series and I have referred numerous individuals to it time and again.

I recently wanted to refer someone to it and it took me a while to find the particular URL I needed. So, in order to save myself some time, here is the link to the table of contents to all of the articles in the series.

Posted in system testing, web testing | No Comments »

December 14, 2008 by mensming.

While it has been a while since I needed to do installer testing, I recently researched a product that may be helpul in this task. Sandboxie strives to isolate programs from making changes to the rest of the operating system. The Sandboxie website describes it as:

Sandboxie runs your programs in an isolated space which prevents them from making permanent changes to other programs and data in your computer.

and lists as a benefit:

Windows Stays Lean: Prevent wear-and-tear in Windows by installing software into an isolated sandbox.

Sandboxie adds a third approach to installer testing. The first approach, to install over and over on the same instance, is expediant but not necessarily the most realistic. The second approach is to re-image / restore the system. When I first started, this meant using a program such as Ghost, now owned by Symantec. This approach has been supplanted using virtual machines. Virtual machines have made this process much faster.

Using Sandboxie, it is possible to have the cleanliness afforded by virtual machine or images plus the expediency of installing over and over on the same system. On a single system, multiple sandboxes could be created - one for each install test. This approach seems reasonable to me. It is still necessary to do installer testing on native system, but this can be done after many bugs our found and resolved.

Pricing is very reasonable for a commercial license. Check it out.

Posted in test tools, system testing | No Comments »

April 30, 2008 by mensming.

A lot of developers I have worked with have been amazed to see that I expand a suite of tests they developed by a factor of four. This is almost always because I am stressing error handling which is often not tested at all. Unfortunately, these tests will often cause many warnings. Sometimes I want to verify the exact warning. Other times it just clutters the output being sent to the screen. This can be very dangerous - it may lead someone running the test to believe there is a newly introduced bug.

Luckily, perl has a well defined mechanism for overriding the warn behavior. If you want to suppress warnings from being displayed in a section of tests do the following:

Once the scoping ends, whatever warning handler was in effect before will be active again.

Posted in test tools, system testing, web testing | 1 Comment »

March 10, 2008 by mensming.

Scott Hanselman, on his Hanselminutes podcast interviewed Quetzal Bradley of Microsoft in show 103. I enjoyed the discussion because it covers two topics I find myself explaining to less technical managers time and time again:

• Why test coverage is a necessary but insufficient measure of when testing is done
• Why 100% automation is not necessarily a good thing

It is a quick listen - recommended.

Posted in test tools, system testing | No Comments »

February 18, 2008 by mensming.

I recently ran into a couple of bugs that we should have found in our development, test and staging environments. However, they slipped through into our production environment. These were not major issues, just a couple of helper utilities. We thought they were working fine. When we tested the utilities they behaved correctly.

This was just luck of the draw. We had done our testing with clean databases — databases in which the various primary key id all started at an auto increment value of 1. Since, all of the inserted data started at the same id value, the functionality accidentally worked.

We considered a couple of approaches to prevent this from happening again. We already have a copy of our production data that we use for some testing. This is not always feasible since the amount of production data results in a substantial setup time. Most of the time, we will use the same scripts we have always used to create a clean database. However, we added an additional step to the scripts where they set the auto increment starting values for each table at 10,000 value increments from each other. This will prevent us from “accidentally” thinking our application is working.

Posted in system testing | No Comments »