April 27, 2010 by mensming.

Nagios is a well known tool with operations teams. It is used to monitor all kinds of operational parameters - from simple machine up/down monitoring to detailed data collection. However, I have rarely seen this useful tool used in test environments. Here are three ways Nagios can provide benefits to a test team.

First, just using Nagios to monitor whether test systems are up and running can provide useful information and possible time savings to a test team. Knowing that a database server has gone down might save the entire test team time and frustration from tracking down "bugs" which are just the result of a machine outage.

Second, consider basic CPU and memory utilization monitoring of all test systems. This data can be collected and graphed with a variety of tools. I have had success using RRDTool and nagiosgraph. This toolset allows the team to see the variation of CPU utilization, memory utilization and whatever else you decide to measure over time. This view may allow the team to spot potential performance or scaling issues long before formal performance testing begins.

Finally, consider writing your own plugins for measurements unique to the system under test. Once you start doing this, you will discover all kinds of things Nagios could be used for to aid in not only monitoring the test environment but actually testing the application. For example, I once wrote a plugin that would run a database query to verify the number of record processed in the last 15 minutes. I set appropriate thresholds. Weeks later, I received a monitor email alerting me that no records had been processed in the last 15 minutes. Even though I was not testing that part of the system, I immediately knew we had a major issue with the latest build.

Posted in IT, system testing | No Comments »

March 20, 2010 by mensming.

I have written about accidental correctness before. I recently released a fix for an item that was "accidentally correct" for nearly a year.

Early last year, we released a new feature which used Lucene to index the entries. As part of the UI, the entries were displayed - newest entries first. Things worked great. Until the new year.

It turns out we were sorting not on the date but on the date string stored in Lucene. The string was in MM/DD/YYYY format. So, the entries that started with 12 were before 11, etc. All appeared correct. The newest entries were at the top. Then January came. All of a sudden, the newest entries were at the bottom of the list. Sigh.

Lesson re-learned: make sure your test dates span years.

Posted in unit testing, system testing, web testing | No Comments »

June 4, 2009 by mensming.

A friend sent me a link to this graphic. Not sure who to attribute it to. The image is hosted on http://media.oldben.com.ar.

Posted in system testing, web testing | No Comments »

April 18, 2009 by mensming.

In a prior post, I described an example of a side effect of a feature that our users came to rely on as a feature in its own right.

While reading Raymond Chen’s The Old New Thing: Practical Development Throughout the Evolution of Windows, I came across what I consider the mother of such side effects. In an essay entitled “The hunt for a faster syscall trap”, Raymond describes how an Intel representative was perplexed when a Windows engineer requested speeding up the fault when an invalid instruction is executed. This would seem to imply that the Windows code was buggy in some sort of way. The story goes on to relate that executing an invalid instruction was intentional - as a way to get into the CPU kernel mode.

A version of the essay can be found online here.

Posted in software engineering, system testing | No Comments »

March 22, 2009 by mensming.

In show 146 of Hanselminutes, Scott interviews agile coach Scott Bellware.

It is an interesting conversation about whether test driven development is a misnomer, the misuse of the word testability, test smells, etc. A valuable conversation for testers to consider when discussing ways to improve the quality of a product. (Especially if someone asserts that additional testing is not required since "we use test driven development".)

Posted in software engineering, process improvement, system testing, web testing | No Comments »

February 18, 2009 by mensming.

On the last day of 2008 - a leap year - some models of the Zune were reported as not working. When the date rolled over to January 1, 2009, all was fine. Even though I was not impacted by this bug, I can relate.

At my second job out of college, our product was hit by a similar bug. The product used both Gregorian and Julian calendars (I do not recall why…). We had done all of the normal leap year tests around February 29 for the year in question. On New Years Eve and New Years Day, our support lines lit up. Some month end, quarter end and year end processes were failing. Sure enough, they did not expect 366 days in this year.

Ever since that time, I always wanted to have a set of systems that were running some number of months in the future so I could discover these misses before our customers did.

Posted in system testing | No Comments »

January 3, 2009 by mensming.

With the new year, it is time for me to write my annual bug. Like many products, ours includes a copyright statement of the format "© 2006–2008 Company Name" that is displayed to our users. Every year, I submit a bug to change the ending year.

The United States Copyright Office provides the following guidelines in the copyright basics PDF.

Form of Notice for Visually Perceptible Copies

The notice for visually perceptible copies should contain all the following three elements:

1. The symbol © (the letter C in a circle), or the word “Copyright,” or the abbreviation “Copr.”; and
2. The year of first publication of the work. In the case of compilations or derivative works incorporating previously published material, the year date of first publication of he compilation or derivative work is sufficient. The year date may be omitted where a pictorial, graphic, or sculp­tural work, with accompanying textual matter, if any, is reproduced in or on greeting cards, postcards, stationery, jewelry, dolls, toys, or any useful article; and
3. The name of the owner of copyright in the work, or an abbreviation by which the name can be recognized, or a generally known alternative designation of the owner.

By these guidelines, the second date is not required. However, sometimes it is better to go with the flow.

Posted in system testing, web testing | No Comments »

December 27, 2008 by mensming.

Over several months, Michael Hunter, in his blog on Dr. Dobb’s Portal, ran a series of posts with the title of "You Are Not Done Yet", each of which picked an area of a software deliverable and listed items that a tester needs to consider before considering themselves done testing it. This is a great series and I have referred numerous individuals to it time and again.

I recently wanted to refer someone to it and it took me a while to find the particular URL I needed. So, in order to save myself some time, here is the link to the table of contents to all of the articles in the series.

Posted in system testing, web testing | No Comments »

December 14, 2008 by mensming.

While it has been a while since I needed to do installer testing, I recently researched a product that may be helpul in this task. Sandboxie strives to isolate programs from making changes to the rest of the operating system. The Sandboxie website describes it as:

Sandboxie runs your programs in an isolated space which prevents them from making permanent changes to other programs and data in your computer.

and lists as a benefit:

Windows Stays Lean: Prevent wear-and-tear in Windows by installing software into an isolated sandbox.

Sandboxie adds a third approach to installer testing. The first approach, to install over and over on the same instance, is expediant but not necessarily the most realistic. The second approach is to re-image / restore the system. When I first started, this meant using a program such as Ghost, now owned by Symantec. This approach has been supplanted using virtual machines. Virtual machines have made this process much faster.

Using Sandboxie, it is possible to have the cleanliness afforded by virtual machine or images plus the expediency of installing over and over on the same system. On a single system, multiple sandboxes could be created - one for each install test. This approach seems reasonable to me. It is still necessary to do installer testing on native system, but this can be done after many bugs our found and resolved.

Pricing is very reasonable for a commercial license. Check it out.

Posted in test tools, system testing | No Comments »

April 30, 2008 by mensming.

A lot of developers I have worked with have been amazed to see that I expand a suite of tests they developed by a factor of four. This is almost always because I am stressing error handling which is often not tested at all. Unfortunately, these tests will often cause many warnings. Sometimes I want to verify the exact warning. Other times it just clutters the output being sent to the screen. This can be very dangerous - it may lead someone running the test to believe there is a newly introduced bug.

Luckily, perl has a well defined mechanism for overriding the warn behavior. If you want to suppress warnings from being displayed in a section of tests do the following:

Once the scoping ends, whatever warning handler was in effect before will be active again.

Posted in test tools, system testing, web testing | 2 Comments »