You are currently browsing the archives for the test tools category.
December 14, 2008 by mensming.
While it has been a while since I needed to do installer testing, I recently researched a product that may be helpul in this task. Sandboxie strives to isolate programs from making changes to the rest of the operating system. The Sandboxie website describes it as:
Sandboxie runs your programs in an isolated space which prevents them from making permanent changes to other programs and data in your computer.
and lists as a benefit:
Windows Stays Lean: Prevent wear-and-tear in Windows by installing software into an isolated sandbox.
Sandboxie adds a third approach to installer testing. The first approach, to install over and over on the same instance, is expediant but not necessarily the most realistic. The second approach is to re-image / restore the system. When I first started, this meant using a program such as Ghost, now owned by Symantec. This approach has been supplanted using virtual machines. Virtual machines have made this process much faster.
Using Sandboxie, it is possible to have the cleanliness afforded by virtual machine or images plus the expediency of installing over and over on the same system. On a single system, multiple sandboxes could be created - one for each install test. This approach seems reasonable to me. It is still necessary to do installer testing on native system, but this can be done after many bugs our found and resolved.
Pricing is very reasonable for a commercial license. Check it out.
Posted in test tools, system testing | No Comments »
April 30, 2008 by mensming.
A lot of developers I have worked with have been amazed to see that I expand a suite of tests they developed by a factor of four. This is almost always because I am stressing error handling which is often not tested at all. Unfortunately, these tests will often cause many warnings. Sometimes I want to verify the exact warning. Other times it just clutters the output being sent to the screen. This can be very dangerous - it may lead someone running the test to believe there is a newly introduced bug.
Luckily, perl has a well defined mechanism for overriding the warn behavior. If you want to suppress warnings from being displayed in a section of tests do the following:
{ ... }
{
local $SIG{__WARN__}=sub{}; #Temporarily suppress warnings
# Insert test code here
}
Once the scoping ends, whatever warning handler was in effect before will be active again.
Posted in test tools, system testing, web testing | 1 Comment »
April 15, 2008 by mensming.
One of the best conferences I have ever attended was the 2007 VMWorld conference held in San Francisco. Part of this was probably because I was new to VMWare. The quantity and quality of the information shared was uniformly high. For a vendor sponsored conference, this is saying a lot.
However, most of the sessions I attended were presented by VMWare employees. I would like to see more user based presentations. In particular, VMWare experiences outside of the standard server consolidation usage pattern. With this being said, VMWare has issued a Call for Papers for the 2008 VMWorld to be held in Las Vegas on September 15-18, 2008.
Posted in test tools | No Comments »
March 24, 2008 by mensming.
I use VMWare extensively in my daily job. Most of these are Windows virtual machines (VMs) that get refreshed on a fairly regular basis. We use the clone feature along with a customization script to refresh a VM. Add a perl script which uses the API to drive this process and it is fairly painless. If a change needs to be made to the VMs, we make the change to the template and the next time the VMs are refreshed, the change is propagated to all of the instances.
Recently, I needed to add a routine scheduled maintenance task to the VMs. Easy enough — I thought. I created the script that would perform the task. Tested it locally. All works. Tested that it works as a scheduled process. Great. Transferred it to a test VM. Performed the same tests. All is going according to plan. I add the script to the templates. I schedule the script to run at the specified times. Double check my work. Now just wait until we refresh the VMs.
After the refresh of the templates, I go to verify that the script is behaving as expected. Nope. Is the script there? Yes. Is the scheduled task there? Yes. Can the scheduled task run? No. Since our customization script creates new security IDs during the clone, the security ID associated with the scheduled tasks (which was valid in the template) is not valid in the cloned instance.
The work around was rather simple. In the customization script, there is an option to set a "Run Once" command that will execute the first time the new virtual machine is powered on. We already use this feature to invoke a batch file to download our latest code and do some configuration based on the machine name. I modified this batch file to use the "at" command to schedule the script. Test again. All looks good. Perform test clone. All looks good. Wait for VMs to refresh. Problem solved.
Posted in test tools | No Comments »
March 10, 2008 by mensming.
Scott Hanselman, on his Hanselminutes podcast interviewed Quetzal Bradley of Microsoft in show 103. I enjoyed the discussion because it covers two topics I find myself explaining to less technical managers time and time again:
It is a quick listen - recommended.
Posted in test tools, system testing | No Comments »
March 8, 2008 by mensming.
One of the developers I work with keeps telling me that I should take a look at SQLite. We have an application that has a memory and CPU intensive data structure that SQLite might be a good use for. It has been on my list of things to learn more about for some time.
This morning, I discover that there is an interview with Richard Hipp in my feed from FLOSS Weekly. While not an overly technical interview, it is a great overview of SQLite, including typical applications, how the project came to be, how it was released to the public domain, etc.
I can imagine myself using SQLite, especially in some of the more data intensive test and evaluation scripts that I write. I just increased the priority of looking into SQLite.
Posted in test tools | 1 Comment »
May 2, 2007 by mensming.
I suggest that testers learn at least one scripting language. The first one that I learned was perl. Yesterday, I was reminded why a little bit of perl can save a lot of time (and eliminate the possibility of human error). Due to some changes in our corporate email system, we needed to change several hundred datafiles from using one email address to a different email address. While there are many to do this (editor macros, shell programming using tools like sed, etc.) I decided some perl executed from the command line would be appropriate.
I started with:
perl -p -i -w -e 's/oldemail\@meesqa.com/newemail\@meesqa.com/g;' *.data
For those unfamiliar with perl, this line can be explained pretty quickly:
perl — executes the perl interpreter
-p — assume loop like structure. Essentially execute all input to the line as if this were inside a while loop.
-i — edit files "in place" - Change the current copy of the files. It is possible to create a backup copy. I didn’t because all of the data files were checked into source code control.
-w — enable warnings
-e (following be some perl code) — The one line program to execute
’s/searchvalue/replacevalue/g’ — This is the generic form of the program. The g at the end means to apply this expression globally — replacing all instances of searchvalue with replacevalue. In the code above, I needed to escape the @ character so it would be interpreted correctly.
*.data — Finally, what input to pass into the program, in this case all files that match the pattern *.data
After running this in one directory and examining the output, I realized that we had not been consistent with case for the email address domain. While I had fixed all of the instances of @meesqa.com, I missed @MEESQA.COM (or any other permutation of upper and lower case. Easy enough. I made the following minor change:
perl -p -i -w -e 's/oldemail\@meesqa.com/newemail\@meesqa.com/gi;' *.data
The i added after the /g tells perl to ignore case. Rerunning and all of the email addresses in that directory had been changed appropriately.
That fixed one directory of files. However, the rest of the files were scattered over a deep hierarchy of directories. Since I was doing this all on unix, I executed the following line:
find . -name *.data | xargs perl -p -i -w -e 's/oldemail\@meesqa.com/newemail\@meesqa.com/gi;'
The find . -name *.data creates a list of all of the *.data files in the current directory and subdirectories. This is piped to the next command. The xargs command allows this piped list of filenames to be passed in as an argument to the command following xargs — in this case my perl program. Running this was fairly quick. I verified the changes and checked into source while be reminded that a little scripting can go a long way.
Posted in test tools | 1 Comment »
February 6, 2007 by mensming.
One of the things I have been looking for a long time is a system to manage test cases, test results (both manual and automated), support a variety of reports, etc. In addition, I don’t want to be tied into an entire product suite - I want the system to work (I don’t mind doing some integration) with the commercial and homegrown tools that I use.
I have tried a variety of solutions. I currently use a homegrown database backed system with a web front end. However, there are enough problems with it that I continue to look for another solution. I recently did a web search and came up with the following list of items to review (in no particular order):
Posted in test tools | 1 Comment »