Behind the Scenes: Printer and MFP Testing at Wirth Consulting
Wirth Consulting’s hands-on testing includes everything we’ve distilled from testing and evaluating thousands of office-imaging products over the years. We’ve been testing digital office-imaging products from their inception, from the 2-ppm HP DeskJet 500 to the 250-ppm Océ VarioPrint 250, and everything in between.
Our test reports are designed for a wide audience: the end-user, purchasing agent, sales executive, IT department, vendor’s marketing departments, and product engineers…they’re easy for both end-users and executives to use, while providing the depth and technical know-how required by product engineers.
Here’s a look at what goes on behind the scenes when we test a new product:
Step 1 – Unboxing and Setup
Set up and ready to record the unboxing and installation process.
We evaluate how easy it is to unbox and setup a device, and the evaluation is largely based on how long it takes to complete the process. Initially, we simply timed how long it took. However, we discovered that when it came time to box-up a test unit to return it to a vendor, we could not remember how everything went back together. Because we were concerned about how a test unit could be damaged during shipping because of improper packing, we decided to video record the process. Once we began the recording the process, the next steps were self-evident, publish a library of narrated unboxing and set up videos on our Web site and YouTube channel.
The feedback that we get from end-users who use our video library can be enlightening. Here’s one from a user who consulted our videos in order to figure out how to install ink cartridges in his new printer (indeed, installing ink cartridges in this particular product wasn’t too easy):
“Oh man, thanks a ton. Frustration with loading the printer cartridges. First printer I’ve ever owned really so it was making me pretty antsy (only bought the printer because I had some time-sensitive stuff needing to print). So thanks again.”
Step 2 – Software and Network Installation
Narrating a video capture of the software and network setup processes.
Because of our extensive contact with end-users, we know that office and IT managers value quick and easy deployment of a new imaging product. Consequently, we have always evaluated the software and network installation and deployment processes.
Similar to above, we used to time how long it takes to install the client software and the device on the network. However, we decided that it may be more useful to video-capture the entire process and provide narration based on our experience of installing hundreds of these devices over the years. Those interested in software functionality will find this process particularly useful, as we also include an overview of software use and functionality. Finally, we screen-capture frames from the video for inclusion in the test report and include them in our library of videos on our Web site and YouTube channel
Step 3 – Maintenance and Use Photo Shoot
We take still photographs of all controls and maintenance procedures for inclusion in our test reports. On the right, the tester uses a remote camera control in order to photograph removing the duplex unit. We then create a catalog of photos and include them in our Photo Galleries on our Web site and Flickr Photostream.
Step 4 – Produce and Analyze Image-Quality Samples
Using the most appropriate quality settings, we print this selection of image-quality test samples on premium ink-jet/laser paper and the recommended photo paper.
We analyze key components of each of the printed samples under magnification, then photograph them and insert the uncompressed TIFF photos into our test reports.
Step 5 – Print and Copy Productivity
Using standard test procedures and a suite of test originals, we time and calculate print, copy and scan productivity and determine the print file sizes for inclusion in our test reports.
Productivity test suite and the resultant print file sizes
All testing is performed under controlled conditions with a stabilized network, and a “clean” computer workstation with the latest service packs and all application-software updates installed. We test using good lab practices and replicable testing based on available standards. Our current test platform is Microsoft Windows 7 Professional, Microsoft Office 2010 and Adobe Acrobat 9.5. We also have the ability to test using Apple Mac OS and iOS, Android and Linux.
It’s essential before testing begins to start with a standard clean PC in order to obtain replicable test results. We take the following steps before commencing a test:
We un-install all unnecessary software because our experience has shown that it may compromise testing.
We ensure that unnecessary print drivers are completely removed (left). We then perform a disk cleanup. Disk defragmentation is scheduled to automatically run after hours.
Finally, we run a registry cleaner to ensure integrity. Whenever possible, we utilize the Windows “system restore” feature.
Step 5 – Analyze Software
We install the full-suite of included software. Once it’s installed, we perform an analysis of both client and management software. During the analysis we record details and strengths and weaknesses of all installed software, including print and scan drivers. We then assemble a library of screen captures for inclusion in the test report, and remember, there’s narrated software installation and overview videos that back it all up.
We catalog and analyze all software installed using the full installation procedure. We catalog, analyze and report on all software installed during the full installation procedure, and detail the functionality of any processes that run in the background.
We also describe the entire installation process, count the number of “clicks” required, report the time it took to perform the installation and the availability of any deployment aids, such as driver packaging utilities or scripting tools.
We analyze and report on the functionality of any included client software, including scan and print drivers.
Step 6 – Compose the Test Report
Once Steps 1-5 are complete, we assemble the test results, photographs, screen captures and detailed specifications and produce the test report. Our reports begin with an Executive Summary with an overview of the product and its performance during testing. In general, our Hands-On Test Reports exceed 30 pages so there’s plenty of detailed information within for every type of interested party. Don’t forget that we also include supply cost and yields and calculate Cost per Page and Total Cost of Ownership.
Step 7 – Fact-Check the Test Report
Once we are satisfied with our efforts, we submit the report to key vendor personnel for commenting and fact-checking of the test results, observations and specifications. We then address and resolve any comments and consider any product-placement considerations from the vendor.
Step 8 – Publish the Report
If the vendor purchases distribution rights, we publish the full report on our Web site. Other times, we make full reports available as a complimentary value-add to our customers and readers. More intensive or critical reports are available for purchase.
The Value of a Test Report – How it Makes Life Easier for Marketing Executives, Administrators and End-Users
Bypass the iron fist of corporate IT and get your report up on our public Web site on the day of product launch. Better yet, our readers don’t have to negotiate a paid firewall or pay an expensive subscription–no user names or passwords required–the report will be freely and easily available to the public. We are available 24/7 to assist in your efforts.
Marketing executives and end-users can use our reports in a variety of ways:
- Leverage our trusted impartial “tell it like it is” reputation.
- Use our customized logos and certificates as product endorsements on your Web site, in show rooms or in sales collaterals.
- Use the test results and intensive hardware/software feature analysis for customer/seller/reseller education, product planning, competitive analysis and sales training.
- Distribute the test report as a customer “leave-behind.”
- IT decision makers get an impartial and in-depth look at the installed software, ease of software and network installation, ease of deployment and can gauge the scope of user training and support required.
- Application specialists obtain detailed information on client software and ease-of-use from an end-user perspective.
- Decision makers can get comparative information on performance, value, Cost Per Page and Total Cost of Ownership.
How to Get Your Product Tested
Vendors who wish to participate in our free test program should contact us. We will evaluate any digital-imaging device free of charge. You’ll get an opportunity to review the entire report before we publish it. All you need to do is ship us a test unit, preferably with duplex printing and networking. We also evaluate software and solutions. We will also promptly ship your product back to you in like-new condition.
That’s it, but not necessarily in a nutshell. We hope that you find this information useful, and hope to follow this up with more guides on how to use our services that will make your job (and life) easier.
For more information on our testing and procedures, see here.
Commissioned and Benchmark Testing
Wirth Consulting is also available for confidential commissioned and benchmark testing at our location or yours – whether it’s productivity, image-quality, competitive research, or consumables-yield testing, we’ve done it all. With our pre-launch testing, we will pinpoint product strengths and address any limitations before introduction, so that you can most effectively face off against the competition. All test results remain strictly confidential and become the sole property of the vendor.
What Others Have to Say
“Many business-machine analysts base their recommendations almost entirely on performance test results, but Wirth Consulting goes a significant step beyond that. Wirth Consulting filters its lab results through a screen of end-user expectations in areas like ease of installation and maintenance, software quality, network compatibility and job management.” – Product Marketing Manager