Consumer reports type testing for machinery???

[Michael Ballent]

In another post Rick Thom brought up the following issue, which got me thinking...


It seems that many companies design and build a winning product, then look for all sorts of ways to compromise it. Never know if the product you buy is exactly the same as that tested, or if cheaper parts or lax quality control etc are substituted to save a few $. Trusting brand reputation isn't a given any more.

Does anyone know if there is a magazine that goes out to a retail outlet buys a machine puts it together and reports their findings on the entire experience? I know that Consumer Reports does this for some of the smaller power tools, but they seem to shoot for the DIY crown rather and us :D Rick's point has me wondering if such an animal even exists, especially since CR does not accept advertising from anyone, so they are pretty much free to give their opinion without worrying if they will irk off an advertiser. I know that the mags claim that they report their findings, but I am sure that if they have a clunker on their hands they will probably not even include it in their results rather than report it as a lousy product.

[Dave Anderson NH]

A friend on mine, a former editor of a woodworking magazine, once explained to me that with all of the models available in all of the types of machines, doing really detailed and in-depth testing is a very time consuming and expensive proposition. Magazines have deadlines to meet and they tend to run very lean when it comes to staffiing. Add to this the fact that few, if any, of the staff have an engineering background or the ability to design fully controlled and random variable free tests, and you end up with relatively simple evaluations. It is simply too time consuming, too expensive, and too difficult for major power tools to get anything more than an "impressionistic" type of testing. If you look at tests done by various magazines over the years, they tend to use the manufacturers specs listed in a table, pick what they think are one or two fothe most important performance measurements, and test only those. Given the fact that the magazines derive most of their income from advertisers (rather than subscribers), thee mags will always try to find several positive things to say about even the worst machines. They will tell the truth and they won't lie, but they will avoid outright saying," this product is substandard" jwhenever possible.

The bottom line is that thorough testing requires:
1. Designing a good and unambiguous set of test parameters
2. Having the proper test equipment and the knowledge of how to use it.
3. Plenty of time to do multiple tests for statistical analysis
4. Adequate financial and personnel resources
And finally, the most important requirement is that there be some hope of gaining an appropriate reward for expending the resources whether it be financial or in the form or recognition.

[Cecil Arnold]

Michael, I agree. My feeling is that advertising $$ would sway almost any magazine in its reporting to some extent. Unfortunately, the viscous circle of subscriptions vs. ad bucks seems destined to give us colored reporting until someone decides to develop a narrow-cast publication that can be consumer/subscriber supported. I'm not sure the support will ever be there for such a publication.

[Mark Rios]

I agree wholeheartedly. I subscribe to CR for $20 a year. Not too bad for what you get IMAO. However, they work hard for donations from the public. They also review a very wide range of consumer products. The tool market, even including hand tools, power tools, and machinery still probably wouldn't appeal to enough of a market to support it. Maybe, however, it could include not only woodworking/ construction but also all blue collar trades. But given my personal life, one that was raised around tools (raised in a body shop) and having only worked with tools for a living (construction), and will always be using tools, I would pay a larger price for a subscription. It would have to be the same format and premise that CU has though; NO advertising accepted and not allowing manufacturers or retailers to use the reviews in their adverts.

[Dan Stuewe]

Years ago when Woodsmith was starting to spin off Shop Notes one or the other magazine would do tool tests where they purchased the tools and then had three testers of varying skill/needs (cabinet maker, construction, hobbiest) use the tools and give impressions. Each individual picked his favorite and spoke to why. They didn't seem too overburdened by trying to come up with the most objective testing procedure, just "which tool works best for you".

And btw, as far as I know, Woodsmith and Shop Notes still don't have advertisers except for the last page which give stores to purchase the hardware at.

[Dan Forman]

What I think would be interesting is to something like CR does with their used car issue, to have consumers rate their own tools based on given criteria, plus tabulate comments on overal performance, design, and so on. Part of that would be a reliability component. This would be sort of like a meta version of the Amazon reviews, but done on such a scale as to render false reports by the competition or others with an agenda as statistically insignificant. It would also be interesting to break this down into pro's and hobbyists.

Dan

[Peter Gavin]

Unfortunately, I don't think reviews of tools bought in the past would do much good in evaluating current purchases. Too many 'manufacturers' are subcontracting the actual build to overseas companies and slapping there own name on it to take advantage of their name recognition. Unfortunately, in this race to the bottom that advantage is going to be gone pretty quick and then there won't be much choice at all. I suppose the Chinese/Taiwanese/ Indian companies will someday wake up and realize that if they put out a superior product (paying attention to Quality, consistency and standards) they could sell their own goods that is both better AND cheaper than the US labels.

[Howard Acheson]

The following might be of interest given the topic. In general, Wood Magazine is recognized as doing the best job of machine testing and reporting.

QUOTE

Nothing that has been said so far has changed your opinion that WOOD magazine reviews are somehow influenced by advertising schedules. So I don't expect that I'll change your mind either. But, for those of you who are still listening, let me explain about our tool-testing procedures.

To keep the tests manageable, given the typical 6-page allotment, we must limit the number of models we test to a handful. That may be 6 tools, it may be 10 tools, depending on the complexity of the tool and where we can draw a meaningful line (might be price, might be power, might be capacity, might be a combination of those or others).

Tools are usually acquired 10-11 months before the cover date of the magazine. (Right now, I'm acquiring product to test for our June 2002 issue.) Unlike other magazines that use their editorial staff as testers, we use ndependent testers who are given 1-2 months to complete their tests and
provide the results of the tests.

If we find serious problems with a tool, we report them to the product manager of the manufacturer for their response in the magazine. The article is written from the tester's notes and completed 5-6 months before the cover date. Most of the advertising for a given issue comes in about 2-3 months before the cover date. (It takes about a month for the magazine to actually be color-separated and printed, and the issue usually come out the month before the cover date.)

Neither the manufacturers nor our advertising salespeople know who will get the recommendation(s) at the end of the article. They do know, however, that we'll be reviewing Tablesaws Under $900, for example, and which specific models are in the test.

Frankly, as an editor, I could care less who "wins" any given tool review. My responsibility is to our readers, not the advertisers.

Dave Campbell
Woodworking Products Editor

CLOSE QUOTE