Magazine 'Tool Tests'

[russ bransford]

This is sort of a follow-up to Gary Herrmann's thread of 'congrats dino'.

How much stock do you put in a mag's tool test? Are there certain mags you 'follow' and certain ones you stay away from?

I think Dino appreciates the publicity, but it seems that many times the comparisons don't measure up to your experience. Of course right now I'm speaking about the ez guide. for those who have the ez guide and have the seen the Wood mag test, what do you think?

russ

[Dennis McDonaugh]

I have mixed feelings about tool tests. I think the sidebar boxes listing pros and cons of tools are usually pretty accurate in what they describe. However, most tests are done over too short a period of time to identify all the pros and cons you'll identify after using the thing for a year or two. There's usually no question over the over the "Top Tool" or "Best Value" either. Basically, you get what you pay for. Buy Lamello, Festool or Fein and you probably don't need the tool test in the first place. Where you really need help, and aren't likely to get it, is when you want to select a tool from the Portable Cables, DeWalts, Milwaukees and the like. All of these manufactures produce "pro" quality tools and every one of them has produced a clinker or two in the past. The magazines are no good here because they just don't use the tools long enough to discover the PC 7529 had a bad switch setup or the PC 310 Trim router base wasn't perpendicular to the shaft on many of the production routers. I'm not picking on PC, these are just two of the tool mistakes I've bought. I still look at the reviews if I'm contemplating a new tool purchase, but I just use them to weed out the obvious clunkers.

[Jeff Sudmeier]

What Dennis said!! In rare occasions do the mag's find faults with the tools, in other words wearing them out. They will tell you out of the box which is the best, features, power etc... but not long time wear. It would be a much bigger process for them to do that. (Could get a lemon from the company so you have to test more than one of each model, etc).

[russ bransford]

yeah, i agree with the idea of testing over time. it just seems that with many comparisons and sales pitches you end up getting more features than functions. i realize i may be playing with definitions, but in my thinking, salesmen use features and try to be one-up on the competition. but after you have used the tool a while you discover the real functions of the tool.

it may look nice in the pitch or promo but over time you find out some things not mentioned in the comparison that are great and then again some of the mentionables aren't worth repeating.

some tools are so different in their functions that comparing features doesn't do it justice. I think that's what Dino is running into when the ez is compared to other 'similar' items.

I do like some tests in 'tools of the trade'. they seem to have a pretty good way of testing in the real world, and they usually admit their bias'.

[Gary Herrmann]

Some of them especially seem to think very highly of all tools they test. I like the Fine WWing Tool comparison issue, but mostly to be able to compare specs easily.

That said, I posted on EZ Smart because I've been very happy with it, and despite what I just said above, their eval of the EZ Smart agreed with my own. The other 2 piece guides I've used flexed way too much - I couldn't get a straight cut over 8 feet without clamping or something else.

It's like car reviews. Your own experience will wind up being your best guide.

[Jim Becker]

I've had the pleasure of actually watching some of the process that happens at WOOD Magazine when they do tool testing. It's hard work and they are really careful about documenting things and keeping the playing field level. In all of these things, I think they and most other magazines do a good job.

But I also have to agree with the sentiment that testing needs to go beyond the immediate and longer-term commentary is critical to do a complete job. (Something I like about certain groups that do testing on say, automobiles) Of course, with many tools tests, the models change often enough that this might not work out as well as with cars...and it takes 6-9 months from the beginning of a testing or re-testing effort to publication. The lead times are brutal.

[Harry Goodwin]

I just read one of these magazines to which I subscribe and they are showing ways to save money and have people recommending straightening nails, drier sheets used for finishing and others and I am goiing to buy a recommended tool Ha, Ha. They are still great magazines. Harry

[Jim Becker]

Harry that article is a compilation of posts from their online forum by woodworkers "like us" after a question by the "world famous" Buck Nall...a very interesting fellow and friend...about how folks are thrifty around their shops. The examples are not recommendations of the magazine.

[Steve Clardy]

I haven't actually read the article, as I dropped Wood magazine some time ago.

I have Dinos system. No complaints here, just compliments for his system.

Steve:)

[Dev Emch]

I read them, sometimes, but I dont put a great deal of value into them. The mags are bound to following the sponsors who buy advert space and sponor shows. Thus there is often a lack of objectivity. Also, many times, the top end entries are never reviewed. Of late, I have seen very little comparisons on the made in canada general machines. Often when you see a table saw comparision, you will see the entry from General international and not the General of Canada. Another case in point although there is a lack of electrons here..... Plane reviews often stop at Lie Nielson. What about the rest of the super planes out there? A Holtey, for example, can smoke the pants off a Lie Nielson any day of the week. Yes, holteys are custom made and carry a price tag to beat the band making them not affordable to most. But, they are an option and frquently, its an option not covered. When asked, the mags will say that these items have been left out because then are not in alignment with our reader's demographics. O.K., I can see where they would hold such a belief but shouldnt we be the judge of that? Isnt that not being totally objective to the comparision? So that is why I read them but dont allow them to sway my thinking one way or the other.

[Howard Acheson]

Here is something written by the then Woodworking Products Editor of Wood Magazine responding to a comment about advertiser influence in the tool test results. It addresses many of the issues raised here.

QUOTE

Nothing that has been said so far has changed your opinion that WOOD magazine reviews are somehow influenced by advertising schedules. So I don't expect that I'll change your mind either. But, for those of you who are still listening, let me explain about our tool-testing procedures.

To keep the tests manageable, given the typical 6-page allotment, we must limit the number of models we test to a handful. That may be 6 tools, it may be 10 tools, depending on the complexity of the tool and where we can draw a meaningful line (might be price, might be power, might be capacity, might be a combination of those or others).

Tools are usually acquired 10-11 months before the cover date of the magazine. (Right now, I'm acquiring product to test for our June 2002 issue.) Unlike other magazines that use their editorial staff as testers, we use ndependent testers who are given 1-2 months to complete their tests and
provide the results of the tests.

If we find serious problems with a tool, we report them to the product manager of the manufacturer for their response in the magazine. The article is written from the tester's notes and completed 5-6 months before the cover date. Most of the advertising for a given issue comes in about 2-3 months before the cover date. (It takes about a month for the magazine to actually be color-separated and printed, and the issue usually come out the month before the cover date.)

Neither the manufacturers nor our advertising salespeople know who will get the recommendation(s) at the end of the article. They do know, however, that we'll be reviewing Tablesaws Under $900, for example, and which specific models are in the test.

Frankly, as an editor, I could care less who "wins" any given tool review. My responsibility is to our readers, not the advertisers.

Dave Campbell
Woodworking Products Editor

CLOSE QUOTE

Howie..........

[Steve Rowe]

I've got to agree with Dennis on this one. I place essentially zero stock in magazine reviews of tools, including their editors choice, best value, best buy or whatever. Often, it is difficult to distinguish what is fact from what is the authors opinion or perhaps even fiction or hype. Much of what we as users like (or dislike) about tools is subjective touchy feely stuff that is unique to the individual user. Additionally, we are biased by our own habits and preferences. The same goes for the authors of these articles although often they don't appear to acknowledge this. There are obviously some exceptions to this but, overall very few. I am often amazed at those who post on some forums about they were ready to purchase a particular machine based on some review but then read a different review that reached a different conclusion and now don't know what to do. Sometimes you just have to go with your gut based on your own research and needs and sometimes you make a clunker of a decision and that is the school of hard knocks. Such is life.

[Jim Becker]

Thanks, Howie. Dave is a friend of mine and he's the one who took me through the process when I visited a couple years ago. He's as honest as you could ever find a person.

[Dev Emch]

Thanks Howie for that quote.... Unfortunately, the issues still remain. Clearly the reviews need a structured settlement in order to be reasonable and we accept that. But at the same time, you are excluding options that may be better suited to many readers.

I personally believe strongly that many woodworkers should follow the route layed down by Lou Sansone or myself and others. That does not always mean constructing a stand alone shop of Lou's caliber; however, it does not exclude many with a basement shop from getting "eccletic" (sp?) tool collections. I love that word even though its hard to spell:)

What really bothers me is that our high schools no longer train folks in woodworking and many newbies to the hobby are drawn in through the mags. Forums such as this really do open the eyes of many as to what is out there and how it all works. Shop tours, tool reviews and project pictures really go a long way to introducing newbies to the hobby. Having said that, the mags, by their own choice and article aritecture, focus the attention of knowledge hungry newbies in a limited direction. If you have ever attended a Woodworking Shows show in an active town, you will know how dynamic this force can be. I have personally seen Lie Nielsen Co sell $4000 to $5000 dollars worth of product in a two to three hour time window on an opening Friday afternoon!

My only point was that too often, the top 20 percent of all tools, both hand tools and power tools, are never covered by these reviews. Show me the last Fine Woodworking review of a martin table saw or martin jointer. Where is the holtey review from wood magazine? Etc. Etc. Thus, we should all take these reviews with a grain of salt knowing that the cream has been skimmed and will not be reviewed. Those reviews fall into the laps of folks who blog them out on forms like this.:D

[Peter Pedisich]

The things I find most useful about reviews and comparisons is when they bring up stuff like "the handles on the Milwaukee were thinner than the Bosch and more comfortable after an hour of use". This at least lets me decide, hey, my hands are the size of a mountain gorilla's- the Bosch is for me.

I buy based on a few things:

1) Hold it, if it feels right it is.
2) Has this brand been good to you in the past?
3) Make a list of pros and cons and decide what is important and what's nonsense. Only you can make this list. Don't be swayed by the printed word unless it's your own printed words.

Pete

[Bill Fields]

Howard Anderson is on the mark.


As a retired Publisher of a few consumer magazines, I can tell you the pressure is great on the Editors to "color" their negative comments about a product.

Particularly so, as these comments pertain to a strong advertiser.
We Creekers. observant readers all, can filter this stuff out, right?

Our defense remains our right to RETURN poor performers, and write about our experiences.

BILL FIELDS

[JayStPeter]

I personally believe that magazine editors do their best to remove any bias from their tests (either personal or professional). The fact that a significant number of the tests agree with the popular opinion on the online forums has sort of confirmed it in my mind.
But, I also believe that no review is completely without bias. Even the reviews here tend to have some. Think about any thread you've read asking about a Craftsman product for example (no matter how good or bad the actual tool in question is). When we like a tool we thump our chests and proclaim our brilliance for picking the best possible choice ;) . But let's face it, few of us get to actually test drive all the choices before making our own.
Magazines are the only real source of a woodworker comparing the choices head to head. I agree that they sometimes tend to shoot below what I'm looking for. They have to shoot for the average of their readers. Unfortunately for Dev, there are not many pulications whose average reader is looking for a Martin slider :cool: .

Jay

[Dino Makropoulos]

This is sort of a follow-up to Gary Herrmann's thread of 'congrats dino'.

How much stock do you put in a mag's tool test? Are there certain mags you 'follow' and certain ones you stay away from?

I think Dino appreciates the publicity, but it seems that many times the comparisons don't measure up to your experience. Of course right now I'm speaking about the ez guide. for those who have the ez guide and have the seen the Wood mag test, what do you think?

russ

Russ.
Yesterday I received the hard copy of the wood magazine and the new woodworker magazine from UK.
You can see the UK review/test on line.
http://eurekazone.com/mediaUK.html
After reviewing the 'tool test' of the wood magazine, I bought a Sharp video camera and the Easyshare Z730 zoom KODAK digital camera. :)

YCF Dino

[Jim Becker]

Dev, I do understand your point...it would be nice if the truely high-end stuff would get reviewed or even mentioned. But these magazines are businesses and their market is primarily the hobbiest. The demographics of that market at present show little interest, percentage wise, toward machinery and tools in this space. The fact that the Deltas, Grizzlys and Jets of the world stick to the same-old, same-old doesn't help. Fortunately, forums like this that have a great cross-over population of pros and hobbiests of all economic and age demographics help to fill in the gaps. We can all learn from each other and spread the word farther. That doesn't mean many folks are going to run out and buy Martins...but it may mean they start to expect and demand more from other firms who cater tools.

Pro-oriented publications, such as Woodshop News, should be reviewing this stuff, but so far in my short subscription, there has been minimal exposure. That's sad as many pro shops, even very small ones, could realize a good return on investment due to efficiencies by updating their gear to better and more capable tools.

[Aaron Mills]

I think magazine tool reviews have some merit when used in conjunction with other research. Their tool reviews are set up for a direct comparison of most major manufactuers, whereas the comments or advice you may find on forums are based on experience with a single machine. Where I find the two meet to make a compromise is when both sources agree on the pros/cons of a machine.

As an example, I've been looking at 3 HP plunge routers for a while now. WOOD magazine just did a review on these routers, rating the Milwaukee the best table top router, Dewalt 625 the best hand held, and Hitachi/Freud as the top value winners. I personally couldn't put much stock into the top value winners simply because the cost of the Hitachi and Freud are practically the same price as the mid-market routers. However, I had been considering the Bosch 1619EVS, but had my reservations about the product. Although the Dewalt only outperformed it slightly, this combined with the overwhelmingly positive reviews on some forums have pushed me towards the 625. What negative feedback there has been about the Dewalt is generally pertaining to the soft start, a problem the magazine noted too. I have yet to make the purchase, but it's helped shape my considerations for purchasing.

Magazines and forums also compliment each other because magazines usually have the inside track on when a company will be debuting a new model for that piece of machinery. This can save you a lot of hassle since you can delay your purchase and research more about the new features that accompany the newer model. No one wants to drop their cash on yesterday's child only to have the new kid on the block cost just as much, but has significant improvements. Conversely, forums provide the long term insight that magazine reviews lack. Again, with the routers - I strongly considered the fixed base Milwaukee after reading the WOOD review, but problems with plastic parts on it, made it an unwise decision. While it was readily adjustable above the table, the parts tended to breakdown after a couple of years. Thanks to forums, disaster averted.


What really bothers me is that our high schools no longer train folks in woodworking and many newbies to the hobby are drawn in through the mags.

Even with a little high school shop experience, the mags can be very influential unless you stumble across the wood forums. Consider that most high school shops only feature one brand of major machinery and middle of the road handheld tools. In my case, I nearly jumped at the chance to buy a brand new General 350/650 because that was what I was familiar with in my high school shop. Similarly, I was more partial to Makita/Dewalt tools because they were major players for primary tools in our shop. Familiarity with a brand is very powerful, but brand loyalty should only be a factor in a dead heat. This leads me to:


My only point was that too often, the top 20 percent of all tools, both hand tools and power tools, are never covered by these reviews.

Magazines and forums simply make information available. Most of the time, both only stop to discuss the major brands, but it's true that on forums you can find elusive information on high end tools. The new woodworker will generally opt for one of two routes, buying new, mid-market name brand tools, or scouring classifieds for old iron. The former is generally influenced by magazine reviews and mainstream forum discussion, while the latter is largely the product of sage advice in forums from experienced woodworkers. Seasoned woodworkers who would be willing to put extra money towards the added performance are generally already aware of the merits of the product and tend to put less stock into magazine reviews anyways, so I'm not sure it makes sense for these companies to donate one of their machines for testing.

The responsibility ultimately falls on the end user to inform themselves about their options. No single magazine or forum discussion is sufficient for making a smart, tool purchase. Those guys who go out and buy the tool that is best magazine rated have the same basic expectations as the ones who go out and buy the most expensive tool expecting the best performance. Ditto for guys who solicit advice on forums and then go out the next day and buy that tool, no questions asked. No tool is perfect (though some are close), and in many instances it comes down to weighing pros and cons alongside personal preferences. Those personal preferences can only be established by handling the tool, and debating whether certain features will benefit you as a woodworker.

In summary, my humble thoughts:

1. When magazine and forum advice is in alignment, it means something
2. Use magazines to look ahead to new models/features & forums to identify potential problems
3. Unless models from a different brand are nearly identical, don't rely too heavily on brand loyalty (especially when it's to make your shop look more uniform)
4. Info on up-scale machinery is easier to find on forums, but generally requires some solicitation
5. The responsibility is on the buyer to do research, meaning magazine/internet reviews, visiting forums and asking questions, plus handling and testing the tool in person.

Thanks,

Aaron

[russ bransford]

Dev, you hit one things I was thinking;" What really bothers me is that our high schools no longer train folks in woodworking and many newbies to the hobby are drawn in through the mags."(dev)

this is a point of view, that the mag's seem to be stuggling with. and maybe it gets closer than that, even to the point where you've detected which authors/testers are more reliable.

think of another hobby, that maybe you don't take as seriously as wwking, and think about how you learn those skills. i know when i go into an electronics store looking for an mp3 player, i am at the mercy of someone very young, speaking in very strange terms.

back to the wwking. it's more serious and has the element of danger. i dare say we spend more time figuring out what and where to buy, than we do practicing and figuring out the functions before we start a project.

i think some of the camparisons that are aimed at newbie's need to connect the features with more real life functions of the tools. i realize i'm repeating myself on this feature/function thing, but i'm just stuck in that gear.

you have experience, your mind works toward the functionality of the tool because you have used it/something similar before.

suppose something new comes out and we're not quite sure what we're looking at, what do we do then? ( other than head to the creek ) are we waiting for Norm to use it? just a thought.

i was looking back at some of the posts and have a few more questions: the quote in Howard's post, talked about the long lead times for articles. my question would be to Bill: you've been behind the fence. what do you think the future holds for mag's with long lead times, and yet we're getting used to receiving instant feedback with things like forums, google, digital pic's, etc ?

as for the tool tester not trying to be bias, i would say admit bias. tell what tools you have and use regularly. the bias is already there, trying to dodge it or cover it doesn't work. you're still going to smell it.

[JayStPeter]

My Wood mag arrived today. I have to agree that the test that started this thread was pretty poor.
I'm not saying this because I feel the EZ-smart was ripped off. I just think it is an awful job of a test. Their result was fundamentally flawed.
They named the tru-grip pro the best because of its performance ... fine. But, they named it top tool because of cost (only $88 they claim). However, many of the other competitors come with an extension to make it either 4' or 8'. In this case it takes another clamp (at an additional $99), making its total cost one the highest in the test.
I guess we did learn what the best guide for making a 4' or less dado is. But, we didn't learn what the best general use guide is.

Jay

[Dev Emch]

Many folks have responded to this thread and brought up some excellent points. In this regard, thanks.

As far as including martins and holteys in the tests. Well, the odds are that 99.9 percent of all the readers would not consider yet even purchase one of these super high tools. Many of my tools were purchased used and not new so you know where I am comming from.

But by placing one and only one super high end tool into the test, you are covering the total gammet from super cheap to super best. Clearly, the bulk of the selection will be gathered along a median line in the middle. That is where the bulk of the action will be unless our shoping for super cheap and then you dont need the review. Just drive down to your local harbor freight store and be done with it.

But I would just like to read about the super high end options. If only to live vicariously for a few seconds. Its entertaining and it covers the total sprectrum. By leaving these out, the motivated student begins to wonder what else is out there. Delay the purchase and spend more time doing research:)

So yes, I read many of these reviews and I read the structure of how the review was designed. Interesting? Yes. Follow the reviews blindly and buy the suggested products? No Way. Hold out for the unusual and special? YUP:)

[Ellen Benkin]

What bothers me most is when a "review" seems like it was taken directly from the manufacturer's publicity release or spec sheet. This usually happens when magazines have a "news flash" about some new product. I find this happens more with computer magazines than it does with woodworking magazines, but it seems obvious when it does occur.

[Ian Barley]

Dev's point is very interesting. I have stopped taking magazines because the tools that they feature rarely mean anything to me as I am moving mainly to small industrial scale machines . I also understand that this is not the case for most readers of these mags. I might be tempted to buy again if they included some sort of "how the pros do it" sidebar. So a review of drill presses might have a short piece talking about CNC borers or multiborer machines - not necessarily models - just general capabilities and so on. I suspect that this might be interesting to the the weekend warrior while also helping people like me who are learning about the capabilities of the industrial grade machines mainly from forums (fora??) like this.

[Dino Makropoulos]

My Wood mag arrived today. I have to agree that the test that started this thread was pretty poor.
I'm not saying this because I feel the EZ-smart was ripped off. I just think it is an awful job of a test. Their result was fundamentally flawed.
They named the tru-grip pro the best because of its performance ... fine. But, they named it top tool because of cost (only $88 they claim). However, many of the other competitors come with an extension to make it either 4' or 8'. In this case it takes another clamp (at an additional $99), making its total cost one the highest in the test.
I guess we did learn what the best guide for making a 4' or less dado is. But, we didn't learn what the best general use guide is.

Jay
Hi Jay.
With an edge guide you need offset measurements. And you need to reset the edge guide if you need multiple passes. Good luck here.
The "elaborate" Ez on the other hand gives you 10" ( better every day) to play with. You set the guide anywhere from 4" to 14" parallel to the rout line and you use the sliding arm for precision placement. For multiple passes-flouting mortise and even duplicating you don't have to reset the guide.
The rout area of the EZ on 48" is 960" square inchess
The rout area of an edge guide with 3/4" bit is only 72" square inchess (if the guide is di-directional)
If you need to rout anything longer than 8'-00" you're out of luck with an edge clamp and tool guide.
Same goes for angle routing and thin and flexible materials. The edge clamps don't work there either.
Today I rout few carpet board samples. For a 36" x 54" area I reset the EZ 6 times. With the same bit and an edge guide you need to reset the guide min. 196 times. If you're taking off 1/2" with each pass.
I'm not taking part on this review of the review. :rolleyes:
If nothing else, the numbers can do the talking better than me.
960" VS 72" and 6 VS 196 times.
YCF Dino

[Dennis McDonaugh]

There may be two different issues in this subject: First, tool reviews aimed at those who intend to buy, and I think that's where this thread started. Second, is the whole issue of introducing readers to the exotic whether machine or hand tool. I like to read about anything connected to woodworking and have to admit that Dev mentions brands that I am not famaliar with. I think a review of planes from Anant to Sauer and Steiner or comparing dovetail saws made by Stanley, Sephar, and L&N would be good reading. There's a lot of interesting machinery out there too like Woodmaster (when would you actually use gang sawing with a blade that small?) and woodrat which I've never seen a review on.

I have no idea how magazines get their tools. Do they buy them or are they given/loaned to them by manufacturers? If they have to buy them, I can see why they wouldn't want to buy high dollar tools when most of their readers wouldn't be interested in purchasing them and the manufacturers wouldn't be interested in advertising in the magazine. Still, an article about high dollar hand tools and industrial machinery would be of great interest to a lot of people. Maybe we'll stir up something with one of the publishers with this thread.

[Aaron Mills]

I have no idea how magazines get their tools. Do they buy them or are they given/loaned to them by manufacturers? If they have to buy them, I can see why they wouldn't want to buy high dollar tools when most of their readers wouldn't be interested in purchasing them and the manufacturers wouldn't be interested in advertising in the magazine. Still, an article about high dollar hand tools and industrial machinery would be of great interest to a lot of people. Maybe we'll stir up something with one of the publishers with this thread.

The 'zines dont' usually pay for the tools featured in their tests. The magazines generally have an agreement with manufacturers to donate the tools to an organization like Habitat for Humanity after they finish the testing.

[Jim Becker]

The 'zines dont' usually pay for the tools featured in their tests. The magazines generally have an agreement with manufacturers to donate the tools to an organization like Habitat for Humanity after they finish the testing.

Not necessarily true, Aaron. Depending on the product, it may be bought on the open market anonymously to insure it is a real production piece. And sometimes tools that are provided by the manufactuers have to go back to those manufacturers.

[Aaron Mills]

Apologies for the misinformation. Just couldn't help but notice in a number of magazine tool tests that the tools were provided by the manufacturer and later donated. Is this scenario usually only played out with more expensive, larger machinery in which case it would be nearly impossible for the magazine to test as many machines as they do without breaking the bank?

Also, Jim, you mentioned that sometimes the tools are returned to the manufacturer. I'm curious to know what happens to these machines once they are returned to the company. Do they get repackaged and shipped out as refurbs/demos or are they put to some other use?

[John Renzetti]

If you read between the lines on some of the tests you can see which are the favorites. I just read a review of portable planers in the "Journal of Light Construction." The three reviewed were the Bosch, PC and the Festool. The Festool got the most praise. PC next. The Bosch had some negative features and these were pointed out, but in a nice way.
In the past year or so, I have seen Festool and Hilti included in reviews. Usually in Tools of the Trade and JLC. Holty hand planes have been reviewed, but in the British magazine, "Furniture and Cabinetmaking." This magazine has also done some reviews on larger machines like Dev refers to. They had a review of a Felder K975 and an Altendorf WA-80. I don't think we'll see a review of a 4000lb Martin T-73 in Fine Woodworking any time soon. I can see the write up now, "Although our testers really liked the T-73, it's $40,000 price tag is a turn-off...our best buy award goes to the BT3000 for $695."
take care,
John