Some reviews are hard to believe

[dennis thompson]

I saw a review of scroll saws by Woodsmith.
They rated the Wen as the best, better than the Dewalt and the Delta.

I find this surprising and ,to be honest, hard to believe.

[John Redford]

I have a wen scroll saw and find it to be pretty decent.

[dennis thompson]

I have a wen scroll saw and find it to be pretty decent.

I don't doubt that the Wen is pretty decent, I have a Wen sander and an air cleaner and they are good tools. I just don't believe the Wen scroll saw is better than a Dewalt or Delta.

[glenn bradley]

You have to look at the qualifiers. If price is one of the qualifiers cheap tools tend to rate higher than one might expect.

[Alex Zeller]

I don't know how you can open up a box, take a tool out, and rate it. My best tools are ones that function just as they did the day I bought them years (or even decades later). In fact I have had cheap tools with features that seem better than more expensive tools but quickly failed. Even the so called long term test of cars where they keep them for 40k miles is mostly meaningless. Check back after 140k miles and see which car is the best.

[Andrew Seemann]

Well, just because someone reviews something and posts it, or puts it into a magazine or website or video, doesn't mean the actually know what they are talking about. Same if they are unbiased or an engineer (even woodworkers in woodworking magazines sometimes), they still might not know what they are talking about (even if they are truly unbiased; that doesn't offset being unknowledgeable).

Consumer Reports sometimes is the worst. They do their tests, and the tests might be done correctly, but they often don't matter to the actual usage of the product, or what a professional user of the item would be looking for. My favorite Consumer Reports one is where they differently rate two brands that are in fact identical (e.g. Ford Tempo/Mercury Topaz), and then defend it by saying "that is what the data said". Well, maybe you don't understand your data then. I recently saw a digital piano action comparison on a music website that showed different downstrike weights on different keyboards . . . with the exact same action (sometimes even scientific oriented reviewers don't understand the concepts of tolerance and variance).

The other thing is that the review criteria might not be relevant or consciously/unconsciously manipulated. Another of my favorite Consumer Reports examples was a ranking of vehicles that lasted the longest. They did a some analysis of vehicle registration milages and found out what vehicles had the most miles on them. The list was exactly what you would expect: Ford F-150 pickups, Chevy Suburbans, Dodge Durangos, Jeep Cherokees, etc, basically the vehicles that run forever and that people fix and drive forever. This apparently surprised them, because none of them were vehicles they rated highly. So, rather than reexamining their rating criteria, they changed the criteria of the ranking and made it so Hondas, Toyotas and Lexi came out on top.

Even fairly knowledgable people can be off the mark sometimes. There is a YouTube guy that normally does pretty good reviews, but he did one on what wood glue was the strongest. He then did a bunch of tests that would make most woodworkers cringe, as they were non real-world woodworking situations. The main problem with the test though, is that as most woodworkers know, any standard wood glue is stronger than the wood itself, so the relative strengths between the glues really aren't relevant, what matters are the other characteristics (open time, initial tack, cure time, water resistance, etc). Despite having an unbiased review from a relatively qualified person, the tests themselves were flawed and the entire test not relevant.

[Dave Anderson NH]

PLUS ONE ANDREW. An excellent presentation of why we all should apply the "old saw" Caveat Emptor. Classic apples and oranges.

[Ken Fitzgerald]

When I read reviews I try to find several different sources that reviewed a product to get different views.

[Stan Calow]

I disagree. I am a long time fan of Consumer Reports, and they base much of their recommendations on reliability as reported in surveys of owners, and owners willingness to buy again. Major remodels of vehicles score low when there isn't much history of the newer models, but they always show the owner reported history over several years.

[dennis thompson]

I often look at "Project Farm" . He does some incredible testing and reviews, things such as glues, tools , etc

[Bernie Kopfer]

I disagree. I am a long time fan of Consumer Reports, and they base much of their recommendations on reliability as reported in surveys of owners, and owners willingness to buy again. Major remodels of vehicles score low when there isn't much history of the newer models, but they always show the owner reported history over several years.

Thank you Stan for defending CR. Using the information from their test reports has saved me many dollars and time over the years.

[Ronald Blue]

Thank you Stan for defending CR. Using the information from their test reports has saved me many dollars and time over the years.

If one takes their (CR) data as gospel there would be Ford F-Series, Chevrolet Silverado's, and Ram pickups littering the roads broken down because they are so unreliable. The reality is there are more pickups with extremely high miles than most passenger cars. I know of several in all three brands with 3 and 4 hundred thousand miles and the only shop time was for wear and tear maintenance. Brakes, shock's, and scheduled maintenance. Sure some people have trouble but while there are indeed some lemons out there some people also have trouble with everything so there might be a pattern there. I'm not dinging Toyota or Honda because we know their reputations for usually getting it right. Every dealership has a service department and I'm certain you will find mechanics working on vehicles at everyone of them. I just view CR skeptically. I've bought my share of new vehicles over the years and I was never contacted for my opinion by them on any one of them. So where does their data come from? Their subscriber base? I don't subscribe and never have. I've looked at their reviews at different times but I bought what I felt fit my needs and not what they recommended. The rule of thumb is shy away from brand new models for the first year until it's known if they have issues. I just see CR as another source of info but they aren't the authority to go by exclusively.

[Stan Calow]

Yes, of course they can only survey their membership base. They ask about other products too, not just vehicles. And they never said those trucks are unreliable. Just point out if and when they have above average number of reported problems over time. They report on new cars based on clearly stated performance measures, and long-term durability is not the primary or only factor. Their goal is not to sell cars (unlike other testing platforms) but to inform people shopping for cars. And they dont rank each individual vehicle like some kind of contest, just give a rating on performance and value. Sure its all just information and data that no one takes as gospel. If there is another independent source that you think is more reliable please share.

[Jerome Stanek]

I rate CR, Better Homes and Garden, JD Powers all in the same boat. you need to ask them to compare and you have to pay them

[Mike Kees]

If you really want to know what is good and what isn't about a vehicle you are looking at check out Phil Edmonstons "Lemon Aid" series. Truly unbiased and shows no mercy exposing all recalls ,issues and warts of cars and trucks. Unfortunately he has now retired and discontinued his life work of publishing these books.

[Ken Combs]

I rate CR, Better Homes and Garden, JD Powers all in the same boat. you need to ask them to compare and you have to pay them

IMHO, the flaw in all is the lack of weighting of failures in the auto reports. Infotainment system issues can be frequent with a lot of brands but should 'would not link with my garage door', or 'monitor hard to read in bright light' be counted the same as 'my brakes required replacing at 5k miles'? I d.on't think so. Either weight or measure by category and the info would actually be useful