Brand reliability--Perception vs reality

Then why does Scion rate higher?

I don't really pound on my tC, but I don't baby it either. It has 165 HP,
and I use it to a fair advantage.

Did I ever mention this car is *FAST* ?

 

Perhaps, any car can go fast but it certainly is anything but quick if one
needs to get our of the way.

 

That's been a complaint I have had about CR's "polling" for a very long
time. Their surveys are useless.

Mike

 

It seems to me that you are overlooking the real reason. Scion only
makes about 4 vehicles, and they are based on solid, simple platforms.
Toyota makes many more vehicles, and there is no Scion V-6 Camry
automatic, so Toyota takes the hit for the few models it makes that are
less reliable. Pretty obvious, to me at least...

 

Howing plowed through all of the posts -- or at least most of them -- I
guess the conclusion is that there are no reliable statistical data vis
"reliability." So, you go with what has worked for you over the years, I
suppose.

 

That's true once they start selling a half million or more of a model more
of the 2%, that they all make that are not up to snuff, begin to appear.

 

I wonder how many of the 16,000,000 buyers the bought vehicles in 2006, are
among CR's reported 320,000 subscribers and the percentage of those 320,000
subscribers respond to their surveys?

 

It's interesting to see the perception, since the reliability surveys
from CR are simply the results of owners of each brand filling out a
survey about problems they've had with the vehicles. Since it's a huge
statistical sample, the CR reliability ratings are very accurate.

I think it was a good idea for CR to publish the perception list,
because I've often seen posts from people that don't understand both
statistical sampling and the survey methodology. Some people believe
that the CR reliability ratings are actually just the perceptions of the
people filling out the survey, and they don't realize that what's
being surveyed is actual owner's experiences with the vehicles _not_
what the survey takers believe is the most reliable vehicle. Similarly,
many people have no concept of statistical sampling and margins of
error, believing that if you don't survey every single owner you don't
have enough information to make reliability predictions. Separating
perception from reality should solve the first misconception, though
explaining sampling theory is more difficult.

 

For the perception part of it you're correct. But for the reliability
side, the owners are simply filling out the survey for the vehicles they
own. Unless you believe a Toyota or Honda owner is less likely to put
down actual problems in the survey than a Ford or GM owner, the surveys
are statistically sound. Who knows, maybe a Toyota or Honda owner has
much higher expectations and would complain more about defects than a
Ford or GM owner that expects more problems.

 

You see the problem - that the group is self-selected and therefore
statistically unsound. We can't arbitrarily say there are no differences
that would cause a Toyota or Honda owner to have different motivations than
a Ford or GM owner, especially since we hear from Japanese or American brand
partisans so frequently. In the same way we can't say just what the effect
of those passions and prejudices are. The results would also vary depending
on when the owner bought the car. Was it brand new, in its second year or in
its fourth year? The owner experience and the gist of what he reports will
depend on that - particularly whether he owned it during its first year of
life - to a great extent but it is not included in CR's methodology.

The surveys are also susceptible to "gaming": if a 2004 Malibu (for example)
owner wanted to improve the market value of the car he is planning to sell,
he can send in very many responses claiming to be another perfectly
satisfied owner even if his car was actually a lemon. Similarly, if he
wanted to buy a 2008 Camry he could flood CR with reports that he had
nothing but trouble with his (fictional) 2008 Camry in hopes of driving the
market price down.

No matter how you slice it, CR surveys are a textbook example of sample
selection problems. Any resemblance to typical owner experience is
coincidental.

Mike

 

The problem is that perennial bugaboo of statistics, sample selection. As
you say, a small *truly random* sample is plenty. Beginning stat textbooks
are fond of pointing out that even teh largest nation's election could be
precisely predicted by fewer than a hundred properly selected samples.
That's just way the math side of it works.

Getting a random sample is invariably the biggest challenge to any survey,
and CR doesn't even pretend to try. Self-selection - even when it is more
subtle than CR's open invitation format - will turn any survey to trash.
Many a survey has been invalidated by the mere fact that in most cases the
potential respondents can't be compelled to respond, and can't be compelled
to respond honestly. No quantity of self-selected respondents can produce
usable results. And that is the case with CR's surveys.

Mike

 

That's about all you can do for new cars. I prefer used cars, with at least
70K miles on them. 100K miles is the sweet spot; at 50K it is hard to tell
how a car has been treated and how it is holding up, but at 100K it is hard
to hide. I have developed a method that works for me... at considerable cost
over the years. There are a handful of mechanical checks I do for any used
car, taking about ten minutes, a flashlight and a pair of coveralls (plus an
ODBII reader for 1996 and later cars). But even more important is some
internet searching for complaints. Usenet, reviled as it is, is a gold mine.
I have ruled out Subarus in the year range I am interested in because of the
prevalence of head gasket and "torque bind" posts at alt.autos.subaru, and
the regulars responded very frankly when I asked what the deal was with
those problems. Good people on alt.autos.ford gave me their best skinny on
models to seek and models to avoid. These are not statistically valid, of
course, but I am not planning to write a doctoral dissertation. I want to
know what to look for. Instead of some insipid chart showing poorer than
average reliability for the engine (whatever that may mean) I know that
failure to "burp" a 2.5L Subaru engine after changing the coolant will
quickly lead to head gasket failure.

New cars are a riskier proposition. Changes from one year to another can
plunge buyers into nightmares, and if you are going to keep a car for, say,
five years the model isn't safe to buy until it has been relatively
unchanged for five years. I have owned only three new cars in my life, and
the first two (1970 Mercury Capri and 1984 Dodge 600ES) were two of the most
troublesome cars I've ever owned.

Mike

 

It all balances out with such a huge sample. It's not a double-blind
study, but you'd never get that. Even if you sampled randomly you'd
still have the different motivations of the different owners.

It's not perfect, but it's a very large sample with a very small margin
of error.

The CR subscriber base tends to be higher income and more highly
educated. It's unlikely that they'd be buying used cars.

 

You're confusing "usable results" with the "perfect results" of a
double-blind random survey.

Unless you believe that a large number of the subscribers that respond
to the survey are intentionally lying only about certain vehicles, while
telling the truth about others, the reliability survey is in fact very
usable. You have to take it for what it is, a survey of owner's
experiences of the reliability of their vehicles, with results only
reported if a sufficient number of respondents own the vehicle in
question. If 30% of Camry owners report problems with the transmission,
and 5% of Accord or Taurus owners report similar problems, you have some
usable information. Maybe a double-blind random survey would have
slightly different percentages, but the information in the CR survey is
still valid.

It doesn't matter what the survey is, or what the source is, you always
have people that don't like the results trying to attack it if it's not
a double blind random survey. You see the same thing with the CR survey
on cellular carriers, the largest survey of its kind. Again, you have a
few subscribers of the carriers that do extremely poorly year after year
whining that the perception of Verizon of a carrier with superior
coverage is causing the Verizon subscribers to rate it highly, while the
AT&T subscribers somehow are out to bash AT&T. This is despite the fact
that every other survey from non-advertiser based organizations reports
the same results.

Then you have the people that are confused about quality versus
quantity, claiming that since McDonald's sell the most hamburgers of any
restaurant, that proves that McDonald's has the best hamburgers, and
that since GM sells more vehicles than Honda or Toyota in the U.S., that
proves that GM produces the best vehicles.

 

In addition to the responses SMS gave you, Michael, your last paragraph
is incorrect. As I recall, each subscriber only gets one survey.

The factors which can (not will, mind you, but can) skew results are
reasonably well understood, and generally apply across the board to
responders. For example, CR subscribers, on average, take better care
of their vehicles than non-subscribers. This is attributable to the
higher than average level of education, as is their higher level of
income. But since these factors apply to the general population of
subscribers (rather than, say, only the Ford owners), no inherent skew
is apprehended.

There are other factors, and some of them may present variations in the
distribution curve; for there to be a great deal of variation, though,
we would have to impute that the results were invalid. Toyotas and
Hondas are not generally more reliable than other vehicles, for example.

The observable world doesn't seem to support such a conclusion.


RFT!!!
Dave Kelsen

 

No - the problem increases with larger samples. In fact, the poor sample
selection makes the size of the sample totally irrelevant. For example, if
you are doing a survey of luxury homes built by premiere architects and are
collecting your data by asking people who pass you at the exit of WalMart,
more answers only mean more useless responses. There is no cure for sample
selection errors.

It's a sample with an indefinite margin of error. When samples are properly
selected the margin of error is a simple calculation, but when the samples
are contaminated the margin of error can't be calculated. With a properly
selected sample set the margin of error drops like a rock with more samples
and it is rare that more than a hundred samples are necessary, but when the
sample selection process is flawed the margin of error is unpredictable
after the first sample is collected and never improves.

You have data on those assertions, perhaps? I gave up on CR when I was about
20 because they often incorporated their stupid ideas into their reviews; as
an avid cyclist their shocking recommendation of "dual action brake levers"
on bicycles (now a thing of the past - the current high end devices of the
same name are completely different from the dangerous levers of the 70s CR
was touting) tore it for me. I have six figures on my W2 again this year and
have worked in high tech jobs since I was 17, when I put my first class FCC
radiotelephone license to use. I have only recently bought my third new car
ever and have bought 11 used cars that I can recall. New cars are a blind
wager and the depreciation is not the sort of thing most intelligent people
welcome... unless they are so rich they don't care how much trouble a car is
anyway. Of course the CR readers (maybe not the ones who respond to the
surveys, but who knows?) could still be highly educated; intelligence and
education are distinct concepts.

Mike

 

Thanks for the correction, Dave. I must be behind the times - there used to
be an on-line survey that was anonymous, but I could also be mistaken that
it was CR. I withdraw that objection.

Mike

 

You are demonstrating the worst effect of selection errors - the apparent
usability of useless results. Double blind techniques are used to prevent
reporting and collection bias when the sample has already been selected, and
that doesn't apply here. Randomness is important here, and that is where
CR's surveys are most seriously deficient.

In your example of transmission problems, we can see how that works. If ten
percent of Camry owners are interested in responding to the survey, and of
those 30% are interested because they had transmission problems while the
rest are interested because they loved their car so much or they had some
other complaint, the result would be to magnify the transmission problem
from as little as 3% in the real world to 30% in the survey results because
30% of respondents (not owners) complained about their transmissions.
Conversely, if 90% of Taurus owners felt compelled to respond because there
were daggers embedded in the driver's seat (seating comfort = poor) but only
5% of those had transmission problems, the survey results would be as you
describe... although the reality would be that the Camry transmission
failure rate was as little as half the Taurus rate rather than six times as
high. We can't even say what the actual rate was for Camrys even though we
know - as CR does not - only 10% of Camry owners responded, because we don't
know what the other 90% experienced. Did they have no complaints at all, or
did a lot of them have transmission trouble and were tired of dealing with
it? http://en.wikipedia.org/wiki/Berkson%27s_paradox

Check out http://en.wikipedia.org/wiki/Selection_bias and
http://en.wikipedia.org/wiki/Self-selection for more. In the first link, the
heading "Overcoming selection bias" warns, "In the general case, selection
biases cannot be overcome with statistical analysis of existing data
alone..."

Anyway, to each his own. When I have occasionally looked at CR's predictions
for cars I have owned I have been struck by how far off base they usually
were (although they got pretty close to my experience once!) And to give
them their due, when a problem is as pervasive as the AC and tranny failures
were for the 1993 and 1994 Volvo 850s they picked up on that. If you want to
credit them you can. Personally, I regard the CR surveys as not worth my
attention. I have access to a dartboard that is as reliable.

Mike

 

Double blind techniques are used to prevent reporting and collection bias
when
the sample has already been selected, and that doesn't apply here.
Randomness
is important here, and that is where CR's surveys are most seriously
deficient.
Nor does it matter who complains about what survey; the real problems remain
with CR's methodology.

In your example of transmission problems, we can see how that works. If ten
percent of Camry owners are interested in responding to the survey, and of
those 30% are interested because they had transmission problems while the
rest are interested because they loved their car so much or they had some
other complaint, the result would be to magnify the transmission problem
from as little as 3% in the real world to 30% in the survey results because
30% of respondents (not owners) complained about their transmissions.
Conversely, if 90% of Taurus owners felt compelled to respond because there
were daggers embedded in the driver's seat (seating comfort = poor) but only
5% of those had transmission problems, the survey results would be as you
describe... although the reality would be that the Camry transmission
failure rate was as little as half the Taurus rate rather than six times as
high. This is an example of ascertainment bias
http://en.wikipedia.org/wiki/Ascertainment_bias
We can't even say what the actual rate was for Camrys even though we
know - as CR does not - only 10% of Camry owners responded, because we don't
know what the other 90% experienced. Did they have no complaints at all, or
did a lot of them have transmission trouble and were tired of dealing with
it? As long as we don't know why the responses were sent and the rate
of response for the various models there is no valid data to be had.

Check out http://en.wikipedia.org/wiki/Selection_bias and
http://en.wikipedia.org/wiki/Self-selection for more. In the first link, the
heading "Overcoming selection bias" warns, "In the general case, selection
biases cannot be overcome with statistical analysis of existing data
alone..."

Anyway, to each his own. When I have occasionally looked at CR's predictions
for cars I have owned I have been struck by how far off base they usually
were (although they got pretty close to my experience once!) And to give
them their due, when a problem is as pervasive as the AC and tranny failures
were for the 1993 and 1994 Volvo 850s they picked up on that. If you want to
credit them you can. Personally, I regard the CR surveys as not worth my
attention. I have access to a dartboard that is as reliable.

Mike

 

Dumb Outlook express - this was a draft. Check the other post.