In sci.agriculture Torsten Brinch wrote:
On 1 Sep 2003 11:52:50 GMT, Brian Sandle
wrote:
No you need to be quite good at the subject to deal properly with
outliers. I am thinking that sometimes people do not eliminate them when
they should be, and others do but don't acknowledge it.

Imagine you represent a conservative govt applying as little as possible
health funding to a village of 100 people of mainly low income, based on
whether they can pay for it themselves or not. When calculating the
average will you include the income of the one multi-millionaire in the
village? That would make the average income rather higher, so you can fund
less. But the other 99 people would have no ability to pay, consequently.
The place would become a real eyesore.

But that's not an outlier problem, Brian. Indeed, there's not much of
a statistical problem in it :-)

Except in that part of statistics is deciding what measures to use and
what to measure.

You have sampled the whole population,
you know the income of each and every individual in it, you know their
average income. The average is the average. It is just not a very good
descriptor for what the politicians want to describe.

Then you might change to the middle income. That might not work either if
there is a big tail of very low incomes.

Then if you were looking for how much the village could potentially donate
to a cause would the high earner still be an outlier?

Yes. In that situation average income might be a more suitable
descriptor. But again, this is not an outlier problem. The high earner
is known to be part of the population studied, so the data point
representing his income can never be considered an outlier.

So you might change from a purely latitude and longitude basis for the
sample to some other. Perhaps it is the subset of employees in the region.

Say you wanted to persuade people that potatoes in general are not high on
solanine. How many sweet potatoes are you allowed in the sample?

Given the figures the sweet potatoes might appear as outliers. This might
lead back to calling into question whether a sweet potato is a potato.

I think Gordon has a little point, that he needed to be told a bit more
about the outlier categorizing. But when you search the web for how
frequently `Monsanto' occurs in studies mentioning outliers, how much do
you get?

If you were recording times for a cross country race would you include
ones where runners had obviously taken a short cut or joined the race some
way through it because they were rather briefer than really possible?
(Memories of school cross-countries). Well you might if you were trying to
catch cheats, or runners mistaken about the route.

That's more like it. If you observe from the recorded racing times
that all racers completed the race within a time range of 3-7 hours,
except one racer -- who has been recorded as completing it in 7
minutes -- that does raise the question if the racing time of this
runner should not be consider an outlier.

I think they used to be picked up in a car and dropped off again near the
finish, to come in not suspiciously too early. In this case the finishing
time is not an `adequate measure' of running ability.