I teach research methodology to family therapy students. Your
questions are right on target when we are in the threats to validity
section of the course. We spend time thinking of all sorts of things
that could possibly influence the outcome. Professional research
generally tries to describe the limitations of the work to guide
future researchers.

I also push my students to asses the threats as to the magnitude of
their probable influence. This set of decisions is harder to make and
imprecise because so much is unknown. A good class will produce
hundreds of possible threats, more than can be controlled unless the
researcher has nearly infinite financing. The result often reduces
the number of clearly-relevant-and-significantly-impactful threats.

We also discuss the magnitude of threat to validity that we would
consider to be important. If we were examining frequency of dyed hair
among clients, we could allow lots of threats to go by. If we were
testing medication with lethal side effects, we would want to run down
lots more of the threats.

My guess is that koi/goldfish research will provide us with some
answers to some of your questions. If Jan can find some research, we
could look at it in terms of possible threats to validity. Your
questions about who paid for the research are certainly on target.

Statistics has a bunch of sayings: 'Garbage in, garbage out' describes
the effect of poor data collection. 'There are liars, damned liars
and statisticians' describes manipulative use of research. I am sure
we will all look at nutrition studies carefully.

Jim