Quote:
|
But when there is a result far enough outside the expected, one need to find out why. Skewed data?? some factor(s) present that were not present in other findings??
|
One should always be sure to include the possibility that there's not a problem with the results but with the expectations.
Here's a thought experiment.
Let's imagine that there is a topic that is of extreme interest to many people--to those who find it important, they consider it literally a matter of life and death. It is also true that it is a topic of interest to many large and well-funded organizations. The topic comes down to whether A or B is better. It doesn't matter what 'A' is, it doesn't matter what 'B' is, we just want to know which is better.
Because it's of interest, people study it. Organizations study it. Data is collected, studies are performed. This goes on for years. The problem is that no one can call a winner. People can quantify differences between A and B, in fact that is done to excruciating detail. But none of that seems to help. When it comes right down to practical performance in the real world, as opposed to parameters measured during controlled testing, no one seems to be able to show that one is better than the other.
People keep trying. Organizations keep trying. Decades pass. Still no one can prove that one is better than the other in terms of pure practical application in the real world. People hold strong opinions, but the data collected and analysis performed doesn't support declaring a clear winner.
Now, here's the part where the thought comes in. Which conclusion makes the most sense?
1. A and B provide significantly different performance in terms of pure practical application in spite of the fact that no one has been able to prove it. The lack of ability to show a practical difference means nothing, we just know that one is significantly better than the other in spite of the lack of proof.
2. If A and B really do perform differently in terms of pure practical application, the difference can't be significant or it would not be so difficult to find the evidence. The lack of ability to show the difference in the real world after decades of trying is adequate evidence that the difference can't be significant.