Have you ever been in a situation when everything goes as expected and you think it is too good to be true? Such situations happen in science experiments too, when researchers receive overwhelming evidence for their hypothesis. However, now scientists at the University of Adelaide have performed a mathematical study to see if popular expression “if it sounds too good to be true, it probably is” is true in science, and found that overwhelming evidence may actually be a bad thing.
Getting good results from the scientific research is undoubtedly satisfying. However, now scientists say that overwhelming evidence can actually be a disadvantage for the study. It is said that without a dissenting opinion it can indicate lack of credibility of a case, or even point to a failure of the system.
Scientists remember a similar example from an ancient Jewish law. It said that if all judges unanimously handed down a guilty verdict suspect cannot be convicted of a capital crime. It was considered that system itself can have errors, if there is such a strong agreement about the case. Apparently, this is how scientific study should also be treated.
It is quite interesting how scientists now came up with this idea. They put three different scenarios to the test based on mathematical probability. One scenario is about using witnesses to confirm the identity of a criminal suspect, second – accurate identification of an archaeological find, third – the reliability of a cryptographic system.
Mathematical analysis showed that there is a point at which too strong evidence actually makes the case less credible and it weakens confidence in the results. Professor Derek Abbott, one of the authors of the study, explained: “In our first example, we imagine there are 13 witnesses who all confidently identify a criminal suspect after seeing the suspect briefly. But getting a large group of unanimous witnesses in these circumstances is unlikely, according to the laws of probability. It’s more likely the system itself is unreliable”.
In other words, the more overwhelming evidence is, the less confidence in the results. This is why scientists admire the case of ancient Jewish law. Although sophisticated tools of mathematical analysis were not available for the time, they had intuition that it is very rare for everyone to agree and that it may indicate a flaw in the system. This works for modern researches as well – overwhelming evidence lowers credibility of the results.