Acknowledging that the available evidence may be limited and not representative of the complete evidence

It's good to use evidence to make judgments, but there's a common mistake in the way this is done. This is to assume that the available evidence is sufficient for making a judgment. It may well be sufficient, but the mistake is to assume that it is.

The limitations of the available evidence need to be assessed. Until relatively-recently, we didn't have telescopes capable of detecting exoplanets (planets orbiting other stars). Then we started to get that ability but initially could only detect very large exoplanets. Back then, it would have have been a mistake to look at the available evidence and make the judgment that no exoplanets exist, or that only very large exoplanets do.

We can ask questions like: what are the limitations of our ability to gather the relevant evidence? Has all the relevant evidence been collected, keeping in mind that resources are required to gather evidence (time and money, and people who have prioritised gathering it)? Is it possible that we could obtain more evidence in the future that could support a different judgment? Do we have good reason to believe that the evidence we have gathered so far is representative of all the possible cases?

The mistake is in assuming that a judgment being consistent with the available evidence is evidence that the judgment is correct.

So not only do we need to do more than just consider the nature of the available evidence (like its limitations). We need to consider what different judgments could be consistent with the available evidence. And we need to consider what the different possibilities could be for the missing evidence, and what judgments could be consistent with each of those different possibilities. 

In the exoplanet example, back when we didn't have the technology for detecting them, we needed to consider that the missing evidence could include exoplanets. And, obviously, that the judgment "exoplanets exist" is consistent with that possibility.

We need to be honest with ourselves if this consideration of different possibilities for the missing evidence (and the judgments they would support) shows that that are multiple different (and incompatible) judgments that could be made, like both "exoplanets don't exist" and "exoplanets exist". Being honest with ourselves means admitting it when we don't currently know the answer, and holding an agnostic attitude on this matter. It means not jumping to conclusions.

It can be difficult to do this. Having an agnostic attitude to something doesn't tend to be encouraged in our society. And we tend to have a bias towards information that we have, and against potential information that we don't yet have. We tend to privilege the information we have. It's tangible, whereas the other possibilities are not.

This bias can be subtle, such as if we take the available evidence -- from which we are prematurely forming judgments -- as evidence against possibilities that are incompatible with those judgments. If the available evidence does not include exoplanets, we may take this as evidence against the possibility that further evidence will include exoplanets.

We can't use the available evidence in this fashion. The fact that certain information is available, while certain other information is not currently available, (in general) tells us nothing of how representative the available information is of the unavailable information. What information is currently available is simply an artifact of our current abilities to gather relevant information.

When I talk of 'judgments' in the above, these are often judgments about what explains some evidence. There is one type of explanation that is routinely overlooked. When there is some apparent phenomenon in the outside world, we tend to assume that the explanation of it simply exists in the outside world. But a component of the explanation may be due to the nature of ourselves and our perception.

For example, the coloured bow-shape that we see (and the position we see it at) when we perceive a rainbow is due to the character of the light that reaches the position where we are. The actual explanation has to do with light rays interacting with water droplets over a larger area than where we see the rainbow, and the light there being scattered in a large range of directions -- and thus there is light reaching many different positions, at which different rainbows would be perceived.

There can be details that we think exist in the world but which are in fact phenomena that our brains have subconsciously invented to explain other phenomena in the world. Belief in ghosts, for example. The point is that we may think we have evidence for them, we may think that we can perceive them, and not realise that actually they're kinds of explanations for other details that our brains came up with, without us clearly understanding that this is what has happened.

I would argue (and will argue, in my PhD) that belief in an intangible 'information' is an example of such a phenomena, where part of the explanation is to do with an explanation our brains construct.

When we consider what judgments (including explanations) are supported by some evidence, we need to also consider that the apparent evidence and explanation of it may have to do with the character of ourselves (such as our perception). Doing so may expand the set of possible explanations that could be consistent with the evidence.