A more trustworthy way of asserting which claims are supported by the evidence

This post is a preliminary attempt to express the following ideas.

We need evidence to determine whether a claim is true or not.

But in our society, claims are judged mostly not on evidence, but by who is making the claim.

Is the claim being made by an expert in the field? By a scientist? Or by some random person on the internet? Is the claim being made by a journalist? How experienced is this person? Are they a junior person, or a senior person, in their area? What is their reputation? Are they a nobel prize winner? Is the claim coming from a recognised organisation within that area (e.g. a nutritional claim from "The National Nutritional Board").

In principle, judging a claim based on the characteristics of who is making the claim (like whether they are a recognised expert on the topic) is using those characteristics as a proxy for the evidence for or against that claim. We assume that, for example, an expert knows the relevant evidence, and is able to form sound judgments on the basis of it.

However, judging claims based on who made them can be fairly unreliable.

Someone's status is not necessarily a good indicator of their knowledge or capability. One reason for this is that status tends to have a political element to it. And having more experience doesn't necessarily make someone a better judge of a topic. It might mean they are stuck in old ways of thinking, and unable or unwilling to see new ideas that challenge those ways.

What people or organisations claim is often influenced by political concerns. They aren't necessarily putting truth first.

If the science on the topic isn't settled, then different experts will have differing opinions, and it won't be reasonable to justify any particular claim based on the fact that a particular expert believes it's true. In such cases, there won't be evidence showing that a claim is true, so the 'who' (the expert) isn't a proxy for the evidence, but really a proxy for their opinion.

There can be mature fields containing beliefs that are agreed by pretty much everyone in that field, but where there isn't actually good evidence for the belief. It's just that everyone in the field is taking it for granted. (I believe that, historically, this has not been uncommon).

We may recognise the limitations of judging claims by who made them, but at the same time believe that doing so is our only practical option. No one has the time or ability to learn about every field and evaluate the evidence in it. We must defer to the experts.

While there is a lot of truth to that, there is more nuance to the matter than this. 

As explained above, there are many situations where there is no good evidence, so we can't really judge claims about them based on who made them.

Instead of feeling compelled to make a judgment in such cases, we should recognise that we usually do not need to make one. The only time we actually need to make a judgment is when we need to make some decision that requires making a judgment. If we're not in such a situation, we can remain agnostic about the issue.

And, I believe there's a better option that just deferring to what an expert claims.

The problems that 'judging claims by who made them' addresses is our lack of time and knowledge for evaluating claims. I want to suggest there is another way to address those problems.

Instead of just saying "an authority claims X", we can try to make the evidence and reasoning involved transparent, and in a fashion that opens-up the hierarchical structure of the evidence and reasoning behind the claims. We can include information about the vetting of the claims alongside the claims themselves. I'll get into the reason why I think this is a better approach.

Granted, it won't always be possible to articulate the evidence and reasoning, such as for knowledge that is based on experience. But there are plenty of areas where the evidence and reasoning can be made explicit, such as in science.

Isn't what I'm talking about already to be found in academic papers? No, I'm thinking about a very different way of presenting the information.

Central to what I'm thinking of is that the information should be organised around claims, and their hierarchical structure should be made explicit. The hierarchy of a claim's sub-claims is shown. Each sub-claim is itself treated like a claim in that it has its own page and we can see its hierarchy of sub-claims, as well as the reasoning and evidence supporting it, and information about its vetting by experts (such as pointing out problems with the evidence).

I am not talking about formalising knowledge, like using some sort of formal language/notation to specify the knowledge. I'm not talking about using some sort of mathematical means to evaluate the correctness of claims. I don't think that's possible (maybe one day, but not now). I am talking about expressing them in ordinary language. But, structuring it in a way that makes explicit the hierarchy of claims and sub-claims, and which makes explicit what are the sources of evidence for each.

These claim hierarchies should be made available in a publicly-accessible location, perhaps a website akin to Wikipedia but for claims.

Academic papers don't make the claim hierarchies explicit, and usually the evidence and reasoning for a particular claim will be distributed across several papers. And when viewing a paper you can't see a particular sub-claim and, alongside that, see information about the vetting of that claim.

By making the claim structure explicit like this, it "opens up" the elements of the justification in a way that facilitates more systematic vetting of claim structure, and providing information about its vetting.

Each claim/sub-claim is a thing that can be referenced. It has a name and a URL. It's not just something implicit within some paragraphs of text. It is a thing that can be drawn out and considered just on its own. That facilitates vetting each individual claim/sub-claim, and provides a place where information about its vetting (e.g. which experts have vetted it) can be stored. This information could also include details of any disagreements with those details.

Having the structure of the claim hierarchy being opened up like this would facilitate a more distributed process of vetting its details. It could be examined in a more piecemeal fashion. Someone could examine just a particular sub-claim. Different people could look at different sub-claims. Anyone could examine any sub-claim.

The problems that 'judging claims by who made them' addresses are the lack of time and knowledge for evaluating claims. 

I think the design I've laid out would have superior properties for this. Non-experts could look at just the top-level claim (without having to look into the details about the sub-claims), and see that has been vetted, and see any mention about issues within the sub-claim hierarchy. This information would be more trustworthy than the information that an expert has made the claim -- it would be a more reliable indicator of the truth.

Why would it be more trustworthy? Here are some reasons.

The details are more out in the open, presented in a way that's more suited for vetting, and the details of that vetting are out in the open, alongside the claims.

Because information about vetting (including any disagreements) is there as part of the claim hierarchy, there is better communication of this vetting information. It will ensure it can get more eyeballs.

Information about issues with any sub-claim can be propagated up the hierarchy of claims. So that, for example, if it's a major issue, it can be seen when viewing the top-level claim.

We can trust that the judgments reached on it (that are included along with the claim hierarchy) will be based more on the evidence and reasoning about the evidence. By having to explicitly break down a claim into a hierarchy of sub-claims, it forces more of a focus on evidence and reasoning about that evidence. And when the vetting that is made explicit is similarly broken down (addressing specific sub-claims independently of the other details), it again forces more of a focus upon the evidence and reasoning about that evidence. It will be more difficult to 'get away with' making claims that aren't grounded in evidence.

The above is just the basic picture of this idea. There would be many details to work out in practice. And though, for the reasons given above, I think such a system would likely be superior overall, that's not to say it wouldn't introduce its own kinds of issues. One last caveat is that I'm not trying to suggest that such a system would magically fix all issues to do with judging claims, I'm only suggesting that it would be an improvement over what we currently have.