Benchmarking: welcome to the world of charts, symbols and rankings

We live in a world of information overload. On the one hand we need to distinguish between the true and the false. On the other hand we don’t want to spend too much time doing so.
 
In these circumstances we often turn to benchmarks for guidance. Benchmarks have become pervasive in modern life. However our dependence has not attracted much attention. It should.

What do benchmarks offer?

Benchmarks are summaries of information that are compiled by others in order to simplify the tasks of gathering information, making comparisons and to help us make the right choice.

For example, I like to listen to music but I don’t have time to check out all the new releases. So I go to a chart. I download what is on the top of that chart. The chart has been compiled by the music station, either on the basis of requests received, or on the basis of sponsorship from recording companies. The chart provides my benchmark for what is worth my while to add to my playlist.
 
Benchmarks offer three main advantages to users. First they gather together information, condense it and provide it in a convenient summary form. It is a world of acronyms, symbols, ratings, indexes, charts, rankings and league tables.
 
Secondly, they sort information and distinguish between what is similar and what belongs to another class. When we walk into a five star hotel we know to expect something better than when we walk into a no star fleapit.
 
Thirdly, they save us costs. We don’t have to spend time, effort or money on getting information and making lengthy comparisons ourselves. We are less likely to incur the costs of bad choices. We are less likely to have to face the costs of getting redress.
 
One area where there has been an explosion in the use of benchmarks is in financial markets in the form of indexes and Exchange Traded Funds (ETFs) that use them. All the advantages of benchmarks are there in spades. The indexes summarise information, classify and sort it and reduce costs for users.
 
Benchmarks: why should we care
 
Benchmarks can guide us in making rather trivial, everyday choices. They can also guide us in making important long term choices. We may use benchmarks for our own individual choices where bad choices may be of consequence only to ourselves. We may also use them for making social choices where there may be consequences, good or bad, for others.
 
Benchmarks can save us time and money. They can also be seriously misleading. They can be used to inform. They can also be used to manipulate. The choices they help us make may be easy to repair if they turn out badly. But sometimes the choices we make are not easy to remedy.
 
The variety of benchmarks
 
We make use of benchmarks in all kinds of situation where we want to get information quickly. For example, if we want to choose a hotel for a vacation, or a restaurant for an evening out, we pay attention to how many stars it has, and what overall ratings other users give it.
 
We also use benchmarks where we feel seriously under-informed and are uncertain about how to obtain relevant information. So we turn to health or school inspectors to establish performance ratings for schools, or hospitals, or medical practices. Similarly we may turn to rankings in league tables when considering a university to apply to.
 
We use benchmarks in further situations when we feel technically ill-equipped to take an informed view – so we rely on others to certify that a household appliance is reliable, or that a building is safe, or that a food is gluten free, or that an investment is ‘blue chip’.
 
The sources
 
People are prepared to pay for benchmarking. Those who pay directly may not be the end consumers. More likely it is the provider of a good or a service that is benchmarked, or a market intermediary that uses benchmarks. The willingness to pay means there is an incentive for the private sector to produce benchmarks. Most of the benchmarks that we use are privately produced.
 
Benchmarks are also produced by the public sector. This may be because the sector has privileged access to the information required. It may also be because the public sector is providing funding, or seeking funding, for the service being rated and has a special interest in, and responsibility for, the content of what is being benchmarked.
 
The incentive to multiply
 
Benchmarking can go wrong. There are three main sources of error.
 
The first source of error is that there is a huge incentive to produce benchmarks. Users want them. Providers get paid for them.
 
The multiplication of benchmarks means that we cannot take them at face value. We need additional information about their reliability. We may want to know the sources of information and who is paying whom for their production and use. We also probably want to know how far the benchmark is based on ‘facts’ such as the actual number of listeners or viewers, and how much on the judgment of the compilers (such as their own judgement that this is worth listening to). We want to know if the benchmarks could be skewed by bots.
 
We also need to be aware about the possibility of what is called ‘availability’ bias where compilers reach for what is easily produced or available in numerical form.
 
There is safety in numbers. If all benchmarks point in the same direction then the guidance they give is confirmed. But we still need to know that they are independently produced and not connected to the same source, or compiled in the same way.
 
The incentive to narrow
 
The second source of error is that there is an incentive to narrow the benchmarks. Producers want to differentiate their product. Users may want more precise summary information geared to their own particular wants and needs.
 
The narrowing of a benchmark can be useful. But it also contains additional risks for the user. One risk is in the loss of the broader comparative context. We may need to look at additional benchmarks tin order to supplement the information provided by our own narrower one.
 
Another risk is in composition. A narrower benchmark may be vulnerable to changes in very few components. For example, a decision (since reversed) by a single company (Unilever) to delist from London can have a major impact on market indexes and investors linked to those indexes.
 
The incentive to manipulate
 
The third source of error is that there is an incentive to manipulate benchmarks. Manipulation can take the form of collusion between the provider of a benchmark and an intermediary or actor who benefits from its use. The independence of the provider or compiler may also be compromised in other ways. For example, the compiler may want to show a particular result in order to foster a broader relationship or to reap a direct benefit.
 
Reliability Oversight & supervision
 
When we collect our own information in making a decision we conduct our own vigilance about the information we are gathering. We make our own choice about how much time and effort it is worth for us to get information and to check it. When we rely on benchmarks, we are relying on the vigilance of others in collecting and sorting the information. Our own need for vigilance switches to the benchmark itself.
 
Sometimes we will gain information on the reliability of a benchmark simply through experience and at little cost to ourselves. In other cases the experience can be costly. In general it is likely to be difficult for the ordinary user to find out how data is compiled, how reliable it is, and about who is paying for what.
 
In the case of financial markets it is professional investors who use benchmarks. They are qualified to make judgments. They are paid well to do so. Nevertheless, even in the case of professional, well informed users, it has been found necessary to have official, independent oversight of benchmarking.
 
Social benchmarking: rights
 
One area of life in which benchmarks are used extensively is in connection with social choices. The benchmarks are expressed in the form of social ‘rights’. These might include a right to unemployment benefits, to social security, to health protection, to housing, to family protection and so on.
 
Potentially such benchmarks offer the same advantages as other forms of benchmarking.
 
They help summarise relevant information by pinpointing the most relevant ethical element in a choice. They help to classify by stating that something belongs or not. For example they may help to identify eligible claimants. They reduce costs by reducing the need for prolonged fact gathering and political debate about what to do.
 
Errors in social benchmarking
 
Unfortunately benchmarks to help us make decisions in areas of social concern are vulnerable to the same defects as benchmarks in other areas. Political parties, lawyers, civil society activists and potential beneficiaries have an incentive to multiply them. They also have an incentive to narrow them so as to fit a particular social concern. In addition, they have an incentive to manipulate them. It is the general, unorganised public who carries the cost for paying the beneficiaries of benchmarking.
 
The oversight of social benchmarking
 
In theory it should be possible for societies to provide for independent oversight of the compilation and use of social benchmarks, such as a right to shelter, so as to ensure that they are not abused or exploited. In practice there is no easily identifiable independent source of oversight. Civil society, the law and politicians are all linked in their production and promotion.
 
There is not much cost for us when we make a mistake by relying on a chart to decide on a download, or a star rating for the choice of a hotel. There can however be costs for society in overreliance on social benchmarking. Multiplication and narrowing involve increasing claims on resources for which someone has to pay. Misclassification diminishes social trust. Manipulation is difficult to remedy.