The Rankings Racket

Assembling and serving up index numbers – and the attendant rankings is an artifact of sorts; something that can be easily manipulated to serve the interests or hidden agenda of the index creators.

In my last post I highlighted (and despaired) about how Connecticut was firmly and perennially ensconced among the mediocre, if not embarrassingly, bringing up the rear – in the rankings contest among states.  This is, of course, (or should be) something of great concern to all of us. Yet, assembling and serving up index numbers – and the attendant rankings is an artifact of sorts; something that can be easily manipulated to serve the interests or hidden agenda of the index creators. 

This pliability is a well-known feature of rankings and much has been written on this.  Here – for example - are two recent papers where the authors cleverly question the relevance and applicability of the popular Business Climate Index (Fisher 2013; Kolko, Neumark and Mejia, 2013).  Succinctly, the authors impugn and impeach the resulting Business Climate Index Rankings alleging that the indexes are subjectively constructed, technically suspect, and assembled in a self-serving manner.  As a result, the authors contend, the rankings convey misleading or downright incorrect information at all.

What is to be done?  Disregard the indexes altogether?  No.  In fact, indexes and rankings are useful. And it is precisely “how useful” that is important to illustrate. By the way, and not surprisingly, their relevance is on the one hand exaggerated by those who stand to benefit from the results and just as vehemently disparaged by those who are afflicted with less flattering results.  First, let’s understand the nature of indexes.

To assemble a ranking a summary index needs to be constructed.  It is a “summary” figure by design because it aggregates (the sum being a common aggregate) the disparate pieces of information provided by the various constituent elements selected for the index.  And to construct an index – four things need to happen.

1.     We need to determine the variables that go in the index.  This exercise is known by many names: e.g. the reference class problem, the Shonubi problem, among others. And although there could be many dimensions in the construction of an index, for our purposes, let’s do this at two levels: attributes and geography. 

2.     First attributes.  If you are a businessman you will pick taxes, wages, regulatory barriers, education of the work force and so force.  If you are an inveterate green then you go for green-spaces, sidewalks, ethnic diversity, club scene, amenities and so forth.  Second is geography.  Rhode Island is smaller than the city of Houston.  And therefore comparing the “Business Environment” of Rhode Island to Texas borders on the silly.  One would think establishing a more equivalent geographic commonality is important if only to avoid elements of the geography itself enter surreptitiously into the calculus.  So for instance it’s conceivable that rents are more expensive in Rhode Island because it is smaller and has less available space than Houston (all else equal).

3.     The attributes selected are then ranked, re-scaled, standardized or otherwise recast; this is necessary because the index construction exercise essentially entails comparison of apples and oranges with no natural scale on which to compare them.

4.     The attributes or variables are then weighted  - presumably to reflect their “importance;”  so a generous index will include all the variables above but then weigh ethnic diversity and greenspaces less that tax burden and wage rates (for instance).  And even an “equal weighting” is a weighting; and even “no weighting” is a weighting.

5.     Last, the transformed or ranked variables are added or averaged. But there is no reason on couldn’t apply a different treatment: say add the variables, calculate the proportion of the total sum constituted by each variable, square this proportion, and last, to add them up. 

The result?  Not one; but many.  Which is an unfortunate resolution because it leads to the uninformative observation that Connecticut ranks anywhere from second (2) on the bright side, to forty-seventh (47) on the not so good side (cited by Kolko et al). 

So let’s do it ourselves.  Here are the results of is an index construction exercise (drawn from Fisher 2013 and expanded slightly to illustrate my comments); the actual calculation are at the bottom.

The first bar chart displays the variation in the rankings for three hypothetical States {A,B, & C}  based on an index developed using the following variables:

1.     Wage Premium (the average amount paid by the state above the average minimum wage: in %).

2.     The top corporate tax rate in the state.  (in %)

The resulting rankings are based on an index where the variables or attributes are treated as follows. (by the way all of these constructions are used in the “Index building business.”)

            Index 1 is constructed by adding the various attributes.

            Index 2: is constructed by adding the rankings based on each attribute

            Index 3: is constructed by adding the weighted rankings. In the first chart the weights are {3 & 1}; and in the second graph {3,1, 0.5, 0.5}

            Index 4: is constructed by adding the rescaled measures (by setting the highest value of each measure equal to 10).

            Index 5: is constructed by adding the standardized measures

The second bar chart has the variation in state rankings based on an index that has been enlarged to include the following additional variables/attributes.

3.     Diversity (in effect, minorities as a % of the population)

4.     Green Areas (as a percent of the total state area, in %)

Figure 1

The State Variation in State Rankings Based on a 2-attribute Index.

The Rankings Differ Because of Different Index Construction Methods.

Figure 2

The State Variation in State Rankings Based on a 4-attribute Index.

The Rankings Differ Because of Different Index Construction Methods.

As you can see one can make indexes say whatever you want them to say – with artful selection and manipulation of the inputs.  However, despite their pliability, not all the information in an index is useless; more on this on my next post.

Follows: