There’s an old saying - “measure twice and cut once”. This feels like sensible advice which can help leaders and organizations avoid small or potentially catastrophic errors, but it’s not that simple. What you measure may be straightforward – it might be the share price, purchases, or profit, but ‘how’ you measure and compare it other things is where the devil creeps into the detail. Accurately measuring means always comparing so you can evaluate if you really got the outcome you desired. Even if you are only measuring one thing, you are still comparing it to something else, and this is where it gets complicated and risky.
This complexity and risk is well documented in concepts like ‘Goodhart’s Law’ - developed by economist Charles Goodhart - which states that “when a measure becomes a target, it ceases to be a good measure.” Once a metric or indicator is used as a basis for decision-making, it loses its effectiveness because it can be manipulated. Understanding and navigating the difference between ‘measuring’ and ‘evaluating’ is essential for businesses to truly know if their solutions are creating meaningful value.
Simple measurement can be deceptively complex
If you take a simple example like measuring your desk, then you might use a ruler. It seems like you are only making one measurement, as there is no second thing you are comparing it to. However, you are still comparing the measurement of the desk to either centimetres, inches, or some other measurement system. If you don’t know which one you’re using you might crash a spaceship into Mars if one team uses centimetres and another uses inches, as NASA once discovered, fortunately before any catastrophic consequences occurred.
This illustrates that mistakes can easily happen, even when highly intelligent people are using a standard system of measurement. Imagine how much riskier evaluation becomes if the comparison system itself changes – like currencies do with inflation, or populations do with births and deaths. The underlying system being used may itself change, making it a bit like measuring with a rubber band rather than a metal ruler.
Why evaluation trumps measurement
Not knowing these changes can radically alter a measurement from being valid to being invalid. For instance, if you earned $100 per hour ten years ago and still earn $110 an hour today you could say that you earn more but given inflation over the last decade that is not a valid statement – $110 today does not get you what $100 did ten years ago. Comparing only the monetary increase becomes invalid if the evaluation was actually “are you better off?” rather than the simple measurement of “do you earn more?” And it gets even more complex - what if the underlying comparison is to something that should naturally increase or decrease. At seventy years old, being able to run as fast as you could in your twenties is very impressive. The underlying measurement indicates a decrease in running speed is expected, and the underlying comparison is opposite to money and inflation.
So is this really an issue? How common are these types of inconsistent and unreliable measures in business? If we look at just one of the most common measures in business such as percentage increases in sales volumes, it really tells us nothing about the expected change or bias of the underlying system. A 15% increase might sound impressive, as well as objective, data driven, and valid. However, if the market is growing at 20% then the 15% increase is disappointing. However, there are plenty of sales teams living in their own little bubble celebrating a 15% increase when their competitors are growing at 20% or more. This is why measurement is easy, but evaluation is difficult. You need to measure the right thing, in the right way, comparing it to the right underlying system. You need to know what type of outcome measure you should be using - revenue, growth relative to market, or something else.
Leaders must focus on measures that matter
Psychosocial risk factors such as ambiguity and conformity are significant impediments to effective evaluation. If there is ambiguity on what’s being measured, how it’s being measured, or what counts as success or failure, then it is nearly impossible to provide an accurate evaluation. Similarly, if stakeholders just agree it worked, or agree it’s okay when it’s not, then the evaluation process becomes meaningless and may cause more problems if nobody knows whether success was achieved or further improvements are required. Businesses must ensure leaders responsible for evaluating outcomes have the right personality to do it effectively. Research by TGS Leadership shows that a sincere and unconforming ‘steady statistician’ is the ideal personality profile for this critical business function, however most organizations have no way of assessing a leader’s innate ability in this area.
Getting evaluation right is essential primarily because getting it wrong has compounding negative consequences. Evaluation guides the next iteration of the problem-solving process. Businesses need to know if the original solution to the problem that needs more work, or has the original problem been solved and now it’s about scaling the solution? If the wrong measures and tools have been chosen to evaluate the process and the output is meaningless, then the next iteration is likely to just compound the meaninglessness. This is why businesses need robust evaluation processes, not just measurements.
To overcome the complexity of evaluation and gain a genuine competitive advantage, organizations must identify leaders who are the ‘steady statisticians’ with a natural ability to effectively evaluate business outcomes. To find out more and assess the evaluation potential of your leaders and their impact on the psychosocial safety of their teams, check out the science behind The GreyScale Leadership Assessment tool and find helpful resources at http://tgsleadership.com
