Brass in Pocket – It’s not what you didn’t train, it’s what you did train.

A few years ago we were asked by an established financial institution to run a Root Cause Analysis to help them uncover why some of their senior team had “gone rogue” and behaved in a manner that was in direct contradiction to the values of the company and the extensive guidance and support they’d been given.  The firm were keen to uncover the gaps in their training which allowed such damaging behaviour to occur.  On completion of the investigation the conclusions of the RCA shocked them.  The problem wasn’t what their staff hadn’t been trained to do, it was what they hadbeen trained to do.

In their outstanding 2016 book Algorithms to Live By: The Computer Science of Human Decisions, Brian Christian and Tom Griffiths explore the effect of ‘Overfitting’ – a major cause of poor performance in machine learning.  In the workplace, many managers will already be familiar with over-fitting in nature, if not in name.  A typical occupational example would be staff focusing their behaviour solely on the established KPI’s, incentive structures or real-time analytics upon which they are measured, completely losing sight of the underlying reasons the measures exist in the first place, and the overall company goals – in practice a form of data idolatry.

Popular examples we might share in the training environment could be police officers picking easy cases over important ones at end of month, in order to hit case-closure targets. Or factory bosses obsessing on production metrics and ignoring maintenance issues, thereby storing up catastrophic failures for the future.  Furthermore, with this kind of narrow focus, it becomes increasingly difficult to know who the star performers in any business are, versus someone who is just really, really great at matching their activity to departmental KPIs.  The truth is, it is much harder to create incentives that don’t have some form of negative alter-ego than almost any organisations acknowledge.

As Steve Jobs said “Incentive structures work.  So, you have to be very careful of what you incent people to do, because various incentive structures create all sorts of consequences that you really can’t anticipate”.

This over adherence to incentives, rules or training, can, in some circumstances result in much more than just damage to performance figures or a balance sheet. In some cases, it’s the difference between life and death. In their book, Christian and Griffiths use the example of police and military ‘Line of Fire’ drills as a sobering illustration of what can happen when the lines between models and real life become overly blurred;

‘There are stories of Police officers who find themselves, for instance, taking time out during a gun fight to put their spent casings in their pockets – good etiquette on a firing range.  As retired Army Ranger and West Point psychology professor Dave Grossman writes, “After the smoke had settled in many real gunfights, officers were shocked to find empty brass in their pockets with no memory of how it got there.  On several occasions, dead cops were found with brass in their hands, dying in the middle of an administrative procedure that had been drummed into them.”

Similarly, the book reports that the FBI had to change their training after agents were seen reflexively firing just two shots and then holstering their weapons – because that was the standard action taught in their training – regardless of whether their shots had actually hit the targets and whether or not there was still a threat.  Known as “Training Scars”, they are very difficult to overcome, particularly in periods of high stress.  One dramatic case recounts the story of an agent instinctively removing a firearm from the hands of an assailant before instinctively handing the weapon back! Just as he had done with his trainer countless times before in practice.

So, what then of our financial institution and their rogue traders? It was revealed by the Sologic RCA investigation that they had also trained their team to put brass in their pockets; but in this instance ‘brass’ as the slang term for money.  For almost a decade the firm had carefully refined their recruitment policy, their incentives and their KPI’s to reward increasingly self-centred and ruthless behaviour.  Increment by increment the activities that were measured and rewarded were drifting, unnoticed, further from the company’s wider values.  As the firm grew the hidden impact and influence of this behaviour eventually revealed itself, and almost destroyed the company in the process.  Rewarded and encouraged every step of the way!

We can leave the final words of warning to US tech entrepreneur Sam Altman, when he tells us; “It really is true, the company will build whatever the CEO decides to measure”.