An implemented all-singing, all-dancing predictive model doesn’t automatically improve business operations or results. Sometimes it’s barely used. Why not? Based on many years of experience and academic study, Tom Fletcher shares his perspective on why this happens, and presents considerations for optimally deploying model development resources and gaining the full benefits of a new model.
When it comes to large-scale technical interventions, such as the implementation of a predictive model, pessimism is often encountered. Studies, opinions, and observations abound and are consistent in suggesting that less than 1 in 6 such projects reach their stated finish line. While technical reasons for this exist (data limitations, systems readiness, timing), more often the issues lie in the more mundane – people issues. With careful attention to the psychology of large-scale interventions and the impact on those involved, one can increase the likelihood of success. A professional practitioner should be able to gauge the environment, understand the people, and ultimately understand when an opportunity for large-scale technical intervention exists.
Choose wisely to increase your chance of success. What makes a good candidate for an analytic product (or other technical intervention)? In my view there are at least three primary considerations. The intervention should:
If any, or all, of these considerations are not met, then the timing is not right and success is unlikely.
The business problems that statistical models often impact the most include one or more of the following: creating consistency, driving efficiency, making better (risk) decisions, and/or seeking opportunities (expand the bar below for examples). These problems may not be clear at the outset of a project but should be stated early enough that a clear line of sight to success is articulated.
If a team has a high variance in skills, i.e., only a few individuals have the necessary specialized skills, this can lead to inefficiencies, lower overall performance, and poorer customer experience. Analytic models can help to equalize staff performance and reduce these variances.
Likewise, there are times when a process (often manual) relies on copious amounts of data/information to arrive at decisions. To the extent that this information can be reduced to something more manageable – with analytic models this is termed the model score – then the process itself can be made more efficient.
Similarly, when a highly multivariate decision is being made, i.e., a decision with dozens of inputs, a mechanical solution can often result in a better ultimate risk decision compared to a manual one – or at the least, to a more consistent decision.
Finally, opportunities can emerge from model outputs, e.g., from the ranking of (underwriting or claims) cases, identification of outliers, or from other positive output signals that can then be pursued.
An organization must also be open to change, and that positive momentum must be maintained long enough for success. For this, an individual is ideally identified to serve as the champion for change, seeing the project through from start to finish, helping to develop openness to change (a transformational leader) and keeping up the necessary change momentum. This individual will ideally understand the business problem and how the model will solve that problem, and be familiar with the outputs, workflow implications and impact on those affected.
Finally, one should make note of the relative size of the business problem to the company and impact to be delivered. Often, small segments of a business do not resonate with the larger organization and smallish problems don’t garner the implementation resources necessary to see success (small distractions don’t get traction).
Figure 1: Considerations for successful model implementation. The likelihood of success can be increased by picking a winner from the start and thorough planning that includes careful attention to the psychology of large-scale interventions and impact on the involved individuals. Source: PartnerRe.
A feasibility assessment should be conducted prior to any lengthy model development. Often, well-intentioned projects begin with data for a one-time model build, but sufficient data aren’t in fact available for longer-term application.
Check whether the analytics team has essential commitments from the relevant business unit in terms of SMEs and/or other resources (e.g., data access and explanation of data).
Change rarely occurs without significant effort. When considering a large-scale intervention, such as a model implementation, considerable planning is required because the status quo and current equilibrium must be challenged.
With respect to planning for change, there are several key considerations. What is the core content of the change? What people, processes, systems, etc. will be impacted? What is the change management strategy to be implemented? That is, careful attention should be given to the process of change.
Considerations in the process include: the content, the role of the change champion, inclusivity of key stakeholders (there should be at least one key individual who will ultimately be using the model or leading a team that will be using the model, who is open to collaboration with the model implementer, i.e., shows signs of active interest and engagement in a potential change), and planned communications (of which there should be many) including timing and proper storytelling.
Want trumps need.
However, most critical to change management are the identification and addressing of resistance and ensuring change readiness. The most well-intended interventions will fail if the organization is not made ready for the change. Needing a model (i.e., a clear goal exists) isn’t enough. The users of a model also need to want the model. They must have become dissatisfied with a current state and see a vision for something better. This can be achieved without overly negative commentary. Transformational leaders have mastered this aspect of change. Ultimately, once the individuals know that there is a better path and/or there is a critical mass that exceeds any resistance, then change is possible. If not, more time is needed, or a change champion is required to help “get them there”.
Strategy is important, but ignoring culture is tantamount to failure. The “best laid plans of mice and men often go astray”, is as relevant today in the hyper-modern tech world as when it was written many years ago. Careful consideration must be given to the underlying assumptions that people hold, the values and beliefs that they share and the normal way of behaving. Introduction of a new work design implies change, and that impact should be acknowledged.
The impact on the motivation of those affected is perhaps the most critical and most often overlooked factor. Careful consideration should be given to the workflow changes. While there are many well established theories of motivation to draw upon, a few key concepts are worthy of highlighting.
Models that create efficiencies can also impact the design of work in unintended ways. Autonomy, a fundamental psychological need, could be threatened. The new work design may limit skill variety or otherwise limit challenges. The sense of joy and focus one obtains in work – “flow” – could be hampered. Consider, for example, the potential for demotivation if medical underwriters are left mainly with applications that are ultimately declined, or if claims personnel are left mainly with claims where the individual is unlikely to recover. In general, models should supplement performance by reducing monotony and enabling greater attention to complexity and challenge (not the reverse).
Does the new work design:
No business or model stands still. An understanding of and plan for future model monitoring and a consensual understanding of model recalibration are essential. Also critical, however, is that recalibration should only take place when material changes in the data or business practice have occurred – viewing data too soon may result in unnecessary overreactions.
Much of the pessimism surrounding real experiences in implementing technological interventions such as predictive models, can be attributed to limited attention to people issues. Picking a winner is a good first step. Strive to recognize limitations early on and ameliorate those or patiently wait for the right opportunity. Plan for change effectively, but don’t discount the culture in which the change is to take place. There will always be resistance, so look for it and be prepared with a change champion or transformational leadership. Finally, recognize that while intentions may be valid, some change undermines the motivation of those who are intended to benefit – don’t reduce motivation in a quest to be efficient, but craft careful discussions with those impacted.
Tom Fletcher, PhD., VP, Data Analytics, North America Life;
When working with our clients on developing new predictive models, we also assist with change management. The one cannot, indeed should not, exist without the other. This is a specialist field, but one in which we have extensive experience. Please contact us to find out how we can help your business to improve its operations through effective model development and implementation.