Reading Time: 6 minutesIndustry solution powered by the Lumin-Snowflake partnership
Decision Intelligence is one of the most sought-after AI activations by enterprises for the unprecedented advantage it offers businesses. Although there are quite a few decision intelligence solutions in the market, Lumin presents a list of unique attributes that make it stand out from the competition. As a Snowflake-ready technology, Lumin has native connectors to consume Snowflake datasets directly. Lumin also leverages Snowpark for its core AI/ML computations, and integrates with Streamlit, offering close-knit secure solutions right within the Snowflake Data Cloud environment.
In fact, Gartner has recognized Lumin as a Representative Vendor in the Gartner Market Guide for Augmented Analytics 2022, and this positions Lumin as a go-to tool for enterprises to traverse their data-to-decisions journey with ease.
In this blog, we talk about an industry solution for the healthcare sector, combining Lumin as a Decision Intelligence (DI) product, Refract by Fosfor as a Machine Learning (ML) platform, and Snowflake as a Data and Compute Warehouse. The solution addresses the prominent issue of staff attrition, which is a major concern for service providers in the healthcare industry.
The challenge
The US Healthcare system has a severe shortage of registered nurses, and since the pandemic, the problem has only further aggravated. The US Bureau of Labor Statistics projects 194,500 average annual openings for registered nurses between 2020 and 2030, with employment projected to grow at 9%.
There are many reasons for this shortage, such as an aging workforce, inadequate workforce support, and an increased demand for healthcare services. Another prominent reason, which can be attributed to local as well as social factors, is Staff Attrition. Understanding the challenges and factors at play, and retaining the staff has become a matter of paramount importance for service providers. Industry experts see these challenges to present multi-faced implications if not addressed immediately.
Also, since hiring RNs, especially as contractual staff, is expensive, the situation can trigger a perpetual cycle of negative effects as listed below:
- Mounting financial burden as service providers fill open positions with contractual staff.
- Overburdening of current staff, and a consequent possibility of increased attrition rates.
- A negative impact on patient care and customer service.
The Lumin + Snowflake solution
Looking at the historical trend of attrition among nurses, it has emerged as a critical problem that needs to be addressed with speed. Since nurse attrition could be occurring due to multiple reasons, it’s important to zoom in closer to understand not just the probability, but also the actual cause of attrition.
We aggregated sample data from a 50-year-old historical reference (The raw data is extracted from the Snowflake Marketplace, and is provided by Revelio Labs) at the nurse level, capturing specific behaviors that led to attrition. Looking at historical trends, it is important to note that the attrition rates for nurses have increased in the last few years. This blog addresses both the what & why of this phenomenon.
The dataset has been viewed through 2 lenses: one on an overall nurse basis, and the other on monthly churn behavior.
The study attempts to understand the association of nurse behavior with churn probability. Hence, we predict the data at 2 levels:
- Find out who is expected to leave, and
- Find out how overall attrition can be curbed, and retention increased.
Prediction of nurse attrition
Refract by Fosfor, an end-to-end ML platform, has been utilized in this case to predict nurse attrition. We first analyzed the raw data, and then trained a Random Forest ML model for the predictions. This model was trained and tested to accurately predict which nurses were most likely to leave. The raw data was consumed by Refract, and a predictive model was created by utilizing nurse-level attributes as shown in Figure 1 (Solution Architecture) with Refract. The data wrangling, feature engineering, model training, and predicting were performed on Snowpark using Python libraries within Refract. The predictions were then consumed in Lumin and analyzed from different perspectives.
The data was engineered at monthly intervals to understand monthly attrition, and drivers like salary, overtime hours, etc., were measured. This data engineering was done on Snowpark, and the data was saved in Snowflake (as shown in Figure 1 – Solution Architecture) utilizing Snowpark, which was consumed by Lumin for further analysis.
Then, the attrition rates were studied at multiple levels such as distance travelled by staff, type of hospital, etc., and the importance of these variables on attrition were also recorded.
Figure 1 below shows the high-level architecture of Fosfor products working together to provide a tightly coupled comprehensive DSML solution.
Figure 1: Solution Architecture
Breaking down the data
About the data
The data has local as well as social attributes at an individual staff level. It contains the below information:
- About the Employer: Company, hospital type, ownership of the hospital, role of staff, etc.
- Employment details of staff: Employee ID, joining date, salary, job end date, tenure, seniority, overtime hours, etc.
- Social parameters: Educational Degree, school end date, age, ethnicity, sex, distance from work, etc.
As seen in Figure 1, the solution has two facets of analyses – Descriptive and Prescriptive.
For the descriptive analysis, the data is aggregated at a monthly level to understand the dynamics of churn and the trend of attrition. Major metrics such as Active Staff and Staff churned in a particular month are recorded.
For the prescriptive analysis, the output of the ML model trained on Snowpark is ingested and stored in the Snowflake ML schema. It consists of the additional predicted variable of churn prediction as well.
Key performance metrics
A few KPIs have been calculated to gauge the pattern of attrition and compare them across different hospital types, companies, and social parameters.
Attrition and Retention Rate: Although they represent the same data in different perspectives, the goals of the business made it imperative to see both the data points. Also, the distribution of data stipulated that it would make sense to look at retention numbers in alignment with business goals. With Lumin’s forecast capabilities, figuring out the trend of retention rates was simple.
Predicted Rates: Predicted churn percentage is an important factor in making strategic decisions w.r.t a company or hospital facility at a macro level.
The best way to understand the solution and get an understanding of the business use case is to go through a sample analysis.
Solution walkthrough
Descriptive Analytics to understand the attrition behavior
Let’s start with some simple questions that you may want to ask.
Being a service provider administrator, you would like to understand how attrition rates have trended historically. You can ask a descriptive question to Lumin about the trend of churn across the industry.
What is the trend of growth in attrition?
As depicted in Figure 2, Lumin precisely decodes the intent and brings up the column trend chart with easy-to-read variance indicators. The trend shows a strong negative drop of 21% in June 2022. Now let’s understand which type of institutions contributed to this change, and by how much.
Figure 2 is illustrative of Lumin’s capabilities in offering clear and directive insights, with the help of GPT technology, for specific descriptive questions.
Figure 2 : Growth trend of Churn
Contribution to change of growth of attrition by hospital type in June 2022
As shown in Figure 3, we can see that acute care hospitals contribute the most towards attrition. Let’s focus on acute care hospitals and move on to some deep-dive questions.
Figure 3 shows the break-up of the data in the form of a waterfall chart for the specific question.
Figure 3 :Contribution by different hospital types to June 22 drop
Going beyond the what and understanding the why: Unleashing Lumin + Snowpark
Diagnostics has always been Lumin’s strong suit, and with Snowpark, diagnostics is fast and seamless. The transfer of the analytics workload to Snowpark makes heavy diagnostics involving several data profiling and modeling relatively fast. The combination of Lumin and Snowflake dishes out compute-intensive deep dive results within seconds.
As we look at how the numbers look across Hospital types, and you would notice an inflection in the data around June 2022. Now let’s ask Lumin why the inflection happened. Picking up from previous insights, we already know that acute care hospitals were major contributors to churn rates; hence, you would like to know –
Why has churn reduced in June 2022 in acute care hospitals?
The comprehensive analysis shown in Figure 4, gives you actionable insights in terms of key focus areas. Voluntary non-profit, privately owned hospitals contribute majorly to the drop, while we see a drop of 40% in the case of female nurses. Also, Lumin illustrates that key drivers of this change are led by a drop in overtime hours, and active population. It directly tells acute care hospitals where to focus their attention, and what drives employee retention.
Figure 4 illustrates the reasons for the drop in churn rates in June 2022.
Figure 4 : Diagnosing acute care hospitals’ attrition in June 22
Understanding the predictions and developing a retention strategy
The next logical step of the analysis is to understand what the custom ML model prediction looks like. In line with previous findings of churn across hospital types, you can simply ask a question about how prediction looks across hospital types.
To have an unbiased view, you can ask for both the predicted count and percentage metrics in the same question.
What is the Predicted Churn Percentage and Predicted Churn by hospital type?
