Empowering organizations to solve attrition with AI

Reading Time: 5 minutes

Introduction

Employees who start and end their careers in a single business organization rarely come by. Employees often switch jobs after a few years of service in any given organization. Although the reasons may vary on a case-to-case basis, these switches could be either voluntary attrition, or organization-driven.

That being said, abrupt voluntary attrition is no longer a simple matter – it can seriously impact business growth KPIs. For example, frequent and unplanned hiring cycles are costly, and hamper operational productivity in businesses. Global HR teams have been looking for solutions for this phenomenon and have devised several programs and perks, exploring the potential of artificial intelligence and business intelligence to essentially retain employees. But it has not been adequate, or more accurately, precise in achieving its intended purpose.

Leveraging AI technology to understand employee attrition could be the way forward. Forecasting attrition trends could help organizations make meaningful changes in the present, enhancing retention in the future.

In this document, we will explore how enterprises can build resilient HR departments, capable of understanding and mitigating employee attrition issues.

At Fosfor, we have built Refract, a solution where we can understand, and accurately predict future churn events. An easy-to-use solution with visually consumable insights, business users can easily use Refract to understand the underlying attrition trends, and solve for future attrition possibilities, in the present.

But to do this, we had to first understand the causes of attrition or churn in any given organization.

While every employee’s reason for departing the organization may vary, there are several common factors that often contribute to employees’ voluntary churn. Employees who are planning to leave an organization are often looking for:

  • Higher compensation
  • Career advancement
  • Improved work-life balance
  • Better organizational culture and values
  • Better recognition in an organization
  • Easier commute or relocation

Although the organization’s culture, and lack of recognition for the employee play crucial roles in employee attrition, these factors are very difficult to be collected and quantified. So, in this case study, we will try to study drivers such as compensation, work-life balance, and commute, in correlation with demographics such as age, sex, etc.

An American healthcare service provider solved attrition issues with Fosfor’s Refract.

Leading American healthcare providers are facing serious personnel shortages issues due to rising attrition rates. Nursing staff, who play a pivotal role in delivering necessary healthcare to the patients at the grassroots level, is at the core of this phenomenon.

A few key challenges that organizations must overcome to mitigate churn:

  • Inability to accurately analyze the termination history data and respective reasons for attrition.
  • Not having an appropriate understanding of external factors like industry headwinds, global macro trends, industry sentiment data, and their possible impact on the HR data.

A lack of intelligence mechanisms to build and maintain data pipelines that can present the data in easy-to-consume formats. Inability to understand the history of events and data, and look into the future to know how to pivot.

The Refract Solution:

Refract provides users with an enterprise AI platform to track and consume the model performance. The data manipulation and modeling happen in the Snowflake environment, ensuring data security. The Streamlit applications are built and hosted on top of Refract, providing users with an interface to consume insights generated from the model directly through dynamic visualization widgets.

Key benefits of using Refract:

  • Powerful, flexible Data Science pipelines built through Snowpark for Python.
  • Quick visualizations of attrition facts & figures through Streamlit (quicker turnaround for everyday visualizations)
  • Interactive storytelling with a quick app-style deployment of notebooks

Solution Workflow

Decoding The Data to Insights Journey –

To understand and be able to predict employee attrition we have specific data points:

  • Employee demographics such as year of birth, sex, distance from the workplace, type of degree, tenure in the organization, and ethnicity along with their salary details.
  • Organizational details such as the type of hospital, the type of hospital ownership, and the US state the hospital belongs to.

Data for how much overtime each employee records. The churn variable, which indicates the employees who have churned.

Step 1: We have used Refract’s integrated Jupyter notebooks, which connect to the Snowflake instance, to start a new session. As Snowpark API requires Python 3.8, Refract also provides users with the flexibility to choose the relevant version of Python for use.

Step 2: We created a data pre-processing pipeline for simple manipulations such as missing value imputations, data scaling, and one-hot-encoding. We followed up with a model training pipeline consisting of a Random Forest algorithm and grid search for finding the best parameters. This entire code is written in a separate .py file in the form of Python functions, which is then sent to the Snowflake stage that you will be using. This training function is then registered as a stored procedure using Snowpark SQL.


Using the above code, you can send any file to your Snowflake stage.


This will register the model training pipelines as a stored procedure. Remember to mention the Python modules you will be using in your training pipeline, and if your pipeline has multiple functions, specify the main function as the handler of the SPROC (stored procedure(s)).

Using the above code, you can trigger the training stored procedure, while mentioning the variables that you don’t want to use in your training.

Step 3: Like step 2, you can create a separate file defining your prediction functions, sending that file to your Snowflake stage, and then registering the functions to Snowflake using the below code.

Here, we will be registering the prediction file as a Snowpark UDF (user-defined function), almost having the same imports and handler.

We can call the UDF and trigger the prediction on an entire Snowflake table(sdf).

Step 4: Once the model training and predictions are done, we bring brought the model back to Refract, along with all the necessary files for scoring the model and registered the model in Refract for monitoring and consumption.

Registering models on Fosfor Refract comes with a lot of benefits:

  • A visual representation of all the build time metrics.
  • Model’s decision making is explained.
  • Automatic API creation for basic consumption of the model.
  • Model monitoring can be scheduled to look out for any possible data drift.
  • Multiple versions can be deployed and compared.
  • Option to choose the compute power while deploying the model, giving you better control over resource utilization.

Step 5: Once the model is deployed on Refract, you will get the link to the API where you can provide the input payload and get the predictions. For small scale testing, on the other hand, you can directly consume the model in Refract itself.

Consume attrition insights with the HR Analytics app.

We have built and deployed a Streamlit-based app that gives the user a technical and visual understanding of the data, so that any stakeholder can understand the relationships between the variables, and understand the possible causes of churn among employees.

The data profile tab in the app gives you a snippet of the data, just so that you can understand the data you are working with, and below the data, you get the technical profile of all the variables present in the data, be it the distributions, cardinality, missing values, correlations, etc. This type of business intelligence profiling enables you to understand the data, decide what kind of pre-processing is needed for better modelling, and define the steps for pre-processing.

The Know your Data tab provides users with a visual representation of all the variables, with the appropriate graph for each data type.

In the above chart, you can select any variable and plot it against the salary data to understand the possible relationship between them. Here you can see that Acute Care – DOD hospitals on an average have lower salaries compared to other hospital types, which could be a reason for their high churn rate.

This app also enables the user to consume the trained model in a what-if analysis type of fashion, to predict the churn of an employee, given a certain parameter.

On the left-side pane, you can change the parameter and see how it affects the churn of any employee. Along with this, we get the import features, the confusion matrix, and the model parameters such as accuracy, precision, and recall.

All this information, presented in a single place, can enable any user to better understand the data and model to come up with possible reasons for employee churn. The what-if analysis helps users understand churn possibilities in the future for respective employee groups or individuals by tweaking the tenure and salary variables.

This app was built based on available HR data of the employees – information such as organizational reviews can also be used to better understand the culture of the organization, leading to a better understanding of employee attrition.

Want to learn more? Contact us to set up your free consultation today!

Author

Ayush Kumar Singh

Specialist – Data Scientist, Fosfor

Ayush Kumar Singh has 6+ years of experience in executing data driven solutions. He is proficient in Machine Learning and deep learning and is adept at identifying patterns and extracting valuable insights. He has a remarkable track record of delivering end-to-end Data Science projects.

More on the topic

Read more thought leadership from our team of experts

HR analytics made easy with Fosfor on Snowflake

Learn how Fosfor can help you to build a resilient human capital strategy for your organization by streamlining the MLOps journey.

Read more

Optimizing RCPG with data-enabled decisions

As we usher in the era of data-based decision making, the convergence of business acumen & technology is gradually reshaping data-first businesses into strategic powerhouses, opening a slew of possibilities.

Read more

Choosing the best AI/ML platform from a multimodel vendor

Artificial intelligence (AI) and machine learning (ML) technologies are expanding rapidly as organizations seek to capitalize on the value of their data. Half of the companies surveyed in a 2020 Mckinsey study have already adopted AI in at least one business function.

Read more
We use cookies to personalise content and ads, to provide social media features and to analyse our traffic. We also share information about your use of our site with our social media, advertising and analytics partners. View more
Cookies settings
Accept
Privacy & Cookie policy
Privacy & Cookies policy
Cookie name Active

What is a cookie?

A cookie is a small piece of data that a website asks your browser to store on your computer or mobile device. The cookie allows the website to “remember” your actions or preferences over time. On future visits, this data is then returned to that website to help identify you and your site preferences. Our websites and mobile sites use cookies to give you the best online experience. Most Internet browsers support cookies; however, users can set their browsers to decline certain types of cookies or specific cookies. Further, users can delete cookies at any time.

Why do we use cookies?

We use cookies to learn how you interact with our content and to improve your experience when visiting our website(s). For example, some cookies remember your language or preferences so that you do not have to repeatedly make these choices when you visit one of our websites.

What kind of cookies do we use?

We use the following categories of cookie:

Category 1: Strictly Necessary Cookies

Strictly necessary cookies are those that are essential for our sites to work in the way you have requested. Although many of our sites are open, that is, they do not require registration; we may use strictly necessary cookies to control access to some of our community sites, whitepapers or online events such as webinars; as well as to maintain your session during a single visit. These cookies will need to reset on your browser each time you register or log in to a gated area. If you block these cookies entirely, you may not be able to access gated areas. We may also offer you the choice of a persistent cookie to recognize you as you return to one of our gated sites. If you choose not to use this “remember me” function, you will simply need to log in each time you return.
Cookie Name Domain / Associated Domain / Third-Party Service Description Retention period
__cfduid Cloudflare Cookie associated with sites using CloudFlare, used to speed up page load times 1 Year
lidc linkedin.com his is a Microsoft MSN 1st party cookie that ensures the proper functioning of this website. 1 Day
PHPSESSID ltimindtree.com Cookies named PHPSESSID only contain a reference to a session stored on the web server When the browsing session ends
catAccCookies ltimindtree.com Cookie set by the UK cookie consent plugin to record that you accept the fact that the site uses cookies. 29 Days
AWSELB Used to distribute traffic to the website on several servers in order to optimise response times. 2437 Days
JSESSIONID linkedin.com Preserves users states across page requests. 334,416 Days
checkForPermission bidr.io Determines whether the visitor has accepted the cookie consent box. 1 Day
VISITOR_INFO1_LIVE Tries to estimate users bandwidth on the pages with integrated YouTube videos. 179 Days
.avia-table-1 td:nth-of-type(1):before { content: 'Cookie Name'; } .avia-table-1 td:nth-of-type(2):before { content: 'Domain / Associated Domain / Third-Party Service'; } .avia-table-1 td:nth-of-type(3):before { content: 'Description'; } .avia-table-1 td:nth-of-type(4):before { content: 'Retention period'; }

Category 2: Performance Cookies

Performance cookies, often called analytics cookies, collect data from visitors to our sites on a unique, but anonymous basis. The results are reported to us as aggregate numbers and trends. LTI allows third-parties to set performance cookies. We rely on reports to understand our audiences, and improve how our websites work. We use Google Analytics, a web analytics service provided by Google, Inc. (“Google”), which in turn uses performance cookies. Information generated by the cookies about your use of our website will be transmitted to and stored by Google on servers Worldwide. The IP-address, which your browser conveys within the scope of Google Analytics, will not be associated with any other data held by Google. You may refuse the use of cookies by selecting the appropriate settings on your browser. However, you have to note that if you do this, you may not be able to use the full functionality of our website. You can also opt-out from being tracked by Google Analytics from any future instances, by downloading and installing Google Analytics Opt-out Browser Add-on for your current web browser: https://tools.google.com/dlpage/gaoptout & cookiechoices.org and privacy.google.com/businesses
Cookie Name Domain / Associated Domain / Third-Party Service Description Retention period
_ga ltimindtree.com Used to identify unique users. Registers a unique ID that is used to generate statistical data on how the visitor uses the web site. 2 years
_gid ltimindtree.com This cookie name is asssociated with Google Universal Analytics. This appears to be a new cookie and as of Spring 2017 no information is available from Google. It appears to store and update a unique value for each page visited. 1 day
_gat ltimindtree.com Used by Google Analytics to throttle request rate 1 Day
.avia-table-2 td:nth-of-type(1):before { content: 'Cookie Name'; } .avia-table-2 td:nth-of-type(2):before { content: 'Domain / Associated Domain / Third-Party Service'; } .avia-table-2 td:nth-of-type(3):before { content: 'Description'; } .avia-table-2 td:nth-of-type(4):before { content: 'Retention period'; }

Category 3: Functionality Cookies

We may use site performance cookies to remember your preferences for operational settings on our websites, so as to save you the trouble to reset the preferences every time you visit. For example, the cookie may recognize optimum video streaming speeds, or volume settings, or the order in which you look at comments to a posting on one of our forums. These cookies do not identify you as an individual and we don’t associate the resulting information with a cookie that does.
Cookie Name Domain / Associated Domain / Third-Party Service Description Retention period
lang ads.linkedin.com Set by LinkedIn when a webpage contains an embedded “Follow us” panel. Preference cookies enable a website to remember information that changes the way the website behaves or looks, like your preferred language or the region that you are in. When the browsing session ends
lang linkedin.com In most cases it will likely be used to store language preferences, potentially to serve up content in the stored language. When the browsing session ends
YSC Registers a unique ID to keep statistics of what videos from Youtube the user has seen. 2,488,902 Days
.avia-table-3 td:nth-of-type(1):before { content: 'Cookie Name'; } .avia-table-3 td:nth-of-type(2):before { content: 'Domain / Associated Domain / Third-Party Service'; } .avia-table-3 td:nth-of-type(3):before { content: 'Description'; } .avia-table-3 td:nth-of-type(4):before { content: 'Retention period'; }

Category 4: Social Media Cookies

If you use social media or other third-party credentials to log in to our sites, then that other organization may set a cookie that allows that company to recognize you. The social media organization may use that cookie for its own purposes. The Social Media Organization may also show you ads and content from us when you visit its websites.

Ref links:

LinkedInhttps://www.linkedin.com/legal/privacy-policy Twitterhttps://gdpr.twitter.com/en.html & https://twitter.com/en/privacy & https://help.twitter.com/en/rules-and-policies/twitter-cookies Facebookhttps://www.facebook.com/business/gdpr Also, if you use a social media-sharing button or widget on one of our sites, the social network that created the button will record your action for its own purposes. Please read through each social media organization’s privacy and data protection policy to understand its use of its cookies and the tracking from our sites, and also how to control such cookies and buttons.

Category 5: Targeting/Advertising Cookies

We use tracking and targeting cookies, or ask other companies to do so on our behalf, to send you emails and show you online advertising, which meet your business and professional interests. If you have registered on our websites, we may send you emails, tailored to reflect the interests you have shown during your visits. We ask third-party advertising platforms and technology companies to show you our ads after you leave our sites (retargeting technology). This technology allows us to make our website services more interesting for you. Retargeting cookies are used to record anonymized movement patterns on a website. These patterns are used to tailor banner advertisements to your interests. The data used for retargeting is completely anonymous, and is only used for statistical analysis. No personal data is stored, and the use of the retargeting technology is subject to the applicable statutory data protection regulations. We also work with companies to reach people who have not visited our sites. These companies do not identify you as an individual, instead rely on a variety of other data to show you advertisements, for example, behavior across websites, information about individual devices, and, in some cases, IP addresses. Please refer below table to understand how these third-party websites collect and use information on our behalf and read more about their opt out options.
Cookie Name Domain / Associated Domain / Third-Party Service Description Retention period
BizoID ads.linkedin.com These cookies are used to deliver adverts more relevant to you and your interests 183 days
iuuid demandbase.com Used to measure the performance and optimization of Demandbase data and reporting 2 years
IDE doubleclick.net This cookie carries out information about how the end user uses the website and any advertising that the end user may have seen before visiting the said website. 2,903,481 Days
UserMatchHistory linkedin.com This cookie is used to track visitors so that more relevant ads can be presented based on the visitor’s preferences. 60,345 Days
bcookie linkedin.com This is a Microsoft MSN 1st party cookie for sharing the content of the website via social media. 2 years
__asc ltimindtree.com This cookie is used to collect information on consumer behavior, which is sent to Alexa Analytics. 1 Day
__auc ltimindtree.com This cookie is used to collect information on consumer behavior, which is sent to Alexa Analytics. 1 Year
_gcl_au ltimindtree.com Used by Google AdSense for experimenting with advertisement efficiency across websites using their services. 3 Months
bscookie linkedin.com Used by the social networking service, LinkedIn, for tracking the use of embedded services. 2 years
tempToken app.mirabelsmarketingmanager.com When the browsing session ends
ELOQUA eloqua.com Registers a unique ID that identifies the user’s device upon return visits. Used for auto -populating forms and to validate if a certain contact is registered to an email group . 2 Years
ELQSTATUS eloqua.com Used to auto -populate forms and validate if a given contact has subscribed to an email group. The cookies only set if the user allows tracking . 2 Years
IDE doubleclick.net Used by Google Double Click to register and report the website user’s actions after viewing clicking one of the advertiser’s ads with the purpose of measuring the efficiency of an ad and to present targeted ads to the user. 1 Year
NID google.com Registers a unique ID that identifies a returning user’s device. The ID is used for targeted ads. 6 Months
PREF youtube.com Registers a unique ID that is used by Google to keep statistics of how the visitor uses YouTube videos across different web sites. 8 months
test_cookie doubleclick.net This cookie is set by DoubleClick (which is owned by Google) to determine if the website visitor’s browser supports cookies. 1,073,201 Days
UserMatchHistory linkedin.com Used to track visitors on multiple websites, in order to present relevant advertisement based on the visitor’s preferences. 29 days
VISITOR_INFO1_LIVE youtube.com 179 days
.avia-table-4 td:nth-of-type(1):before { content: 'Cookie Name'; } .avia-table-4 td:nth-of-type(2):before { content: 'Domain / Associated Domain / Third-Party Service'; } .avia-table-4 td:nth-of-type(3):before { content: 'Description'; } .avia-table-4 td:nth-of-type(4):before { content: 'Retention period'; }
Third party companies Purpose Applicable Privacy/Cookie Policy Link
Alexa Show targeted, relevant advertisements https://www.oracle.com/legal/privacy/marketing-cloud-data-cloud-privacy-policy.html To opt out: http://www.bluekai.com/consumers.php#optout
Eloqua Personalized email based interactions https://www.oracle.com/legal/privacy/marketing-cloud-data-cloud-privacy-policy.html To opt out: https://www.oracle.com/marketingcloud/opt-status.html
CrazyEgg CrazyEgg provides visualization of visits to website. https://help.crazyegg.com/article/165-crazy-eggs-gdpr-readiness Opt Out: DAA: https://www.crazyegg.com/opt-out
DemandBase Show targeted, relevant advertisements https://www.demandbase.com/privacy-policy/ Opt out: DAA: http://www.aboutads.info/choices/
LinkedIn Show targeted, relevant advertisements and re-targeted advertisements to visitors of LTI websites https://www.linkedin.com/legal/privacy-policy Opt-out: https://www.linkedin.com/help/linkedin/answer/62931/manage-advertising-preferences
Google Show targeted, relevant advertisements and re-targeted advertisements to visitors of LTI websites https://policies.google.com/privacy Opt Out: https://adssettings.google.com/ NAI: http://optout.networkadvertising.org/ DAA: http://optout.aboutads.info/
Facebook Show targeted, relevant advertisements https://www.facebook.com/privacy/explanation Opt Out: https://www.facebook.com/help/568137493302217
Youtube Show targeted, relevant advertisements. Show embedded videos on LTI websites https://policies.google.com/privacy Opt Out: https://adssettings.google.com/ NAI: http://optout.networkadvertising.org/ DAA: http://optout.aboutads.info/
Twitter Show targeted, relevant advertisements and re-targeted advertisements to visitors of LTI websites https://twitter.com/en/privacy Opt out: https://twitter.com/personalization DAA: http://optout.aboutads.info/
. .avia-table tr {} .avia-table th, .flex_column .avia-table td { color: #343434; padding: 5px !important; border: 1px solid #ddd !important; } .avia-table th {background-color: #addeec;} .avia-table tr:nth-child(odd) td {background-color: #f1f1f1;}
Save settings
Cookies settings