ID|EN
Chat di WhatsApp

Tag: HR analytics

  • An Employee Turnover Prediction Model with “85% Accuracy”: Why That Number Can Mislead, and What to Look At

    Imagine a vendor presenting to the HR team. “Our turnover prediction model has 85% accuracy.” The room nods. Sounds impressive, doesn’t it? Eighty-five out of a hundred predictions are right. The contract is nearly signed.

    Before that happens, ask one simple question: “What percentage of employees at our company actually resign each year?”

    If the answer is 15%, there is some unwelcome news. A model that always guesses “nobody will resign” will be right for 85% of employees without learning anything. Its accuracy is 85%. And that model is completely useless, because it never finds a single employee who is going to resign.

    This is one of the most common traps in machine learning projects for HR, and in other fields where the event is rare: accuracy can be high precisely when the model does not work. Let us take it apart so you can assess such claims critically.

    The confusion matrix: four possibilities

    Every binary prediction (will resign or will stay) has four possible outcomes:

    • True positive (TP): the model predicts resign, and the person does resign.
    • False positive (FP): the model predicts resign, but the person stays.
    • True negative (TN): the model predicts stay, and the person does stay.
    • False negative (FN): the model predicts stay, but the person resigns.

    From those four numbers we compute several measures:

    • Accuracy = (TP + TN) / all predictions. How often the model is right overall.
    • Precision = TP / (TP + FP). Of everyone predicted to resign, how many actually resigned.
    • Recall (sensitivity) = TP / (TP + FN). Of everyone who actually resigned, how many the model managed to find.

    An example with numbers

    Suppose a company has 1,000 employees, and 150 of them resign this year (15%).

    Model A: “nobody resigns”. Its predictions are right for the 850 who stay, and wrong for the 150 who resign.

    • Accuracy: 850/1,000 = 85%
    • Recall: 0/150 = 0%
    • Precision: undefined (it never predicts resignation)

    Model B: a model that really learns. It flags 140 employees as high risk. Of those 140, 90 do resign (TP = 90), and 50 stay (FP = 50). Of the 150 who resign, it misses 60 people (FN = 60). So TN = 850 − 50 = 800.

    • Accuracy: (90 + 800)/1,000 = 89%
    • Precision: 90/140 = 64%
    • Recall: 90/150 = 60%

    Notice: the accuracy of the two models is not far apart (85% and 89%), even though one is entirely useless and the other finds 60% of the employees who will resign. Accuracy hides the most important difference. That is why, when someone offers a single accuracy figure, ask for other measures.

    Which measures should you ask for?

    It depends on the cost of errors:

    • If losing a key employee is expensive and you want to find as many at-risk people as possible, recall matters. The consequence is that precision may drop (more false alarms).
    • If intervention is expensive (for example an individual retention program with a high cost), precision matters so resources are not wasted on people who were not planning to leave.
    • For a picture of the balance: the F1-score (the harmonic mean of precision and recall) and AUC-ROC (the model’s ability to distinguish those who resign from those who stay across different thresholds) are often reported.

    And one important question that is often forgotten: compared with what? Is the model better than a simple rule, for example “employees with under two years of tenure and no promotion in 18 months are high risk”? A complex model must prove its added value over a simple baseline.

    Five other traps in HR prediction projects

    1. Data leakage. A model that accidentally uses information that only becomes available after the event, for example “has submitted a resignation letter” or “final contract status”. Results on test data look extraordinary, but in the real world the model is useless because that information does not yet exist when the prediction is needed.

    2. Testing on the same data used for training. A model must be tested on data it has never seen, ideally data from a period after the training data, so it reflects the real situation.

    3. Unrepresentative data. A model trained on data from one division does not necessarily apply to another. Changing conditions (for example a shifting job market) also make models go stale quickly, so they need to be monitored and retrained.

    4. Ignoring explanation. A “high risk” prediction without reasons is hard to act on. HR needs to know what factors drive the risk (for example time since last promotion, workload, or commute distance), not just a score. Model interpretation techniques help here.

    5. Ethics and privacy. Predictions about individuals involve employees’ personal data. Ask: do employees know their data is being used? Does the model treat certain groups unfairly (for example based on age, gender, or family status)? Will the risk score be used to help, or to punish? A prediction model should be a tool to support employees and improve the work environment, not to surveil or push people out.

    From prediction to action

    A prediction is useless if no action follows. A healthy flow usually goes like this:

    1. Define the business problem: which group’s turnover costs the most?
    2. Prepare clean and relevant data, with attention to privacy.
    3. Build a simple model first as a baseline, then a more complex one if needed.
    4. Evaluate with measures that match the cost of errors, not just accuracy.
    5. Translate results into action: conversations with supervisors, workload adjustments, career paths, or development programs.
    6. Monitor and update: measure whether the intervention works, and update the model regularly.

    In the end, the goal of the model is not to produce a pretty score. Its goal is to help the company retain the people worth retaining in a fair and humane way.

    Seven questions for your vendor or internal team

    If you are the party buying or commissioning a prediction project, ask these seven questions: (1) What is the proportion of the predicted event in the data, and against what baseline is the model compared? (2) Which measures are used besides accuracy? (3) How was the data tested, and is there data from a different period for testing? (4) Which features are used, and could any leak or be unethical? (5) How are predictions explained to users, so HR knows the reasons? (6) How is the model monitored and updated? (7) What concrete action will be taken based on the predictions, and how will we measure whether the action works? If any question cannot be answered, there is homework to do.

    A pilot before full rollout

    An accurate prediction does not necessarily produce better retention. The only way to be sure is a controlled pilot: choose several groups of high-risk employees, apply an intervention to some of them (for example a career development conversation), and compare with those who do not receive the intervention, with random assignment if possible. If the resignation rate in the intervention group is meaningfully lower, we have evidence that the combination of prediction and action works. Without a test like that, we only have a pretty risk score and good intentions.

    Short Q&A

    Does a resignation prediction model violate employee privacy? It can, if done without a clear policy. Healthy practice includes a legal basis and consent in line with personal data protection rules, restricted access to individual scores, using as little data as possible, and transparency to employees about the purpose of use.

    How much data is needed to build a model like this? There is no single number. What matters is having enough resignation events for the model to learn from, not just the number of employees. For small companies, descriptive analysis and exit interviews are often more useful than a fragile predictive model.

    Interested in building analysis or prediction models for HR, sales, or your company’s operations, with honest evaluation? The datalytics.indo team can help with everything from framing the problem, preparing data, and modeling, to dashboards and action recommendations. In-house training in applied analytics and machine learning is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp

  • Employee Satisfaction Surveys That Often Go in the Wrong Direction: 6 Mistakes That Leave the Results Unused by Anyone

    Almost every company has done it: an employee satisfaction or engagement survey. The questionnaire is distributed, the results are compiled, a neat presentation is made, and then it is saved in a shared folder. A year later, the same survey is repeated. The score is up 0.1. Everyone smiles. Nothing changes.

    I have been involved in several surveys like this, including an engagement survey for a retail company with hundreds of employees, whose results were presented directly to the board of directors and the board of commissioners. From that experience, I learned that failed surveys rarely fail because of the statistical formulas. They fail because the direction was wrong from the start. Here are the six mistakes I run into most often.

    1. Chasing a score, not a decision

    The first question I ask clients is always the same: “Once these survey results are out, what decision will you make?” Often the answer is a few seconds of silence. A survey designed without a decision behind it will produce a single number (for example an engagement score of 4.12) that gives no clue about anything. Is that good? Is that bad? What should be done?

    Design the survey backwards. Start from the decisions that might be taken (for example a change in leave policy, better communication from supervisors, or a career development program), and only then derive the questions relevant to those decisions.

    2. Reporting only averages

    An average score of 4.12 out of 5 sounds good. But an average can hide two very different groups: some employees are very satisfied, others very dissatisfied. Their average looks “decent”, while the company is facing a serious problem in one group.

    Always look at the distribution and comparisons between groups: by division, tenure, job level, location. And do not forget to read the percentage of “dissatisfied” or “very dissatisfied” answers, not just the average. Organizational problems usually hide in the tail of the distribution.

    3. Not looking for drivers, only measuring levels

    Engagement surveys often stop at the question “how engaged are our employees?”. But the more valuable question is: “what influences engagement the most in our company?”

    This is where statistical analysis comes in. With regression or path analysis, we can see which factor has the strongest relationship with engagement: direct leadership, compensation, workload, growth opportunities, or communication. This is what separates a survey that merely “takes a snapshot” from one that “points the way”. If you only have the budget to fix two things this year, you need to know which two have the biggest impact.

    4. Questions that are biased, double-barreled, or too sweet

    Some examples that often slip through:

    • Double-barreled questions: “I am satisfied with my salary and benefits.” What if someone is satisfied with the salary but not the benefits? They have no honest answer.
    • Leading questions: “How much do you agree that your supervisor has supported you well?” That wording already implies the answer.
    • Vague terms: “I feel valued in this company.” Valued by whom? In what form?

    A good question is short, specific, one idea, and does not hint at the answer. Always pilot the questionnaire on ten to twenty people before wide distribution, and ask them which parts are confusing.

    5. Ignoring confidentiality and the fear effect

    An employee satisfaction survey is only useful if employees dare to be honest. Employees will only be honest if they are sure their answers cannot be traced back to them. Two bad habits that often damage this trust:

    • Reporting results for groups that are too small. If a team has only four people and results are reported per team, a supervisor can easily guess who answered what. Set a minimum number of respondents for every report.
    • Asking for excessive identifying data. The combination of division, position, age, and tenure can identify someone almost for certain, especially in small units.

    If employees are suspicious, they will answer safely, meaning high scores without meaning. This is where a survey produces beautiful but empty data.

    6. No follow-up, so the next survey gets quieter

    This is the most expensive mistake. Employees fill in the survey hoping something will change. When no visible follow-up appears, they learn one thing: this survey is a formality. Response rates fall, honesty falls, and the next survey produces data that is even less useful.

    A far healthier practice: announce the results (honestly, including the unpleasant ones), set a few realistic priorities, name an owner for each, and report progress before the next survey. When employees see one small thing change because of the survey, their willingness to answer honestly next time rises sharply.

    So what does a good survey look like?

    In short, a useful engagement survey has five traits:

    1. Designed from the decisions that will be made.
    2. Uses a tested questionnaire (valid and reliable), not just a list of questions that sound good.
    3. Analyzed by looking at distribution, group comparisons, and drivers, not just averages.
    4. Confidentiality is protected with a minimum reporting threshold.
    5. Followed by visible, communicated follow-up.

    When those five are in place, that 4.12 stops being just a number. It becomes the starting point of a real conversation between management and employees, and that conversation is what actually changes an organization.

    An example one-page report framework for the board

    Good survey results can be summarized on one page. Here is the framework I recommend: (1) three headline numbers with comparisons (overall score, participation rate, change from the previous survey), (2) the three biggest strengths and three areas for improvement, with numbers, (3) two to three main engagement drivers obtained from statistical analysis, (4) prioritized recommendations with owners and deadlines, and (5) one sentence on the limitations of the data. The following pages are appendices for those who want to dig deeper. A board that reads one clear page is more likely to act than a board that reads fifty slides.

    Do not forget open-ended questions

    Numbers give a picture of ‘how much’, but free comments give the ‘why’. One or two open-ended questions, for example ‘What is the one thing that most needs to improve in this company?’, often produce unexpected insights. Those comments can be grouped by theme (content analysis), then counted by frequency. A combination of numbers and stories is far more convincing to management than numbers alone. Make sure the comments reported are anonymized so they cannot be traced to individuals.

    Short Q&A

    How often should an engagement survey be run? Many companies run a comprehensive survey once a year, supplemented by a short pulse survey of a few questions each quarter to monitor specific issues. Too often makes employees weary, too rarely means problems are detected late. Most importantly, do not run a new survey before follow-up on the previous one is visible.

    Is a high score always good? Not always. A very high and uniform score across all divisions sometimes signals that employees are afraid to be honest or that the questions are too sweet. Compare it with behavioral data such as resignation rates, absenteeism, and training participation as a cross-check.

    Planning a customer satisfaction or employee engagement survey for your company or institution? The datalytics.indo team can help with everything from survey design, instrument testing, and sampling, through analysis (engagement drivers with regression or path analysis), to presenting recommendations to management. In-house training is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp