Read Part I, which covered Evaluation Planning, and Part II, which covered Data Collection, prior to reading Part III of this post.
Case Study
Using the Phillips ROI Methodology™ to Evaluate the XYZ Widget Manufacturing Company Leadership Development Program
Phase 3: Data Analysis
In Phase 3, the data collected during phase 2 (discussed in Part II of this blog) was gathered, rolled up and analyzed to determine if there was an impact on the business of XYZ Widget and whether the ROI met the forecasted amount of 25%.
The following steps were taken during Phase 3 of the ROI study project:
- Isolate the effects of the solution
- Convert all data gathered to a monetary value
- Capture all costs related to the solution
- Calculate the ROI
- Identify intangible benefits
We will look at each of these steps in more detail in this post.
Isolate the effects of the solution
A control group was used as the isolation factor for this ROI study. As discussed in Part I, there were 5 high potential individuals aligned with the manufacturing function who went through the leadership training program, and 5 high potential individuals who did not (but would as part of a future group). Each high potential group was responsible for a production line in two manufacturing facilities.
While the training program was in progress, both manufacturing facilities installed new machinery. It was expected that the new machinery would increase production of widgets by 2%.
Convert all data gathered to a monetary value
Below is the outcome of the data collected and measured for the ROI study. For the purposes of this limited look at the ROI study, we focus below only on those five high potential employees who were in the manufacturing function.
|
LEVEL |
HOW MEASURED – DATA COLLECTED |
OUTCOME |
|
1 |
Satisfaction survey – survey done using an online surveying tool and was sent to all participants on the last date of the program | 8.5 on a 9 point scale |
|
2 |
Action plans created by 5 participants on the last date of the program | Completed in conjunction with managers; considered detailed and challenging |
| Multiple choice and essay exams | Range of 82% – 98% on all exams taken | |
| Classroom participation | Full participation by all participants observed | |
|
3 |
Progress toward completion of Action Plans | After 3 months, there was satisfactory progress on 2 action plans and 3 had made significant progress. |
| Survey to gauge impact on manufacturing functional area | A survey sent to executives, mentors, and participants at 3 months showed that there was significant impact made toward reaching the goal of improving processes and increasing production of widgets. |
Level 4 was the business impact of the leadership training program on a selected business process. This limited look at the ROI study included a desire to increase the production of widgets by 10% within a 6 month time period, which would be an increase of 1,150 more widgets produced per production line per day, or an increase from 11,500 widgets per day per production line to 12,650 widgets produced per day per production line. The increase was required in order to meet growing client demand for more widgets.
The control group was tasked with increasing production by 10% on one of the production lines and the high potentials who completed the training were tasked with increasing production by 10% on the other production line. The same amount of time was given to both groups and neither group started earlier than the other. Other than the training program and the mentoring program, all else was equal between the participant group and the control group.
Production reports were evaluated at 3 and 6 month intervals, and again at 9 and 12 months. Additionally, online surveys were sent to the functional areas (manufacturing, planning and procurement, and materials management) to determine if there was an increase in communication and teamwork between the three functional areas.
|
Level 4 |
Results |
|
| Increase in production evaluated via production reports – (did ROI at 6 month interval) |
Control Group |
Participants in Program |
Cumulative increase
|
Cumulative increase
|
|
| Improved processes | Some improvement seen in processes to increase production – “tweaking” processes was done to meet needs | Significant improvement seen in processes to increase production – some processes eliminated and others changed to meet needs |
| Improved communication | Some improvement seen in communication – increased meetings with defined agendas and improvement in sharing information/data to meet client needs | Significant improvement seen in communication – increased meetings with defined agendas and action plans, improvement in sharing information/data to meet client needs, cross-training of participants between functional areas. |
Capture all costs related to the solution
The total cost of the solution for initial rollout was $350,000. This included, but was not limited to, costs such as:
|
|
Calculate the ROI
Processes were eliminated or changed and new processes were put in place to increase the production of widgets by 10% (the goal) within a six month time period. As seen in the chart above, at 6 months production increased by 11% for participant group.
| Business Performance Measure | Six Months Prior to Steps Taken to Increase Production | Six Months After New Processes in Place to Increase Production | Factor for Isolating The Effects of the Leadership Training Program |
| Output: daily production of widgets | 11,500 widgets produced per production line per day. | 11% increase in the number of widgets produced per day equaling 1,265 more widgets per day on one production line for a total of 12,765 widgets per day produced | 44% isolation factor.
Given that both groups gained the benefit of a 2% production increase due to machinery upgrade – it did not play a role in the isolation factor and was considered a “wash.” The isolation factor was comprised of the 11% increase by the participant group minus the 6.2% increase by the control group = 44% of total increase. |
Widgets are sold on average at $6.50 per unit.
The benefit was measured for a one year time period.
Net program benefit = ((total number of increased widgets per day produced after 6 months) * $6.50 per widget) * the isolating factor * 365 days minus (cost of the program for the five people)
Net program benefit = ((12,765-11,500)*$6.50)*44% *365 – ($350,000/30*5)
The “30” in the formula above reflects the total number of people who went through the program and we are only basing the limited look at the study on the 5 aforementioned individuals.
Net program benefit = (1,265 * 6.50 * .44*365) – (11,666.67 * 5)
Net program benefit = 1,320,533.50 – 58,333.35
Net program benefit = 1,262,200.15
The formula for calculating ROI is: ROI = Net Program Benefits / Program Costs * 100.
ROI = 1,262,200.15 / 58,333.35 * 100
ROI = 2,164%
Identify intangible benefits
The following intangible benefits were identified:
- Improved teamwork and communications between the three functional areas:
- Manufacturing
- Planning & Procurement
- Materials Management
The improved teamwork and communication made for more effective, stronger working relationships. Individuals in each of the functional areas shared more information with each other and actively sought each other out to work on tough production issues. Additionally, it was noticed that they became friends and frequently enjoyed each other’s company over lunch and breaks and after hours. This also led to increased job satisfaction overall.
- Increased customer satisfaction
Clients were pleased with XYZ Widget’s ability to meet increased production needs and were soon telling other organizations about XYZ Widget and their reliability and ability to meet needs. This enabled XYZ Widget to secure additional business and additional investment monies as needed to improve production lines which could be tied to a monetary value.
- Increased teamwork among the high potentials
The high potentials in the program (which included all the participants, not just the manufacturing functional individuals) became a close knit group and would gather together once a month at dinner. Additionally, they utilized a Microsoft SharePoint® site to brainstorm with each other, share information/best practices, discuss process improvement initiatives and generally problem solve as a team.
Summary
The ROI was anticipated at a conservative 25%. Actual ROI for this limited portion of the ROI study came to 2,164%.
The actual increase in production did not quite exceed the 10% initially desired at the six month time period. Although we showed an 11% increase in production per day, bear in mind we were “given” 2% due to a machinery upgrade. Technically, we achieved only a 9% increase in production at the 6 month time period. Of that 9%, only 4.8% (of the 9%) was attributed to the program. That being said, overall, if you look at it from a monetary perspective, the return on investment was more than satisfying to compensate for the lack of achievement in meeting the desired 10% increase. If you were to look at the 12 month time period, you would note the increase was 16.2% which certainly achieves the 10% goal.
The intangible benefits were of significant value to XYZ Widget, most especially the increase in teamwork and communication among the three functional areas related to manufacturing of widgets and the increase in client satisfaction with the company as a whole. This alone had tremendous value for the company.
Regular surveys sent out to clients (1 survey per quarter for the first year after the program ended and then 1 survey per every 6 months after that) showed that client satisfaction continued to increase as additional improvements were made. Additionally, employee satisfaction surveys showed significant improvement in job satisfaction and retention of employees given the improved teamwork and communication and information sharing that was happening. All of this could be tied to a monetary measurement if XYZ Widget chooses to do so.
Next post: Part IV: Reporting the Results of the ROI Study
