Workforce.com

The Nuts & Bolts of Valuing Training

November 1, 1998
The return on investment (ROI) is a Holy Grail of sorts for the training industry. Everybody wants it, everybody has theories on how to find it, and most are willing to go to great lengths to do so. It’s no wonder when you consider employers spent $55.3 billion on training in 1995, according to the American Society for Training and Development (ASTD).

The idea of an absolute number -- a neat package that shows the exact value returned for an exact value invested in training -- is a seductive idea. In some cases, it’s quite attainable. In other cases, it may simply be a seductive idea -- and one that’s best not pursued.

When people ask Laurie Bassi, vice president for research at Alexandria, Virginia-based ASTD, what to consider in calculating an ROI, she answers, “The first thing to consider is: Is it worth doing?”

Before embarking on your ROI quest, you should first contemplate the ROI for calculating your ROI. Is it worth the time and money you’ll spend? The answer, says Bassi, depends on the type of training you’re doing.

Calculating the “easy” ROI.
The ideal type of training suited for calculating ROI is one-time training on a specific skill -- teaching customer-service staff the latest computer program vs. putting the executive team through a series of leadership seminars.

The former is a discrete module in which few other factors will affect the outcome, and in which the outcome can be readily tested in a before-and-after scenario. This is the easiest and most clear-cut way to calculate an ROI.

The latter, although it potentially will have a greater effect on the company, is more difficult to quantify. It’s hard to do a before-and-after test on leadership skills, and other factors will come to play in workers’ performance.

“For the most part, [ROI] is the bane of every trainer’s existence,” says Laraine R. Mancuso, a trainer at the Learning Annex in New York City. “But industries like manufacturing have the easiest time doing it. You have people producing three nuts and bolts an hour, and after the training they produce six.”

ACCO World Corp. in Lincolnshire, Illinois, is a manufacturer of office and school supplies, ranging from paper clips and binders to computer supplies and productivity tools. Valeria Stokes, vice president of learning and development, tracked the effect of training on new production hires. After training, most new hires were able to produce vinyl binders at a 5 to 10 percent higher rate than tenured operators. “As a result of that finding, we recommended that we do recertification of operators on an annual basis to retain that production level.” Stokes was able to leverage that to not only continue the training, but expand it.

Let’s say you’re blessed with one such simple ROI scenario. To measure ROI, you want to start by isolating the impact of training as much as possible. To do this, you can train one group several months ahead of another, so you have a control group to test by. Or you can narrowly focus the training to do a before-and-after comparison. Then you must decide the impact the training should have. If learning the computer program should shorten each customer-service inquiry, then you must attach a cost to that extra productivity.

The bottom line is: Calculate the productivity effect. For instance, each customer-service person is able to handle 10 more phone calls a day. Calculate the dollars associated with that -- each phone call represents $6 of the employee’s time. Divide the final value -- 20 employees each save an average of $60 a day -- by the cost of the training. Voila -- the result is the ROI.

Michelin Tire Corp. in Greenville, South Carolina, recently completed a two-year training effort of its workforce. Its training consultant, Tech Resource Group Inc. (TRG), put Michelin’s 7,000 employees through a multi-week course on the company’s new desktop environment.

TRG, based in Raleigh, North Carolina, tested employees on their skill sets before training, immediately after training, and 60 to 90 days after training. “We wanted to set some sort of measurement so that Michelin could determine whether or not the millions of dollars it spent was getting a return,” says TRG Chief Executive Officer John Wingen.

The pre-training tests -- asking employees to perform tasks and answer questions surrounding software, Internet, e-mail and spreadsheet capabilities -- had an average 22 percent of items answered correctly. Directly after the training, employees answered 84 percent correctly; and 90 days after the training, 93 percent of items were correct. Using these figures, Michelin was able to calculate its own ROI.

In some cases, [ROI is] quite attainable. In other cases, it may simply be a seductive idea -- and one that’s best not pursued.

Another approach to gauging the effectiveness of training is a four-level model developed by Donald Kirkpatrick, a professor of business at the University of Wisconsin. A widely used measurement, it has become known in the training industry simply as “Kirkpatrick’s Model.”

Level one measures reaction. Using surveys at the end of a session, the trainer finds how well participants liked the training. Although the participants’ enthusiasm doesn’t link directly to ROI, it’s important because reaction is a leading indicator of information retention.

Level two measures learning. Did the training change attitudes, increase knowledge or improve skills? Before-and-after testing demonstrates that learning took place, but not whether it will be applied to the job.

Level three measures the application to the job. With help from participants, direct reports and supervisors, the trainer gauges how much participants’ behavior has changed on the job as a result of the training.

Level four covers business results. How did the behavioral change translate into improvements for the company?

This four-level approach may work better for managerial and soft-skills training. Consider a seminar in sales skills, for instance. To test learning, you could have participants explain -- before and after the training -- how they’d approach different sales opportunities. To test application to the job, you could track sales in the months before the training and after. To show business results, you use the sales figures -- acknowledging that other factors may have affected them.

When it isn’t quite that simple.
However, don’t force the issue, warns Bassi of ASTD. In more complicated scenarios, she says, the focus is best placed on value and results, rather than a number-based ROI.

Once you move away from objective “x-more-bolts-sold” scenarios into more vague managerial skills, you’ll have to use subjective judgments to determine the effectiveness of the training. “Then you’re making very strong assumptions which can be pretty difficult to defend,” she says. “I think then it’s legitimate to ask, ‘Is there some other way we should be assessing the value of what we’re doing in these kinds of cases, rather than ROIs?’”

Jim Smithers, an industrial psychologist and professor in the department of management at LaSalle University in Philadelphia, is a recent convert to this line of thinking. “I used to believe [ROI] mattered a lot,” he says. “I’ve changed my mind a bit. HR people are made to feel guilty if they don’t do a cost-benefit analysis. But that’s not the case in lots of other departments, like advertising expenditures or internal auditing practices. We presume it matters -- it’s a noble idea, but at another level it’s a standard many other areas aren’t subjected to.”

Indeed, some companies are turning away from calculating return on investments, and are measuring return on expectations, instead. In this kind of approach, those who are involved decide exactly what they expect to achieve from the training. This set of expectations becomes the baseline for determining success. Months after the training is complete, the stakeholders review their agreed-upon expectations. They then decide if the results are in line.

Once you move away from objective “x-more-bolts-sold” scenarios … you’ll have to use subjective judgments to determine the effectiveness of the training.

This approach allows for more anecdotal analysis. If a group of managers completed a communications training course, stakeholders can then discuss the ways they feel communication has improved. This provides a more realistic picture than if HR were forced to place a convoluted dollar amount on improved communications. “[The approach] is softer; it’s not as rigorous, it’s not as scientific,” says Bassi. “But I think sometimes calculating ROI is false rigor.”

Halliburton Energy Services, a logging-and-drilling products manufacturer in Fort Worth, Texas, has implemented this type of “soft” approach. When the plant reorganized into product-performance teams earlier this year, Kyle Klabenes, training specialist, hired outside consultant George Group Inc. in Dallas to help the 17 cross-functional teams better communicate and work with each other.

Before the training was complete, the teams constructed mission statements, ground rules, and team and individual roles. The teams also had six distinct performance measures that were expected to improve. So far, Klabenes has been impressed: “People are meeting on a regular basis, and using the skills and plans introduced in the training. People are communicating and getting issues addressed, prioritized and categorized.” The division was able to articulate improvements it wanted from training, and now is busy tracking those improvements.

If you choose such an approach, advises Kathy Leck, executive director of LEAP -- Leadership Education Advancing Performance -- at the Lake Forest Graduate School of Management in Illinois, keep a tight focus. When she advises companies on training, she tells them to name specific behaviors they want changed. That allows for a more direct cause and effect. Rather than saying, “I want my people to manage time better,” they’d say, “I want 95 percent of deadlines in my department to be met.”

Leck can compare before-and-after percentages to see if the training is effective. “Then, right from the get-go, you have a chance of having a closer relationship between training and change,” says Leck. “You can’t make it absolute, but you can come much closer.”

Focus on training’s value.
Peter M. Ramstad, chief financial officer of Personnel Decisions International in Minneapolis, has yet another approach that eschews ROIs. Despite his CFO position and background as a certified public accountant, Ramstad says, “From a financial perspective, an ROI analysis isn’t really a useful tool. The question is not: What is the ROI? The question is: Should I invest in training? And you should invest in training any time the return exceeds the minimum threshold ROI.”

How to determine that number? Calculate your company’s cost capital. For example, suppose the finance department demands a 20 percent return on any investment. If your proposed training costs you $100,000, then you need at least a $120,000 return. Your question then becomes “How ineffective could this training program be and still generate the return?” says Ramstad. “You back into a better decision model. We think of this as a training exercise in terms of a new way to think about HR investment decisions.”

Toshiba’s Electronic Imaging Division in Irvine, California, is one company that made a conscious decision to focus on the value of training rather than the ROI of training. Because the company made a shift to multifunctional products -- copiers are no longer just copiers but also scanners, printers and fax machines in one -- dealers had to make a shift, too. In October 1996, training on selling the new products began. “We had to educate the dealers to sell our products,” says Tony Codianni, director of training & dealer development. “Training is an investment. Obviously, how we estimate the return is on increased sales. But we don’t have a one-to-one correlation. We feel it’s an investment in our dealers to help them move toward the 21st century.”

Codianni says the company has been able to double its market share, although it doesn’t credit training alone with that success. Rather, the company treats training as one key piece in an overall business strategy.

It’s not easy to wean a company from its ROI obsession, but those demands aren’t as much for an ROI as they are for trainers to take more accountability.

In fact, the training department was positioned several years ago to become an independent profit center. Instead, Toshiba specifically chose to reposition training as an investment -- so rather than forcing dealers to pay for training, the company asks only for a fee to cover the cost of materials.

It’s not easy to wean a company from its ROI obsession, but often those demands aren’t as much for an ROI as they are for trainers to take more accountability, and for training to be more applicable to jobs. Meet these demands by showing some form of cause and effect for training.

For instance, Stokes has embarked on training for ACCO salespeople in relationship-building and negotiation. Although it will be difficult to quantify an ROI, she plans to track whether sales and customer-approval ratings have risen, and whether the salespeople can decrease the marketing expenses of their customer programs, a sign that they can close a deal without granting too many concessions, but still achieving a “win-win.”

Mancuso says anecdotal evidence can be an excellent tool. Survey the training’s customers -- external or internal. Survey the training participants themselves. The fewer hard-and-fast ROI numbers you have, the more anecdotal information you need.

Align yourself with the line managers, and have them be advocates. “The person on the line is going to see the change,” says Mancuso. “It’s building the relationship with the line managers that’s going to give you credibility, and give you the support you need. So when people are saying you absolutely have to have an ROI, you can say ‘No I don’t. Line managers will vouch for the changes.’”

If your training lends itself to a simple calculation of ROI, doing so gives you more credibility as a business partner. But if your training is too abstract to support an ROI, your time and money would be better spent elsewhere. Remember that when all is said and done, ROI is a means by which to measure the value of training. There are other ways -- returns on expectations, ROI thresholds and anecdotal information -- to evaluate your training’s success. Choose the one that will work best for the training you’re doing. Ultimately, that is the best ROI.

Workforce, November 1998, Vol. 77, No. 11, pp. 80-85.