This is the transcript of episode 7
Click here to visit the full episode page and listen to the interview.
Welcome to this week’s episode of The Hired Trainer.com podcast.
I hope you enjoyed last week’s episode with Katy Casseli of BuildingGiants.com.
We’re staying on the same theme today, which is how can you, as a trainer, confidently answer the question, “Did my training work?”
With this in mind, I recently invited Kevin M Yates to come on the show.
Kevin is Manager of Learning Systems at McDonald’s global headquarters in Chicago and he’s a self-described data detective who looks for clues to prove that training and development programmes have worked.
Kevin, good morning and welcome to the programme.
Good morning, Mark.
So you’re in Chicago. It’s been called the ‘Windy City,’ ‘The Second City,’ “The Big Onion,’ ‘The City that Works.’
What does Chicago mean to you?
Great question. For me, Mark, Chicago means home. It means home-based.
That sounds a little cliché but it’s true.
When I think about the work that I’ve done over the years in my career, particularly with the amount of travel that I’ve done – and have been fortunate to have literally travelled across the globe – I love visiting different cities and learning about different cultures and doing work in different culture, but for me whenever I come back to Chicago, I always come back home.
Chicago interests me. It’s my base. It’s that place that just really brings me back.
It’s the city that I love.
It sounds like ‘it’s your kind of town!’ (terrible joke)
Well, it is. It’s true.
We have an international audience and many people, including myself, have not yet been to Chicago, although I tell a lie – I was in the airport once in the way back from Los Angeles.
So for people who have not yet been there, what exactly are we missing?
Particularly at this time of the year, it is summer.
In Chicago we have pretty brutal winters. It can get bone-chilling cold.
So I think that what you’re missing right now, particularly this time of year in Chicago are wonderful festivals that take place throughout the summer.
Those festivals take place in our neighbourhoods. Those festivals take place on our waterfront along lovely Lake Michigan and so the city just really comes alive during the summer time.
And I believe that that’s true because during the winter months we literally hibernate like bears.
So I think that what you’re missing right now are the wonderful food festivals, the book fairs, the music festivals.
If you’re going to visit Chicago, I highly recommend that you try to do it during the summer months.
The reason I invited you on the podcast actually – and you were kind enough to agree to come on today’s programme – is that you’re evangelical about analytics and data and measuring the effectiveness of learning and development through technology.
Sometimes its technology and sometimes it’s not.
There is certainly technology in place that helps us measure the impact of learning.
One of the things and one of the areas where I am most interested now is XAPI. That really does a great job at helping us measure the impact of our learning ecosystems, but there are methods and practices that help us measure as well.
So, while there is technology, there are also methods and practices and so, for example, a few that I’ve follow – and I consider myself to be method agnostic. I just like to borrow the best of the best.
So, when you think about the Kirkpatrick model, you think about the Phillips ROI model, you think about the Centre for Talent reporting and the Talent Development Reporting Model from David Vance.
There’s a lot of opportunity out there, so I like to combine methods with technology but you aren’t limited to technology to measure the impact.
Does that make sense?
You mentioned something called ‘XAP.’ What is that, for the uninitiated?
I’ll put it in layman’s terms.
XAPI is a technology by which you can take a look at what is happening in the entire learning ecosystem and gain insight on how the learning ecosystem is impacting people’s performance and business results.
So, when I talk about a learning ecosystem, I talk about not only what is happening in your LMS for example, but also taking a look at where employees are going to get other learning and support resources.
Maybe an employee’s going to YouTube on the Internet or maybe an employee is seeking out some type of white paper on the Internet, or maybe they’re looking at some other resources that are in the business that you might not necessarily consider to be part of learning but is what they’re using to help their learning.
So XAPI brings all of those disparate data sources together and allows you to tell a credible story about the performance efficiency and effectiveness of your learning ecosystem.
Does that make sense?
I presume API means ‘application programme interface?’ Is that the same API?
That’s exactly it.
And then the ‘X’ stands for ‘agnostic’ which literally could be anything?
Experience, I believe.
Oh, experience. Okay, that makes sense.
So looking at your CV, you’ve been in Learning and Development for pretty much all of your career as far as I can see, with positions as diverse as Manager of Strategic Learning at Grant Thornton, and Director of Global Talent Development at Kantar Millward Brown.
What originally attracted you to the world of Learning and Development?
Very interesting story.
My career in Learning and Development actually started in a classroom in a Chicago public school teaching second graders.
It was there teaching eight and nine year olds that I realised that I have a passion for the transformation of learning and education.
Just seeing how through knowledge and learning, behaviour can change, performance can change, stock can change.
Really seeing that in a classroom with second graders, that is where the inspiration, I think, started for me as it relates to education.
So what I did is take that passion for teaching and learning and education into the corporate environment.
I started off as a day to day trainer, boots on the ground, six to eight hours a day training in bank-ware applications and customer service, and then that just led to progressive roles throughout the function doing things like constructional design and learning solution development, global learning, learning technology, learning operations.
So it has been a progressive journey in that it started as simple as being in a classroom with second graders and that led me to where I am today.
I’d say it’s your intention to stay in Learning and Development for the rest of your career?
Absolutely. I have found my ‘sweet spot’, as they say.
What are some of the highlights of your career to date?
One of the biggest highlights that I’ve had in my career is having opportunities to work in different cultures and in different countries.
So, for me, what really stands out is the time that I’ve spent in Bangalore, in India, and during the time that I was there I was setting up some training academies for new hires in a Market research and analytics business.
For me, that really stands out as if not the highlight of my career, certainly in the top three or four highlights of my career because it was the first time that I really saw that at the end of the day – no matter where we are in the world – the one thing that we have in common in our humanity.
And so even though it was a very different culture for me, and for the work that I was doing there, what I discovered is that we all want opportunities to learn and grow and develop in our careers and that is what we have in common, again no matter where we are.
So I think that time that I spent in Bangalore building those training academies is really, if not the highlight, certainly among the top.
And it’s something you might like to do again?
I would love to not only get back to Bangalore to see some old friends, but to do that same kind of work, or maybe in another country where I’ve not had the opportunity to do that kind of work. I’d love to, yes, certainly.
You describe yourself as a ‘data detective who loves telling stories about learning and development with the use of data and facts.’
What is your favourite story to tell?
I have a few favourite stories.
I think the one that I like to tell the most is the one where I can absolutely attribute a change of performance or a change of behaviour or a change in a business metric to learning, training and development.
So when I have the data and it is fact based, it unequivocally says as a result of training, as a result of professional development, we’ve seen people’s performance impact a business role in this kind of way. Or where we can say training, learning and development resulted in a change in a business metric.
Being able to follow that chain of evidence, follow those facts and use data to tell that story, I think that is when I am most excited because I do believe in the transformational power of training, learning and development.
And because of that belief, I am most excited when I can support that belief with facts and data as evidence to unequivocally show where training, learning and development is making an impact.
Does that makes sense?
I think it does.
So, technology places a role in this, but not just technology.
I think also, if I get this correctly, you’ve mentioned one of your slide shows or videos about performance evaluations.
Now all trainers are used to happy sheets – these things we have to give out at the end of a training session and people comment about the food and the environment, but often we’re missing the key questions which we should be asking in training evaluation.
Yeah. For me, that is an incredible opportunity.
Yes, there are the traditional smiley sheets that I’m sure you receive, and I certainly receive them after a learning or training experience.
The greatest opportunity is to collect the right kinds of data that lets us know or that can provide a forecast for the extent to which that training or that learning experience will actually influence or change behaviour, performance or the job.
So, for me, when I think about the types of questions that you want to ask and the types of data that you want to collect, you want to ask the kinds of questions that will help you predict or forecast whether or not someone’s behaviour is actually going to change on the job.
I want to underscore and emphasise that it is a predictive metric or forecasting metric, but one nevertheless that will give you immediate insight into the effectiveness of that learning solution or that training programme or that learning experience.
So if you ask if the right kinds of questions, I believe that you can forecast or predict with confidence and certainty, the extent to which something is going to change or something is going to happen that is different as a result of training, learning and professional development.
Does that make sense?
It does, yeah.
Again, I watched one of your videos recently, which I found very entertaining.
You said in that, that the question that people ask you the most about learning and development is, “Hey, Kevin, why don’t more learning and development teams measure the impact of their work?”
And you believe that there are essentially three fundamental reasons.
What are those three reasons?
Those three reasons are accountability, capability and deniability.
When I talk about accountability, what I speak to is the extent to which learning and development teams and organisations are held accountable for results.
That really starts at the top.
So when senior leaders and executives ask the learning and development team for data and evidence of proof that shows whether or not they’re making a difference, trust me, they will deliver it.
When it comes from the CEO and the CEO says, “I want to see some facts for the impact that learning development is making,” again I’m pretty confident that the learning and development team will do that.
So when it comes to accountability, one of the reasons that it’s not happening is because we aren’t held accountable in that way.
The second reason that I believe more teams aren’t measuring is capability and that is by no fault of the learning and development teams themselves.
The way I see it, measuring impact and using data analytics for learning development is an art of science and a skill.
It’s a very specific kind of capability or skill set and so that kind of talent doesn’t always exist on a learning and development team.
If the team lacks the capability to use data analytics and measurement to show insight, then obviously they can’t do it, but there’s a solution for that.
There are a couple of solutions, one of which is you can hire the talent because there are experts out there in the Market that can help with using data and analytics for learning development.
Then you can also build that talent on your team, so that’s a development opportunity and a learning opportunity so that the team can learn how to do it.
And then the third reason for why I believe more teams aren’t measuring in practice is deniability.
If you don’t know that I am not producing results and if I don’t have to produce data that shows that I am showing results, then there’s that deniability there.
So, in other words, I don’t have to produce the evidence that shows that I’m not making an impact.
So if I don’t produce that evidence, I can always rely on deniability.
That’s my least favourite reason because I believe that learning and development teams are doing incredible work across the globe, and so I don’t believe that we need to fall back on deniability because again we are doing great work, we are making impact in the businesses in which we serve.
So rather than deny those times where we don’t believe we’re making an impact, I say just use data. Just use facts to show that you actually are making an impact.
Does that make sense?
Yeah. I often wonder if that’s one of the reasons why training and development is one of the first things to suffer when there are cuts internally in organisations.
Maybe there are pressures externally, but one of the first functions to normally suffer is training.
I wonder if that’s because the people who make decisions at C-suite level don’t feel there’s a strategic requirement for it and then it can’t be evidenced and therefore they think that ‘we don’t really need training and development.’
That is a sad reality, I think, for our role in businesses.
We are the first, or if not one of the first, areas where there is this view that we can cut back or that we can reduce expenses and that’s unfortunate again because my experience tells me that learning and development teams are doing incredible work and are definitely making impact in the businesses where we serve.
Without that data, without those facts that tangibly show that we are making an impact, we don’t have the fire power that we can show to the business that says, “Yes, we are a valuable partner. Yes, we are making a contribution to the business and here’s the data to prove it.”
So, yeah, unfortunately we are one of the first groups that might bear the brunt of reduction and expenses, but I don’t believe that that’s always the right thing to do.
No, of course not.
So thinking of the justification for training and development, I’ve often heard it said in conversation that the only thing worse than training people and watching them leave, is not training people and watching them stay.
You could say that the only thing worse than training people and not measuring the effectiveness of the training, is training people and pretending it works.
What is the missed opportunity or consequence if organisations do not correctly, competently, consistently, reliably measure the impact of their training investment?
I’ve touched on it a little bit just talking about how we bear the brunt of impact when it comes to reducing expenses.
Then I’d also say, just to clarify, there is a belief that I’ve seen some learning development organisations have that relates to needing to measure everything.
I would say that you do not need to measure everything, more so than any reason probably just because you don’t have time, resources and opportunity.
But for those training, learning and development initiatives that are strategically tied to the business in a way that training is going to help the business achieve a goal, those are the ones that you definitely want to measure.
So, to your question then, what do we miss when we don’t measure impact?
I think that the first thing that we miss is an opportunity to show to the business that we are a valuable business partner and just like other parts of the business that are driving toward goals and helping the business reach goals, so does training, learning and professional development.
So, that greatest opportunity that we miss, we don’t have the fact based evidence that tells our story and that tells our story of impact.
We’re really missing that opportunity to demonstrate to the business our value.
I think the other thing that we miss is an opportunity to inform decisions going forward about what works and what doesn’t work.
I often talk about the good stories; the good stories where we can show that training, learning and development makes an impact, but on the other side of that sits the story about those times where training, learning and development does not make an impact.
So what we miss when we don’t measure and use fact based evidence to answer questions is opportunities to stop doing what’s not working and start doing more of what’s working.
Does that make sense and does that answer your question?
It does indeed.
I’m thinking that there are people out there, people who listen to this programme, who may belong to a small training and development unit within a company. It could be four or five, six people, maybe less.
You touched upon capability. What can a team do if they don’t have the internal capability to adequately measure the impact of training?
That’s a great question and it’s so interesting that you ask that because I’ve been thinking about that a lot lately.
There are large organisations out there with these huge robust learning and development teams that are focused on measurement and data analytics.
But I’ve been thinking about those businesses that are smaller, that may have a team of one in learning and development or maybe have a team of two in learning development, but it’s not a large team so they don’t have these large robust methods and resources and such to focus on measurement, data and analytics.
So what do you do? What do you do if you’re a team of one?
I would say if you are a team of one and you want to go on the measurement journey, my recommendation is pretty simple.
Find one thing that you can measure in a person’s behaviour or performance that shows the extent to which training has changed that behaviour and performance and let that be your first success story.
You don’t have to measure everything.
Again, if you’re that one person, if you’re one person, you’re working on a team by yourself and you’re thinking, “Ah, I’ve got to start measuring impact.” – I’m speaking to you right now, wherever you are – you need to find one thing about a person’s behaviour or performance which you can attribute to the impact of your training, learning and development. Let that the basis for telling your story about how training, learning and development impacts people’s performance in the business.
Does that make sense?
What kind of things come to mind just off the top of your head?
If I think about that scenario that I just described where you want to measure performance, I think the first thing you need to be really clear about is where the learning solution or the learning experience is expected to impact performance or change behaviour.
Is it expected to help a person lead in a better kind of way?
Is it expected to help a person perform, maybe like if you’re in a manufacturing environment for example?
Maybe you’re expected to do something different in your manufacturing process or maybe you’re expected to sell differently or maybe you’re expected to administer dosage differently if you’re in a hospital or something like that.
So the essence of what you want to do is just take a look at that specific skill or capability that is expected to change as a result of training, learning and development.
When you can identify that specific skill, behaviour or change, you can then say, “How is training supposed to help this person to do something different?”
And so then if you start to see that person do something different, you can then begin to make the different types of attributes to training towards changing that person’s behaviour that is appropriate.
I use the word ‘appropriate’ deliberately because training by itself doesn’t change a person’s behaviour performance but it is a part of what a person’s behaviour may change.
So if you can dive into that, take a look at all the different ways in which a person’s behaviour can change and pull out the extent to which training contributed to that, then I think that’s where you can start on that journey.
Will that take the form of a report or a video or some data from a spreadsheet?
I’m just thinking because there are people out there listening to this who are -as you called them – one-person-bands. They’re a single trainer or maybe they’re a small training company. They may not have the capability or the first clue to find evidence.
Where would they start in terms of just getting to grips with this concept of having data-driven training?
I can come along to a prospective client, I go along to meet a company I hope to work with and they say, “Show some evidence that your training works.” What kinds of stuff could they come up with quickly?
I know you’ve pointed us in the direction, but let’s just get a bit specific here.
What would readily come to mind for a one or two person company that hasn’t the first clue about data analytics?
Absolutely. I think the first thing that comes to mind for me is something as simple as a survey.
Let’s not devalue the data that can come from surveys because we can get some very valuable, very actionable data that comes from surveys.
So, if you can start with something as simple as maybe even something that’s free, like a Survey Monkey, for example. Survey Monkey is free.
So, if you want to begin to collect some qualitative and even quantitative data, you can start with something as simple as a Survey Monkey that asks the right kinds of questions that will help inform decisions about what you can do with learning and development and that will show how learning development is making an impact.
So again, start with something as simple as say a Survey Monkey.
The other kinds of data that we shouldn’t lose sight of is qualitative data.
Qualitative data can come from something as simple as a focus group or an interview.
You can gather people in a room. You can talk to them about their training experience, but then you can also talk to them about the impact of that training experience on their performance and their behaviour.
So that is another way in which you can gather and collect some credible data and it’s of low to no cost because the essence of what you’re doing is just putting people in a room and you’re collecting some attitudinal and experience data from the target audience.
Again, that’s a great question. If you’re thinking about where to start, you’re a small team, don’t have a lot of resources, I would take a look at where you can start using survey data.
Again, maybe even use something free like Survey Monkey, but also collect some attitudinal real time data from focus groups and from interviews from people within the business.
Those are two reliable, credible data sources.
Does that make sense? Does that help?
Luckily, you have a free guide which our listeners can download. It’s called How to Forecast Training Results: 10 Questions that Predict Performance Impact’
What would someone find in that guide?
I think that what they’ll find are ways in which they can use surveys to predict the extent to which training, learning and development will make an impact on a person’s behaviour or performance.
The other thing that they will find is ways in which you can visualise that data.
So right now, the ‘buzz’ phrase in the industry is ‘telling stories’ and using data to do it. One of the ways that will do that is by visualising the data.
So in that resource what you’ll find in addition to the kinds of questions that you can ask to forecast performance are examples and recommendations for how to visualise that data.
And then the third thing that you’ll find with that is where the action is as a result of collecting that data.
You just don’t want to collect data for the sake of collecting data.
You want that data to inform decisions, provide insight, guide recommendations. So in that guide you’ll also find some thoughts and recommendations for how to make that data actionable.
Right. You mentioned the word ‘visualise.’
Visualise literally means coming up with a way to show it graphically so that people can understand the story when it’s in a visual format.
That’s exactly it.
Reporting numbers is fine, but when you can give those numbers life with a picture – and by ‘picture’ I mean a graph or some type of imagery that tells that story – I think that it makes the story that much more powerful.
When we were little kids our mothers and fathers would read us to us bedtime stories and show us the pictures in a book, so I think that that’s where that desire or that inclination for pictures started.
I think that even today we can continue to gain the attention of our audiences when we tell stories and use visuals to do it.
So there are recommendations in that guide for how to use charts and graphs to create that imagery.
And speaking of stories or talking, you gave a talk at the ATD, which for the people who do not know what that is – and I didn’t until about a year or two ago – is the Association for Talent Development. Is that right?
That is right, and it is a global organisation and so that conference, the International Conference and Expo, was held just this past May.
A very exciting experience for me.
Barack Obama spoke at that conference so I might just say that he opened for me.
That’s another story to tell.
So the ATD is a global learning and development or talent development organisation.
It changed its name – I think – a couple of years ago to I suppose really encompass the global aspect. It’s no longer focused on the USA alone.
It’s a very, very successful organisation, runs a load of courses, certification programmes for trainers.
You gave a talk at that but you’re also scheduled to give a talk in September in London which is not very far from me.
This is a talk organised by the Learning & Performance Institute.
Can you give us a sneaky preview – without giving the game away – of what you’ll talk about at that conference?
In other words, if someone’s listening to this and would like to come along, what would you talk about that might interest them and get them to come along to the event?
Thank you for that.
I am super excited to be in London in September 5th and 6th with the Learning & Performance Institute’s Learning Live event.
What I will be talking about is using data and facts to tell learning and development stories.
I think that right now, more than ever, facts matter.
So, what I’ll be talking about is how do you collect facts that show whether or not training, learning and development is making a difference, and how do you use those facts in a way that is credible?
And how do you use those facts in a way that allows you to speak with confidence about the extent to which training, learning and development is having an impact in the business?
So if you’re going to be in London on September 5th and 6th, stop by.
And will any of that footage be available online at some point?
I’m not actually sure if it will be recorded or not. I hope so, but we’ll see.
Okay. So thinking of the future of data analytics, I’m going to lend you my time machine so that we can see what’s coming down the tracks in your area.
What does the near future, and maybe the future beyond that, hold for data driven performance measurement in the world of learning and development?
Is there anything we should be excited about?
I think that we should really be excited about the thoughts around learning ecosystems because historically we thought about training, learning and development as just really existing only with what training, learning development teams offer.
So the courses that we offer or the eLearning that we build or the webcasts that we host.
So the thought was always that training and learning development are the only ones who can offer that out to the business.
What’s changing now is that in addition to what is offered by training, learning and development, people are also seeking out other kind of resources and support that helps them build capability for their roles in their businesses.
So when we think about a learning ecosystem, we think about everything that happens day to day within a business, or even outside of the business, that supports learning.
So for me, when I look down the line and I look at the future, I see where we will be looking at learning ecosystems in its entirety, all of the things that are happening around the person, or what the person is doing, to support his or her all growth learning and development.
And I think about ways in which with new and emerging technology we’ll be able to measure within that learning ecosystem how training, learning development, professional growth, personal growth is actually helping people day to day in their roles on their jobs in the businesses that we serve.
So for me, in the future I’m just excited to see what learning ecosystems will do not only for people, but will do for us a profession.
And on some levels, the future is now because we’re already seeing where learning ecosystems are emerging and helping with all those things that I’ve just talked about.
Where can our listeners find out more about you, Kevin?
I would invite our listeners to take a look at my website, go out to kevinmyates.com.
Also you can follow me on Twitter @Kevinmyates.
I am on Facebook. Kev M Yates.
I have a YouTube channel and certainly feel free to join and follow me on LinkedIn.
Okay, wonderful. Thank you so much for coming on the programme this morning, Kevin.
Thank you for your time. This has been absolutely fantastic and I appreciate your inviting me