The old, old problem of measuring 'Effectiveness'

Photo by Scott Graham on Unsplash

 I just saw a post from someone in a Facebook group I'm in that asked the following.

What questions could I ask in a survey among students, who have gone through my "AI for managers" course to try to measure effectiveness? Any tips and ideas? What would you ask? Or maybe someone already did such a survey?

I was about to answer but then realized that my answer was going to be way too long for Facebook and deserved some serious writing.

So here goes.

Sergey

I guess it all depends on what you mean by 'effectiveness'. 

Traditionally, based on the work of the late great Don Kirkpatrick, there are 4 levels of measurement we use to evaluate learning. 

1. Reaction
2. Learning
3. Behavior
4. Business Impact

This is also simplified to 

Did they LIKE it? Did they LEARN it? Are they USING it? Did it MAKE A DIFFERENCE? (read more about that here)

For me, 'effectiveness' is going to be somewhere between levels 3 and 4 depending on what your client has asked of you. 

I ran a workshop at the AIN conference in Oxford a few years back about how you can design a workshop for Business Impact. Read about that here 

I hope you got some clear instructions from your client as to why you were running that course. In other words, what was the Business Impact that they were looking for, and what changes in behavior did you agree would bring about that change?

If you did then try something like this

Can you give any examples of the way your behavior changed as a result of something you learned on the course

If you didn't then the best advice I can offer you is to approach this by trying to backfill that now?

So a couple of great questions might be:

On a scale of 1-10, how effective were you as a manager BEFORE you attended the course?

On a scale of 1-10, how effective were you as a manager AFTER you attended the course?

If these are different, what did you do differently that you learned about on the course that made that difference?

Hope that helps

From Certified Professional in Learning & Performance to Certified Professional in Talent Development

In the past I've had a lot to say that was pretty negative about my professional association and it's handling of major changes. (https://alanmontague.blogspot.com/2014/05/talent-developer.html , https://alanmontague.blogspot.com/2015/10/professional-certification-from.html & https://alanmontague.blogspot.com/2015/12/an-open-letter-to-atd-and-atd-ci.html)

So if I'm going to be vocal when they handle things badly, I feel I should be equally vocal when they handle things well.

So here goes

I think that they have handled the most recent round of updates very well.

For the last 20 years, our profession has been defined by a Competency model. The first couple of iterations were called the Workplace Learning and Performance Competency Model and then (after they gave up on the WLP naming) it became the Training And Development Competency Model. It was this model that I was tested against to get my CPLP designation.

Late last year they gave a sneak peek of what they were about to do with that. They were moving from a Competency Model to a Capability Model. (https://tdcapability.org/)

The sneak peek was for some key stakeholders including CPLPs In their sneak peak they explained who that had involved with this change and brought out several of the respected leaders in the L&D (Sorry still can't get round to calling it TD!) community who spoke about how they had been involved. 

They told us then that they would be hosting a webinar for certification holders on how this would impact us specifically early in 2020.

Yesterday they had that webinar where they introduced the changes that this new model would have on the certified community.

Again the changes were well thought out, and had involvement from many of the key stakeholders. They had an active 'chat' function on the webinar and were really good at answering questions raised in there.

I've criticized them significantly in the past for their very bad Change Management. This was one of the competencies in the old model and in the past they could have been used as a case study in how NOT to do it.

Change Management is now a capability  in the new model and the way they handled this change could also be used as a case study, but this time in how to do it well.

So congratulations to the team at the ATD CI for a very well handled change. I'm thinking I might have renew my membership now.

The biggest Learning event of 2018 (so far)

It’s May 29^th and Starbucks have closed all their North American Stores for the afternoon for training’ on racial and other unconscious bias issues.

8000 stores. 175,000 employees. An estimated cost of $4.4 Million.

It’s a big event by far but if Starbucks is to be believed (and I’ll give them the benefit of the doubt here) it’s only the start of their process.

And that’s why I’m writing about it on what has become a very quiet blog. There was a great article in The Atlantic I saw this morning that laid out how the message has changed. Originally it was that the training was happening today. Now it’s that the training is STARTING today.

Unconscious or Implicit bias is the sort of ingrained, deep down habits that are happening to people at the most basic human levels. It’s the results of years or decades of inputs to our brains from thousands of sources. There is no way that this changes in a single training session.  

But it’s also something that can change. When a well thought out program is designed and put into place then we do see changes in behavior on this and other topics. “Another version, which focused on gender bias, was presented to STEM faculty at the University of Wisconsin. in departments that had received the training, the proportion of women faculty hired over the next two years increased from 32 to 47 percent. Departments that hadn’t received the training showed no change at all.”

Training alone is not going to change behavior, especially on something as deep as bias. But if any organization is serious about changing the behavior of it’s people and by so doing changing it’s culture then a well thought out program, designed and put into place is a vital part of that.

4 days in Atlanta. Tweets Galore!

Designing learning for business impact

Using the Kirkpatrick model to generate design questions.

I recently ran a workshop at a conference held at Keble College Oxford.  The organization hosting this conference is the Applied Improvisation Network, a great group of people who have discovered the secret that anyone can learn and use the same skills that make a team of improvisers world class listeners, great collaborators and amazing innovators.

Many of the folks newer to the applied side have a great selection of exercises to pull from but struggle to structure a workshop to ensure it focuses on what their clients need. As a Learning Pro I've always got ideas about that and I've even written about them before in a prior blog post. So I was happy to help creating and running a full day workshop on how to create and run a workshop.

Borrowing heavily from Thiagi and his Rapid Instructional Design methods we started of with each of the 15 attendees writing the title of a workshop they would want to create.  We then used a modified version of the game Thirty-Five to pick the top 3 workshops that we would focus on. And they self selected into teams.

Then it was time for me to do a little work as Facilitator

I shared with my learners the Kirkpatrick 4 Levels model.  I suggested that they use this model not only for evaluation, but also for design.

• As the top level is about organizational impact I suggested that they first define the organizational impacts that their clients want to see. And how the organization will measure that. 
• Once that is clear the next questions is what behaviors do we expect will lead to those business impacts. And how will those be monitored?
• With learning at the next level I proposed the next question that need to answer is; What skills did they need to learn or knowledge they needed to have or attitude they needed to adopt that is currently preventing the behavior that they want to see. 

The teams then split off and did fabulous work defining all those 3 areas while I sat back and had a cup of tea.

Before we broke for lunch we reviewed and refined the designs that they had built together.

One of the workshop design brainstorms

After lunch I briefly shared with them some ideas around Accelerated Learning and the AL Cycle and then each team picked one of the learning points and developed a learning activity to address it.

But that's another blog post....

Happy designing and learning
Alan

Learning Styles: The myth I'm glad I learned about.

Visual Auditory Kinesthtic

I first learned about learning styles when I first learned about learning.

It was a workshop run by a colleague who was introducing us to the concept of Accelerated Learning. A powerful methodology that he had been using to run a successful programmer training school at the company I worked for.

The ideas of Brain dominance, Multiple Intelligences and Learning Styles all made perfect sense to me.

In the years that followed I built many a program and always tried to provide a balance of activities that favored people with different learning styles. I can give you many personal stories of being faced with someone who didn't get something presented one way who immediately grsped the idea or concept when I re-framed it to favor a different learning style. I'm pretty sure that anyone who has been in the workplace learning profession for a while can do the same.

But despite my personal experience I have to concede that all the research that has been done on this topic has come to the same conclusion.

Learning Styles are a Myth.

So, why on earth would I be happy to have learned a myth and implemented it.

I'm happy because the V A K model meant that I provided Variety.
I'm happy because the V A K model meant that I thought carefully about the Activities I used.
I'm happy because the V A K model meant that I engaged my learners with different Kinds of content.

So I'm going to continue using V A K  but now for me it will mean Vary Activity Kinds.

Happy Learning

Systems Training - Principles and a Process

I recently put together a document to share with our business users on the approach we take to create systems training. 

Once my colleagues had helped me take my long wordy document and tighten it up, I had something that I feel is well worth sharing with you my loyal reader(s).

I'd love to hear any commnets you have on this.

Guiding Principles for Systems Training

1.    Know your learners: Understand what they already know and what they need to learn.

2.    Target training: Don’t waste time by making people learn what they already know.

3.    Use the right toolset: Choose eLearning or Instructor led or both, based on the business need.

4.    Context is king: Cover the system in the context of how your organization uses it to meet their specific business needs.

5.    Training is practice: Being told how to use a system is not the same as being able to use a system.

6.    The proof is in the doing: If they need to learn how to do something, you need to know that they can do it.

7.    Follow a process to teach a process: Even if the systems aren’t consistent, the way we teach them should be.

The Systems Training Process

1.    Introduce the concept

•          What this is all about in the world and in business

2.    The process we use to address this concept

•          Every organization tackles a business need a little differently than others. This is where we connect the concept outlined above to the structure and processes of our business

3.    How the process is implemented in the system

•          What are the steps and screens used within the system for each process identified above

4.    An opportunity to see the process being completed

•          Demonstrate the system in use

5.    An opportunity to practice executing the process

•          A chance to practice with feedback.

6.    An authentic test of learning

•          Test the learners’ ability to perform the task

Happy Learning