IU IST R561 Unit 1 Quiz

Unit 1 Quiz

 

Jennifer Maddrell
Indiana University

R561: Evaluation and Change

Unit 1 Quiz

Dr. Knuth

May 24, 2006

Emphasis and Purpose of Evaluation

Type and Approach to Evaluation

Morrison

et al.

·

Emphasis: Instruction

·

Stated Purpose: “Evaluation is used for the purposes of making judgments about the worth or success of people or things” (p.240)

·

Purpose comparison based on 5 common purposes:

o

Feedback: High

o

Control: Medium

o

Research: High

o

Intervention: Low

o

Power: Low

·

Type: Formative and Summative evaluation

·

Timing:

o

Formative: Before instruction is fully developed

o

Summative: After instruction is used, but before sustained implementation

·

Approach: Specific approaches discussed in more detail in subsequent chapters

Phillips

·

Emphasis: Instruction as it supports improved group or organization performance

·

Stated Purpose:

o

“There must be a comprehensive measurement and evaluation process to capture the contribution of human resource development.” (p. 1)

o

Determining customer satisfaction of participants and managers (immediate, senior or top executives)

·

Purpose comparison based on 5 common purposes:

o

Feedback: High

o

Control: High

o

Research: Medium

  • Intervention: High
  • Power: High

·

Type: Summative and Process evaluation within 5 Level ROI

·

Timing: Post instruction

·

Approach: Evaluation method should emphasis “ultimate outcomes of improved group or organization performance” with Business Impact and ROI being the most desired that receive the most support. (p. 44)

Van Tiem

et al.

·

Emphasis: Performance Improvement not limited to instructional interventions

·

Stated Purpose: Evaluation generates information that will (1) help the organization value or judge the results of a performance, and (2) trigger or support a decision regarding the performance, the performer or the organization. (p. 156)

·

Purpose comparison based on 5 common purposes:

o

Feedback: High

o

Control: High

o

Research: High

o

Intervention: High

o

Power: Medium

·

Type: Formative, Summative / Confirmative and Meta (quality of evaluation)

·

Timing: Formative (ongoing), Summative (post intervention), Confirmative (long term) and Meta (during each stage of evaluation)

·

Approach: Evaluation phase of HPT Model with multiple model driven approaches for each type of evaluation type are presented and compared within Chapter 7



 

Emphasis and Purpose of Evaluation

Type and Approach to Evaluation

Brinkerhoff / Dressler

·

Emphasis: Performance Improvement, including the impact of non-instructional “performance system factors”

·

Stated Purpose: An evaluation strategy builds “organizational capability to increase the performance and business value of the training investment” (p. 17)

·

Evaluation must assess:

o

How well an organization is using learning to drive performance improvement

o

What an organization is doing that facilitates performance improvement from learning

o

What an organization is doing that is impeding performance improvement

·

Purpose comparison based on 5 common purposes:

o

Feedback: High

o

Control: Medium

o

Research: High

o

Intervention: High

o

Power: Low

·

Type: Summative, Process evaluation based on Success Case Model

·

Timing: Post instruction

·

Approach: Survey and more extensive follow up to a relatively small sample of both most and least successful trainees to determine the extent that recent training made a significant different to the business (i.e. the instructional impact on performance).

  • Successes: “Document the nature and business value of the application of learning and identify and explain the performance context factors that enabled these few trainees to achieve the greatest possible results.”
  • Unsuccessful Trainees: ID and understand the performance system and other obstacles that kept them from using their learning.”

·

Provides quick snapshot of information on both business impact of instructional program.

Kirkpatrick

·

Emphasis: Instruction

·

Stated Purpose:

·

Purpose comparison based on 5 common purposes:

o

Feedback: High

o

Control: Medium (as Results level rarely achieved)

o

Research: Low

o

Intervention: Medium

  • Power: Low

·

Type: Summative

·

Timing: Post instruction

·

Approach: Evaluate learner:

o

Reaction: Measure of “satisfaction”

o

Learning: Did participants change attitudes, improve knowledge or increase skill?

o

Behavior: Was there a change in behavior?

o

Results: Did production increase quality improve, costs decrease, profits increase?

·

While considered one of the most popular approaches, most do not complete the behavior and results levels that are deemed the most valuable.



 

Emphasis and Purpose of Evaluation

Type and Approach to Evaluation

Kaufman

et al.

·

Emphasis: Instruction and other human performance “interventions associated with strategic and tactical planning, performance improvement, organization development, customer satisfaction and societal contributions”. (p.206)

·

Stated Purpose: In contrast to those who consider evaluation a process to supply information to decision makers, for Kaufman et al. the purpose of evaluation is to compare results with intentions

·

Purpose comparison based on 5 common purposes:

o

Feedback: High

o

Control: Medium

o

Research: Medium

o

Intervention: Medium

o

Power: Low

·

Type: Summative

·

Timing: Post instruction

·

Approach: Like Kirkpatrick, evaluate Reaction, Learning, Behavior, Results, but also include “Mega” societal contributions to create following evaluation levels:

o

Mega: Contributions an organization must make to its clients and society.

o

Macro: Like Results

o

Micro: Like Behavior (performance) and Learning (acquisition)

o

Process: Subset of Reaction

o

Input: Subset of Reaction

 

 



2. Evaluation Context:

 

Underwriting Training is periodically conducted within the insurance company to address underwriter skill gaps, such as exposure analysis training. In the past, other than short trainee “satisfaction” surveys at the end of the session, an evaluation of the effectiveness of the Underwriting Training course was rarely completed. The unfortunate result is that the business impact and effectiveness of the training program is largely unknown. It has become difficult to determine whether the same training program is worth pursuing in the future or if other alternatives should be considered. Therefore, it is recommended that an evaluation process be implemented to determine the quality and effectiveness of the chosen training program and to compare it with alternatives. The evaluation should focus on the following goals:

  • Feedback: Have participants received benefit from the training? Are they using the new skills in their jobs?
  • Control: What is the business impact of the training program? Are their other interventions that could have been considered to augment the training? Are there other interventions (instructional or non-instruction) that should be considered as alternatives to improve performance?

 

Participants and trainers considered previous attempts to institute an evaluation program to be too time consuming, cumbersome and a burden in their already busy schedules. Further, trainers and managers did not feel the evaluation provided enough information about the business impact of the training.

 

Given the stated evaluation goals and the desire to streamline the evaluation process, the training team is recommending the Success Case Evaluation Model (Brinkerhoff and Dressler, 2002) to evaluate the value and business impact of future training courses. The Success Case Evaluation Model will require a survey of a sample of trainees who completed the training to evaluate the extent that the recent training made a significant difference to the business. A more extensive follow up with a smaller group of both the most successful and least successful participants will illustrate the business impact (if any) of the training, as well as highlight the factors that enhanced or impeded the business impact. This information will help steer our future course of action in this training program, as well as others in the future. While this approach is summative in nature (occurring after the course is completed), it will provide information as we incorporate a formative evaluation within the process in the future. It is felt the Success Case Evaluation Model approach will quickly provide us with valuable information about the impact of our training with the least amount of disruption to our trainees and managers following the training program.

 

 



3. Success Case Approach Model to be used for Underwriting Training Evaluation:  (See attachment)