IDT 848: Evaluation Abstracts

ArticleSummary Maddrell – Upload a Document to Scribd

Article Reviews Running head: ARTICLE REVIEWS 1 Article Reviews Jennifer Maddrell Old Dominion University IDT 848 Evaluation Study Abstracts Dr. Adcock October 22, 2008 Article Reviews Reference Burton, J., & Aversa, F. (1979). Formative evaluation information from scripts, scratch tracks, and rough cuts: A comparison. Educational Communication and Technology Journal, 27(3), 191-194. Summary 2 Given the significant time and expense outlay involved with television course production, Burton and Aversa (1979) sought to understand how early in the televised course development process the learner content review should occur. While prior research on formative evaluation suggested that review should begin when the instructional product is still “fluid”, Burton and Aversa questioned how useful learner script review is at the early production stage and predicted early stage scripts would be too incomplete for the learner to discern the instructional message. Design Sample and Studied Variables 82 adult learner reviewers were selected from a group of potential students who fit the learner profile for the course. The students were randomly assigned to one of three treatment groups, including those who reviewed (a) the written script alone, (b) the written script and an audio scratch track, and (c) the first rough cut version of the video. The three groups were compared based on both learning outcome, as well as on learner responses to the course material categorized into three areas, including the overall appeal of the program, the learner’s affective responses to the subject matter, and the design of the structural elements of the program. Treatment Members of all three groups provided basic demographic information, including age, education level, subject background, and received the same introduction to the course entitled, Japan II: The Changing Tradition. Those in the script group read through the written script once. Article Reviews The learners in the scratch track listened to an audio recording of a single voice reading all narrations while following along with the written script. The rough cut group viewed the initial version of the video without visual effects or music. After reviewing the materials, the learners completed a 5-point Likert scale opinion questionnaire about the instructional product followed by a short answer test to assess their understanding of the presented material. Analysis and Results 3 The collected demographic information confirmed the groups did not differ significantly. Further, the differences in the mean scores across the three groups for the short answer test were not statistically significant. However, in terms of learner responses to the questionnaire, the mean differences across the three groups were statistically significant. For each learner response measures, the mean scores for the scratch test group were greater than for script group which were greater than for the rough cut group. Critical Summary This study provides support for the use of early scripts and audio scratch tests in high production courses. However, as was most striking to the researchers, the relatively harsh response to the rough cut video appears to contradict prior research. As a possible explanation for the poor learner responses in this study, the rough cut used in this study may have been too rough and too far from the finished representations to allow a viable comparison. Application This study offers support for early evaluation, especially when production time and expense is high and late term revisions would be costly. As this study suggests, learner reviewers are able to discern the instructional message in very early drafts within the development process. Reference Article Reviews Jones, T., & Richey, R. (2000). Rapid prototyping methodology in action: A developmental study. Educational Technology Research and Development, 48(2), 63-80. Summary Citing mixed findings in research literature, Jones and Richey (2000) questioned the 4 effect of rapid prototyping (RP) on instructional design development cycle time, product quality, and customer satisfaction. The purpose of their qualitative study was to gain a better understanding of how RP methods are applied, what the customer’s role is within the RP process, how (if any) concurrent completion of design tasks occur, what (if any) instructional systems process and quality enhancements result, and how customer satisfaction is impacted. Design Qualitative Details and Methodology This qualitative study was conducted at an instructional design firm with 14 employees. Several years prior to the study, the firm adopted a RP process for their custom designs that focused on three milestones, including (a) kickoff involving a customer meeting where roles, responsibilities, and schedules are determined, (b) design freeze when full agreement on product format, content, and instructional strategies is reached between the designers and customer and rapid development occurs, and (c) pilot ready when the product ready for learner pilot testing. The activities of two senior instructional designers on two separate projects, as well as one client contact per project, were examined. Both projects were one-day instructor led classes, but were delivered using different media. Data collection included reviews of designer task logs and other project data, as well as personal interviews. Data analysis focused on the nature of the RP process, attitudes about RP and the product, cycle time, and overall customer involvement. Analysis and Results While the projects were completed fairly linearly, especially in the final stages, the data Article Reviews analysis revealed that the 14 key tasks prescribed by the firm’s RP model were performed for each project with concurrent processing occurring in the completion of 10 out of the 14 key tasks. Work time varied between the two projects at the task level, but total work time was similar at 79.25 hours for Project 1 and 74.0 hours for Project 2. Both the designers and customers perceived reduced cycle times as compared with traditional instructional design. The researchers noted the relatively high degree of customer interaction in both projects. Customers were actively involved in (a) analyzing the training needs, (b) providing input, feedback and approval of content, learning activities, and the prototype, and (c) participating in the pilot. Given that learner achievement data was not collected, the researchers focused on satisfaction (of the designer and customer) and usability to the end customer. Satisfaction with 5 the project was high for both the customer and designers. Further, both projects were put into use immediately after delivery to the customer and were in use one year after which the researches deemed to be a measure of product quality from an external consultant’s perspective. Critical Summary Given the results of this limited qualitative review, the chosen RP design process resulted in acceptable production cycle time, a usable instructional product, and a satisfied customer; all good outcomes for an instructional design consultant. However, without a measure of learning outcome, effectiveness was not fully evaluated. Further, it is possible that the relatively high degree of stakeholder involvement during the entire instructional design process, not simply RP process changes, was the key factor in the project’s success. Article Reviews Application The most significant outcome of this study is the reinforcement of the need for frequent communication and buy-in from the stakeholder. As suggested in this study, a project will run 6 efficiently and result in a more satisfactory outcome if there is open communication, stakeholder input, and agreement on key decisions throughout the design process. Reference Brown, K. G. (2005). An Examination of the Structure and Nomological Network of Trainee Reactions: A Closer Look at ‘Smile Sheets’. Journal of Applied Psychology, 90(5), 9911001. Summary Brown (2005) studied training reaction from an affect-based theoretical framework examining affect as a subjective state that is either positive or negative. Based on cited prior theory and research, Brown predicted: (a) affective training experiences create an overall evaluation of satisfaction which in turn influences specific reactions to the training; (b) content interest is positively related to reactions; (c) learner personality traits and orientations are related to reactions; (d) media aesthetic appeal influence reactions, and (e) reactions and learning are related. Design Sample and Studied Variables Two studies were held to examine these predictions. In the first study, 178 undergraduate business students and 101 graduate business students volunteered to answer a survey regarding a pre-recorded videotaped lecture with 64% and 58% response rates, respectively. The second study included 97 undergraduate business students who were randomly assigned to one of three groups who viewed the same lecture presented via three different technologies, including (a) a Article Reviews 7 computer-delivered presentation, (b) audio and print, and (c) video, audio, and print. Between the two studies, a host of measures were evaluated and compared, including learners’ computer experience, personality traits (extraversion, neuroticism, and openness), mastery and performance goal orientation, content interest, technology satisfaction, perception of relevance, enjoyment, overall satisfaction, engagement, and learning. Treatment In the first study, learners viewed a videotaped lecture and took a survey assessing content interest and reaction. In the second study, participants viewed the identical instruction, but via the noted technologies. At the lecture’s midpoint, a brief engagement survey was conducted. After the lecture, learners completed a reaction survey, an intention questionnaire related to future use of the technology, and a 25 item multiple choice knowledge test. Analysis and Results From the first study, a factor analysis suggested that (a) reactions are related, (b) overall satisfaction is a good predictor of other reaction measures, and (c) attitude (interest) and disposition (master goal orientation) predict reactions. Within the second study, a multivariate analysis of covariance with ACT score as the covariate showed statistically significant differences in reactions across delivery technologies with audio conditions having statistically lower satisfaction measures. In addition, regression analysis suggests reactions can predict engagement, intentions, and learning. Critical Summary While Brown’s paper presents an intriguing affect-based theoretical framework for the study of trainee reaction, he acknowledged the conflict between his research findings and prior research, especially within the suggested relationship between reaction and learning. It is Article Reviews 8 troubling that Brown is satisfied that his findings from these very short and limited single session interventions can challenge this extensive body of prior research. Application Brown’s affect-based theoretical framework is an intriguing basis for future research on trainee reactions. If his findings are correct that an overall satisfaction measure is a predictor of other reaction measures and that reaction can predict engagement, intentions, and learning, reaction surveys could be streamline to just a few items addressing overall satisfaction with the experience. Reference Kandaswamy, S., Stolovitch, H., & Thiagarajan, S. (1976). Learner verification and revision: An experimental comparison of two methods. Audio-visual Communication Review, 24(3), 316-328. Summary Kandaswamy, Stolovitch, and Thiagarajan (1976) report on one in a series of studies on learner verification and revision (LVR). Noting increased advocacy and use of learner feedback during formative evaluation, the researchers assess the generalizability of prior studies which support LVR and compare the effectiveness of tutorial LVR and group-based LVR methods. Design Sample and Studied Variables 140 eighth grade girls were randomly selected from two different schools in India. 60 girls were randomly assigned to the LVR group while the remaining 80 were included in a final summative comparison. Four teachers from the schools were randomly selected as evaluator / revisers. The studied LVR methods included (a) tutorial LVR in which the evaluator / reviser probes and monitors the learner’s nonverbal and verbal feedback while the learner completes the Article Reviews 9 material and (b) group based LVR in which the evaluator / reviser analyzes patterns of errors and predicts causes after the learner completes the material. The 60 assigned to the LVR group were stratified based on prior math achievement and one top, average, and poor student was randomly assigned to each of the four evaluator / revisers for the tutorial LVR treatment. The remaining 48 were randomly assigned to the four evaluator / revisers for the group-based LVR treatment. Treatment For the 48 learners in the group-based LVR treatment, a proctor administered a printed self-study instruction booklet which contained a pretest, instruction, and posttest. Upon completion, each of the two group-based evaluator / revisers took 12 booklets each and made independent group-based revisions. The other two evaluator / revisers conducted separate tutorial LVR sessions with the 3 students assigned to them and made independent revisions from the evaluation. In a second phase, the review and revision process was reversed. These two phases resulted in a total of eight revised versions and allowed an evaluation of the order in which the methods were used. The 80 students in summative comparison group were randomly assigned to one of eight groups and completed one of the revised pretest, instruction, and posttest materials. Analysis and Results A one-way analysis of variance of the posttests of the original and eight revised versions show statistically significant differences between each of the eight revisions and the original which supports the research prediction that learner review and revision improves the instructional material. However, there was no significant difference in outcomes between the tutorial and group methods of LVR or from the order in which the methods were used. Yet, the revisions by different evaluators and revisers did have different degrees of effectiveness supporting the prediction that not all revisions by evaluators are of equal value. Article Reviews Critical Summary This study is significant in that it suggests support for conducting learner based review. 10 Further, the findings suggest that the evaluation method and order of use of different methods are less important than the person chosen as the evaluator and reviser. Application These findings suggest that not all evaluators are equally effective at evaluation and revision. Therefore, an evaluator quality control process should be contemplated which includes a cross-comparison of revision recommendations across multiple reviewers and evaluations of the evaluators. Reference Medley-Mark, V., & Weston, C. B. (1988). A comparison of student feedback obtained from three methods of formative evaluation of instructional materials. Instructional Science, 17(1), 3-27. Summary Twelve years after the 1976 study by Kandaswamy, Stolovitch, and Thigarajan discussed above, Medley-Mark and Weston (1988) sought to quantitatively and qualitatively compare the data collected from one-to-one and small groups during learner verification and review (LVR). Given the lack of research on the characteristics of the data collected, the stated purpose of the study was to examine the identified student problems across various LVR conditions. Article Reviews Design Mixed Method Details and Methodology 24 students volunteered to participate in the evaluation of two modules in their undergraduate educational media course. The volunteers were stratified based on grade point average. From this, six students were selected based on availability and assigned into one of 11 three groups, including a one-to-one group with the student with the highest grade point average (GPA) (1-1), a small group of two students with comparable mid-range GPA’s (1-2), and a small group of three students from each of the three GPA levels (1-3). All six students participated in two sessions. In the first, the unmodified version of the prior year’s one hour lecture was given by the instructor followed by Assignment 1 which the learners completed based on their assigned condition, as discussed below. In the second session, learners viewed a video-taped lecture followed by Assignment 2, again completed based on their assigned condition. The two print based assignments included short-answer and essay exercises which were graded and evaluated. In the 1-1 session, the evaluator assumed a passive role while the learner was encouraged to think aloud during assignment completion. Students in the 1-2 session were encouraged to discuss encountered problems as they individually completed the assignments. Those in the 1-3 group were instructed to work independently and passively. After the learners completed the assignments, the evaluator in each group conducted a debriefing session with prepared questions. Analysis and Results Overall, the 1-1 condition identified the most number of problems with the greatest detail, but the evaluation process involved the most time and effort on the part of both the learner and evaluator. The 1-2 group identified the second highest number of problems, but evaluated the Article Reviews product in conditions that were most similar to actual use. The 1-3 group identified the fewest 12 number of issues, but their time to complete the assignment was closest to actual use. In addition, the groups differed based on the types of problems identified. The 1-1 and 1-2 conditions focused heavily on problems associated with the situation statements while the 1-3 group focused on the problem or the problem choice within the assignment. Critical Summary This study is valuable for its qualitative comparison of the one-to-one and small group learner evaluation. While no clear cut winner is established, that was not the point of the study. Rather, the study suggests a comparison and tradeoff between efficiency and effectiveness. While one-to-one evaluation may provide the greater efficacy in terms of problem identification, it will come at a higher cost in terms of time and energy on the part of the learner and evaluator. In contrast, the small group learner evaluation may offer a more efficient and practical evaluation method, but may not offer the most breadth and depth of problem identification. Application This study suggests that evaluators need to consider the practical efficacy and efficiency tradeoffs and implications when creating an evaluation plan. While one-to-one evaluation may lack efficiency, greater efficacy may result in terms of breadth and depth of problem identification. In contrast, small group may offer a more efficiency when the evaluator does not have the luxury of time and budget to run a series of one-on-one learner reviews, but fewer issues may be identified.