turekformrevise

Turek Home

=** Formative Evaluation ** =

=**Introduction** =

**Goal**

The goal of this instructional unit is for instructors to be able to use Microsoft Office PowerPoint 2007 and Camtasia Studio 7 PowerPoint Add-in to 1) set up and test a microphone to record audio 2) record a narrated PowerPoint presentation reading a sample printed script aloud into a microphone and 3) save a recorded PowerPoint presentation to a video file format ready for online delivery after taking individual self-directed direct instruction in the form of an instructional handout with supporting practice files.

**Rationale**

Using Microsoft Office PowerPoint 2007 PowerPoint presentations is a ubiquitous method for delivering instructional materials to Kent State University undergraduate and graduate students taking online courses. Instructors currently utilizing this method of delivery for instructional materials have recognized a need to enhance their PowerPoint presentations with narrated audio to provide online students with a more meaningful learning experience. Some instructors have chosen to use Camtasia Studio 7 with PowerPoint Add-in, a university recommended and supported technology, to create narrated PowerPoint presentations for online delivery of course materials. Current instructional materials for how to create a narrated PowerPoint presentation using Camtasia Studio 7 lack specific step by step instructions detailing the entire process, including how to set up and test a microphone to record audio and how to save a narrated PowerPoint 2007 PowerPoint presentation to a video file format ready for online delivery.

**Product Evaluation**

The design strategy and resultant development of instructional materials, including instructional handout and supporting sample practice files, are being evaluated for their potential effectiveness of transferring practiced knowledge and skills during the course of instruction to the post-assessment. Potential effectiveness of transfer of knowledge and skills from the instruction to the post-instructional assessment is measured by the instructor's ability to successfully complete the necessary sequence of performance objectives to create a narrated PowerPoint presentation without assistance or use of instructional materials.

=**Methods** =

**Test Subjects**

Test subjects consisted of three instructors. Two instructors are currently faculty members at Kent State University and one instructor is a teacher at Greenview Elementary School. All instructors currently use PowerPoint presentations in their classroom activities, although frequency of use and proficiency levels in PowerPoint varied (this was self-reported by the test subjects). Of the three volunteer test subjects, one expressed an immediate need to learn the skills as motivation to participate, the other two reported curiosity and desire to learn new technology skills as motivating factors. All three subjects exhibited characteristics consistent with the target audience, i.e. intrinsically motivated instructors seeking individual self-directed direct instruction to further professional development and/or meet professional responsibilities.

The selection of the third instructor, the elementary teacher, was deliberate in order to evaluate the effectiveness of the instructional materials with a test subject familiar with PowerPoint but less familiar with setting up and testing audio devices to record audio using a computer. The decision was taken after both KSU faculty members reported and demonstrated prior knowledge and experience working with audio devices in a variety of computer applications. While this may not be uncommon for university faculty, it might not be a true representation of the normative technology skills across the faculty population at KSU.

**Trial Run Procedures**

Trial run of the instructional materials were completed in two sessions, A and B.

//Session A// The first trial run session of the instruction was conducted at Kent State University in the Educational Technology department. The test subjects were two KSU faculty members. They arrived together and took the instruction separately during the same time period. Prior to their arrival, two testing stations and computers were prepared. Preparation included booting the computers into the operating system, checking the required computer applications were installed on each computer, transferring the instructional and post-instructional sample files onto the computer desktop, and placing the equipment (USB headset with microphone) and instructional handout and sample narrative script on the desk at the workstation. Upon their arrival, they were given a brief orientation and informed of the schedule of events.

First, they were each administered a pre-requisites test. Results of the pre-requisites rest were recorded. Upon completion of the pre-requisites test, they took the instruction. During the instruction they were observed and allowed to ask for help if they became stuck and could not proceed. Upon completion of the instruction, all applications and windows on the desktop were closed and the USB microphone was unplugged from the computer. The instructional handout was placed out of reach and the test subjects were presented with directions for the post-instructional assessment. During the post-instructional assessment they were observed and their performance was recorded. They were not allowed to use the instructional handout during the post-instructional assessment, however, were allowed to ask for help if they became stuck. The assessment performance was scored using the evaluation sheet. Upon completion of the post-instructional test the students were informed the instruction was finished.

Following the completion of the instructional unit, a brief verbal survey was taken of their impression of the instruction including constructive criticism and suggestions for improvements/additions to the final product.

//Session B// The second trial run session trial run of the instruction was conducted in my home office. The test subjects was an elementary school teacher. Prior to the teacher's arrival, a testing station and computer was prepared. Preparation included booting the computer into the operating system, checking the required computer applications were installed on each computer, transferring the instructional and post-instructional sample files onto the computer desktop, and placing the equipment (USB headset with microphone) and instructional handout and sample narrative script on the desk at the workstation. Upon the teacher's arrival, he was given a brief orientation and informed of the schedule of events.

First, a pre-requisites test was administered. Results of the pre-requisites test was recorded. Upon completion of the pre-requisites test, the instruction was taken. During the instruction the teacher was observed and allowed to ask for help if he became stuck and could not proceed. Upon completion of the instruction, all applications and windows on the desktop were closed and the USB microphone was unplugged from the computer. The instructional handout was placed out of reach and the test subject was presented with directions for the post-instructional assessment. During the post-instructional assessment the teacher was observed and performance was recorded. The teacher was not allowed to use the instructional handout during the post-instructional assessment, however, was allowed to ask for help if he became stuck. The assessment performance was scored using the evaluation sheet. Upon completion of the post-instructional test the students were informed the instruction was finished.

Following the completion of the instructional unit, a brief verbal survey was taken of their impression of the instruction including constructive criticism and suggestions for improvements/additions to the final product.

//Modifications to instruction//


 * The Pre-instructional assessment was dropped. Due to the time constraints the pre-instructional assessment was determined to be less important for evaluating the instruction than the pre-requisites test and post-instructional assessment.


 * Session B required the help of an additional tester. The test subject were only able to participate if they could both take the instruction at the same time. The second tester was provided with written directions for administering the pre-requisites test, instruction, and post-instructional assessment one day prior to administering the instruction for review. The second tester was provided with evaluation sheets to record observations and score both assessments. The main tester was present in the same room during the same time period several feet away and available to help in the event of any questions during the administration of the instruction.

= = =**Results and Discussion** =

**Results**
 * || Pre-requisites Test || Pre-instructional Assessment || Post-instructional Assessment ||
 * Test Subject #1 ||= 8/8 ||= NA ||= 112/113 ||
 * Test Subject #2 ||= 8/8 ||= NA ||= 112/113 ||
 * Test Subject #3 ||= 8/8 ||= NA ||= 113/113 ||

All three test subjects completed the Pre-requisites test within 5-8 minutes. All three test subjects completed the Instructions with 50 minutes. All three test subjects completed the Post-instructional assessment within 10 minutes.

**Observations**

Pre-requisites test

All three of the test subjects completed the pre-requisites test without questions or prompting from the instructor. Each of the tasks was performed correctly without hesitation.

Instructional materials

The following comments, observations, issues were encountered during the instruction (not necessarily in the order listed below):
 * A mouse was set to left hand user, test subject was right handed. Correction was made and instruction continued.
 * Confusion at performance objective #1.3.7 & 1.3.7a. about whether or not microphone needed to be activated.
 * Question on pronunciation of word in narrative script at performance objective #2.2.9.b.
 * Hesitation at at performance objective #3.4.9, screenshot depicts top half of window and test subject instructed to click on a button on bottom of window not depicted (though the button is depicted).
 * Question about whether or not the test subject would hear their own voice in the headset while speaking into the microphone.
 * Comment/question about lack of instructions to leave on headset or put headset back on to listen to the preview of recorded narrated presentation.
 * Question about terminology used to describe results of dangerous audio level - "clipping".
 * Question about clarifying audio level color bar indicator levels - good vs. bad
 * Issue with performance objective substep 3.4.3, hovering mouse over button results in pop-up - test subjects instructed to click on button. Pop-up box confused test-subjects resulting in trying to first click on selection in pop-up and not on button.

Post-instructional assessment

All three test subjects performed extremely well on the post-instructional assessment. One performed all steps correctly without any need of assistance, two got stuck at performance objective 3.4.3 and required assistance from the tester before correctly finishing the remainder of the assessment.

Post-instructional discussion with test subjects

Positives - All three test subjects described the instructional materials as being sufficiently detailed and logically structured for them to easily learn how to create a narrated PowerPoint presentation for online delivery. They stated the screenshots, introductory, and review sections of the instructions were very helpful for staying focused on task and reinforcing/summarizing each cluster of instruction. They also stated the positive messages at the end of each cluster were a nice touch.

Negatives - One test subject commented that the colors/definition of the screenshots did not accurately match the computer screen. Another thought the elaborative text for the main sub-steps within each of the clusters was too wordy. One of the test subjects also commented on a possible contradiction in the instruction, the fact the introduction states they can work at their own pace but the three main parts of the instruction have approximate times for completion listed.

Suggestions - Provide pronunciation guide for any difficult word(s) in the sample narrative scripts, explain terminology "clipping", create a bare-bones job aid as a reference/reinforcement resource to accompany instructional materials, create a video overview of the process that can be viewed prior to taking instruction.

Reflection

Data from the pre-requisites test indicates the target audience possessed the entry level knowledge and skills necessary to take the instruction. Data and observation from the instruction and post-instructional assessment indicate the instructional materials effectively transfer the knowledge and skills practiced during the instruction to performance on the post-instructional assessment. Comments, questions, and observations of the test subjects indicate that though revision to the instructional materials is necessary to help clarify the instructional materials, no essential content or instruction is lacking. Although the test subjects were found to be representative of the target audience, additional testing upon revision of the instructional materials is needed to evaluate the effectiveness of the instruction within a larger population of the target audience representing a greater variance of technology skills.

[ Top ]