Flexible Test Delivery System, NSF

Flexible Test Delivery System

The research was conducted from January 1, 2007 to June 30, 2007. It was funded by the National Science Foundation.

Major Research Activities and Findings

The primary goal of our Phase I project was to develop and pilot a flexible test delivery system (FTDS) that was capable of providing students with several accommodations. These accommodations included Read Aloud of text, Magnification of test items, Masking of test items, and access to a digital talking calculator. In addition, the project aimed to develop an interface that would enable teachers to activate or deactivate specific tools for individual students depending upon their accommodation needs.

An initial pilot test was conducted with a small group of students in Tallahassee, Florida.

In late April, the system was pilot tested a second time with a sample of 31 students with various special needs. Eighteen students attended school in Panama City, FL and thirteen attended school in Lake City, FL. In this pilot study, students were administered two short 10 item tests. Nine of the items were multiple-choice and one was open-ended. One test was taken on paper and one was taken using the FTDS. The tests were composed of released items from the NH 10th grade Mathematics test. The tests were designed to be parallel in content and were identical in terms of difficulty. The purposes of the pilot study were: a) to have students contrast their experiences taking a traditional paper test and a test delivered using the FTDS; b) to perform preliminary comparisons of performance on paper versus FTDS; c) to solicit feedback from students on the FTDS; and d) to pilot and receive feedback on the teacher control interface (used to activate and deactivate accommodation features for individual students).

Overall, the pilot testing was a very positive experience for all involved. Teachers found the control interface very easy to use and were able to set appropriate accommodations for all students. Although many students felt the content of the test was too difficult, the vast majority of students reported that the FTDS was easy to use and expressed a preference for using it over the traditional paper-based approach to testing. Although the sample size was relatively small, the majority of students reported that they felt they performed better on the computer than on paper. A comparison of their performance on the multiple-choice items provides evidence that students, on average, did score higher when taking the test on computer. Whereas the mean score when taken on paper was 26%, the mean using the FTDS was 35%. This difference was statistically significant (p=0.03).

Following the test administration in each school, a focus group was held with students. In the focus group, students in both schools overwhelming agreed that they preferred using the FTDS instead of the paper version. One student stated that he found the text small on paper. In contrast, the text could be magnified in the FTDS. Several students stated that they liked the review features (both mark for later review and the ability to review answers before ending the test). Although several students reported that the initial sound volume was too loud, they liked the ability to control the read aloud and preferred it over having a teacher read the test to them. Some students, however, stated that they would like to be able to change the pace/speed of the reader. Some students also indicated that they would like to change the voice of the reader.

Overall, however, students expressed strong enthusiasm for the system stating such things as: “The program was fantastic.” “I wouldn’t improve any of it. The program is great.” “It was awesome!” “Thought it was good the way it is.”

Analyses of the survey also indicated that students were very positive about the FTDS and favored it over the traditional paper-based method of test delivery. When asked if they would prefer to take a math test on paper or a computer using a system similar to the FTDS, 79% of students said they would prefer using a computer, 7% said they would prefer paper, and 14% said they had no opinion. When asked how helpful each of the accommodation tools were, 75% of students who used the Read Aloud tool reported that it was helpful or very helpful, 60% found the Magnifier helpful or very helpful, 25% found the Microscope helpful (note that none of the students had severe visual impairments), and 82% found Item Masking helpful or very helpful. Similarly, 100% of students who used a given accommodation tool reported that it was easy to use, with the exception of the Microscope for which 80% found it easy to use.

Completed June 30, 2007