Return to CV

Note:    This document was an extra-curricular evaluation
                 made for specific stakeholders in a specific context.

Item Specifications for

 Second Year Spanish Listening CRT

in a Large Language Program

 

Chris Wright Spring 2010

 

 

This document introduces CRT’s and is followed by two item specifications for test items. These can be used in a large language program for a second year Spanish Listening Criterion-Referenced Test (CRT). Each item specification is to be used as a blueprint for generating more test items that are directly related to a course objective of the curriculum. These item specifications measure a student’s mastery of content and objectives found in the chapters 15-16 of the “Sabías que...” Spanish textbook. Each item specification for a test item contains five elements: (1) a general description of the knowledge being measured, (2) a sample test item, (3) stimulus attributes of the material encountered by the students, (4) response attributes of the test item, and (5) specification supplement for clarifying the previous elements.

 

            To further validate the content of generated test items, a rating scale may be created do determine the content applicability (degree that content is appropriate for the given course or program) and content congruence (degree to which an item is measuring what it was designed to assess). This rating scale is given, as a type of survey, to five instructors that teach the same second year Spanish course with the “Sabías que...” textbook. An example of this survey is provided below.

                                               

 

 

                                                                                                Very           Moderate              Very

                                                                                                Poor                                         Good

 

The item is related to the martial that I teach.                                       1           2           3         4           5

Comment: _________________________

 

The item matches with the objective it’s meant to test.             1           2           3         4           5

Comment: ___________________________________

 

 

 

Once a CRT is developed with sufficient content valid test items, a pre-test and post-test may be carried out to see if the test does what it’s supposed to do. The pre-test is given at the beginning of the semester and its distribution should be positively-skewed. After three months of instruction at the end of the semester, the same CRT is given as the post-test. Its distribution should be negatively-skewed. This change in distribution is ideal because it shows a positive effect of instruction. Below is an example of this type of ideal distribution on a histogram.

 

Ideal CRT Distributions

             
                                                                   
                                                 Percent Scores
Text Box: Number of Students

 

There are two quantitative item statistics that may be carried out on CRT’s. One is the B-index, which shows the effectiveness of each item in differentiating the masters from the non-masters in one single test after instruction. The other is the difference index (DI), which does the same but by comparing a pre-test and a post-test, looking for the ideal CRT distributions.

 

It is obvious that the rules and design of CRT’s are not exactly compatible with the rules and design of the ACTFL Listening Comprehension Assessments, which some large Spanish programs use. If a Spanish program does not choose to create a Spanish Listening CRT, these items could still be implemented in an ACTFL exam because they are: (1) content valid with the curriculum in the “Sabías que...” textbook, (2) are graded by machine score for convenience, (3) they may be candidate test items for specific ACTFL proficiency tasks (ex. chapter 15 has a proficiency task level of Novice and chapter 16 has a proficiency task level of Intermediate), and (4) are discrete point, meaning that they measure only listening comprehension and not other cognitive skills.

 

 

Item Specification for Chapter 15

 

 

1. General description

 

These items allow students to show ability to follow spoken directions for arriving to a destination within a city. All content is found in Chapter 15 in the “Sabías que...” Spanish textbook.

 

2. Sample test item:

 

Where did you arrive?

a. La clínica                 c. El gimnasio

b. La escuela               d. La biblioteca

 

3. Stimulus attributes:

 

English instructions before stimulus exposure: “You are about to listen to Spanish directions. Start at the bottom left corner and follow the directions with your pencil until you arrive to the destination. Then, answer the multiple choice question.”

 

Spanish audio: “Siga recto dos cuadras en la calle 3 y pare. Doble a la izquierda y vaya al norte dos cuadras. Después, doble a la izquierda. Siga recto al oeste una cuadra. Cruce la Calle B y siga una cuadra más. Pare. ¿A dónde llegó?”

 

 

 

 

 

La clinica

 

Calle A

 

Calle B

El gimnacio

Calle C

 

   

    N

 Calle 1

 

 

 

 

 

 

 

 

 

 

 

La escuela

 

 

 

 

Calle 2

 

 

 

 

 

 

 

 

La casa

 

 

 

 

La oficina

 

 

Calle 3

Startà

 

 

 

 

 

 

 

 

 

 

4. Response attributes:

 

The map is a drawn on as students listen to the directions. They may track the map with their pencil as if it were a car. Once the destination is determined, the student answers the discrete multiple choice item on a Scranton. The opposite side of this map sheet may also be used as writing space for recall protocols on other test items. This map sheet is returned to the test administrator.

 

5. Specification supplement

 

English instructions must be used first to clarify how the test item works. The Spanish directions need to be said one step at a time with short pauses between phrases. It is important not to fill in all the boxes of the map with destination names because the directions only give a general idea of the correct location. This proposed map would work. Multiple test items, up to four, may be used for this same map size. It is also an option to have a larger map with 5 by 5 blocks, being 25 blocks in total. This would give more destination options for more test items. Also, the word recto (straight) must be used instead of the word derecho (straight). Derecho (straight) may be confused with derecha (right).

 

 

 


 

Item Specification for Chapter 16

 

1. General description:

 

The ability to understand a conversation related to traveling such as; clothing, modes of transportation and types of lodging. Hypothetical phrases are used. Content found in Chapter 16.

 

2. Sample test item:

           

What type of lodging does Paco finally decide to use?

a. Una pensión                        c. Un hotel de cinco estrellas

b. Un motel                 d. Una clase turística

 

3. Stimulus attributes:

           

A photo of Paco and Maria in a travel agency is displayed. Students then hear this preliminary background description in English: “You are about to hear a conversation between a travel agent named Maria and a tourist named Paco. Maria helps Paco organize his vacation. You have 30 seconds to read the questions. (Pauses 30 seconds) Now listen to the conversation then answer the questions.”

 

            Spanish audio sample:

 

            Paco: Hola María, me gustaría viajar al extranjero. Quiero viajar en clase turística para `                              una vacación exótica.

            María: Excelente Paco, te puedo ayudar. ¿Preferías en crucero? Podrías ir al Caribe.

Paco:   Eso no me parece bien. No me gusta ponerme los trajes de baño. También, si yo viajara en un crucero no tendría mucho espacio para alojarme. Las cabinas en estos barcos con pequeñas.

María: Puedo buscarte una cabina más grande.

Paco:   No gracias, es muy caro. Si yo tuviera más dinero, yo compraría un boleto para un crucero elegante, pero es muy caro para mí. ¿Tienes otra idea?

María: Si Paco, ¿Te gustaría viajar al Sur América? Podrías visitar a Perú. Hay mucho que ver allí, como las ruinas de los Inca en Machu Picchu.

            Paco: Interesante, dime más.

            María: Hay vuelos que van para allá cada día. Hay clase turística o primera clase.

Paco: Bien, voy a viajar a Perú por avión, en clase turística. ¿Y qué tipos de alojamiento

 hay?

            María: Hay varias pensiones y hotel de cinco estrellas. ¿Qué te interesaría?

            Paco: Búscame una pensión que sea barata y más económica para mí.

 

Pause for one minute to answer four questions and continues for 4 more questions.

 

 

 

4. Response attributes:

 

            The student answers the discrete multiple choice items on a Scranton.

 

5. Specification supplement:

 

Only two characters with distinct voices speak in the recording. They must be male and female so that listeners may easily differentiate them in the conversation. Test items should avoid prompting listeners to memorize detail points of the conversation. Rather, they should measure the comprehension of the main ideas. The sample recording has enough material to create up to four test items. It can pause for 30 seconds to allow students to answer the items. Then, it continues its second half of the conversation for the next four test items.

 

 


 

Survey for Item Specification of Chapter 15

 

 

 

 

Very           Moderate              Very

                                                                                                Poor                                         Good

 

The item is related to the martial that I teach.                                       1           2           3         4           5

Comment: _________________________

 

The item matches with the objective it’s meant to test.             1           2           3         4           5

Comment: ___________________________________

 

 

 

 

 

 

Survey for Item Specification of Chapter 16

 

 

 

 

Very           Moderate              Very

                                                                                                Poor                                         Good

 

The item is related to the martial that I teach.                                       1           2           3         4           5

Comment: _________________________

 

The item matches with the objective it’s meant to test.             1           2           3         4           5

Comment: ___________________________________

 

 

 

References


J. D. Brown (