Pietilä P., P. Lintunen & H.-M. Järvinen (toim.) 2006. Kielenoppija tänään – Language Learners of Today. AFinLAn vuosikirja 2006. Suomen soveltavan kielitieteen yhdistyksen julkaisuja no. 64. Jyväskylä. s. 69–85.
––––
WILD GUESS, LUCKY GUESS, GOOD GUESS – HAZARDING AT A MULTIPLE-CHOICE TEST OF LISTENING
COMPREHENSION
Joanna Anckar University of Jyväskylä
Inherent in the nature of (foreign) language understanding are the strategies of guessing, anticipating and inferring. When the context is a testing situation where listening comprehension in a foreign language is measured by a multiple-choice test, the factor of guessing appears particularly prominent, as strategic abilities are added to linguistic skills. In their introspective responses collected in a validation study of a test of listening comprehension of French as a FL, many test candidates stated guessing as their reason for selecting a particular option. It seems, however, clear that guesses can be placed on a continuum, where a “wild guess” – when neither the spoken input nor even the written options have been understood – is at one end, and an uncertain choice – when some input, if not all, has been grasped – is at the other end. This presentation proposes to illustrate the nature of this guessing continuum on the basis of introspective responses.
Keywords: multiple-choice tests, test-taking strategies, guessing
1 INTRODUCTION TO THE PROCESS OF GUESSING
Let me start by giving you a multiple-choice question to answer.
Please select a response among the following five options; only one is correct.
In what kind of building does Joanna – the author of this article – live?
a) A bungalow b) A castle c) A farm house d) A terraced house e) A sky scraper
Have you made your choice? My next question is: how did you reach your response? Did you know? Or did you simply guess?
And, if you guessed, was it a blind guess – did you just pick any of the options? Or, were you able to rule out one or more of these options on the basis of some information or knowledge you possess – for example by being familiar with typical housing options in Finland? My point is that when facing multiple-choice questions of any kind, even if you do not know the response for sure, you will only rarely meet with a situation where the guess you have to make is completely random. More often than not you will have some background information that might, for instance, tell you which options are definitely not correct, which is also valuable information. As Buck (2001: 147) points out, test-takers normally have some comprehension of the point being tested. Implausible distractors can be eliminated on the basis of partial knowledge (cf.
Haladyna 1994: 152; Bachman & Palmer 1996: 204). Therefore, even if the test-takers don’t know the correct answer, they are inclined to favour one response over another, perhaps without even knowing exactly why. Bachman and Palmer (1996: 205) relate the test-taking situation to a real language use situation, saying that if
make a totally random guess at meaning; rather, we use the means at our disposal – language knowledge, topical knowledge and metacognitive strategies – to arrive at a possible understanding of the meaning.” Linn and Miller (2005: 343–344) point out that problem solving always involves a certain type of informed guessing. They compare the guesses at doubtful items with
“informed guesses we make when we predict weather” or “judge the possible consequences of a decision”.
2 MULTIPLE-CHOICE TESTS AND THE PROCESS OF GUESSING
For my PhD project I have analysed a multiple-choice test of listening comprehension used at the Finnish Matriculation Examination in French as a Foreign Language (at the Common European A2/B1-level approximately, and at the Finnish B2/B3 level) in order to find out its relative validity as an assessment tool of listening comprehension. For the purpose of triangulating the sources of validity evidence, I have developed a qualitative method that I call “short written introspection”. In a test situation, test- takers are asked to motivate, at each item, their reasons for selecting a particular option.
In my analysis of the test-taker’s responses, I have repeatedly been struck by the multidimensionality of the reasons for guessing.
Among the factors affecting the test-taking process there are the spoken text as one dimension, the task including the written question (the stem) and the three options as another and as the third dimension the candidates with their various personalities, skills and knowledge (language and other) and experiences. All these dimensions influence the test-takers’ tendency to guess at an item (cf. Bachman
& Palmer 1996: 204–205).
The multiple-choice format of testing listening comprehension is often criticized for the risk of allowing test-takers using the
strategy of random guessing to arrive at the correct option without actually understanding the spoken input at all. Another possible threat against the validity and the reliability of a test is at hand when test-takers who do understand the spoken input do not manage to select the correct option among three or four propositions.
As one of the drawbacks of the multiple-choice test format Alderson et al. (1995: 45) mention that students can apply strategies in such tests that “artificially” inflate their scores, namely techniques for guessing the correct answer. Moreover, the multiple-choice test method encourages students to consider alternatives they would not otherwise have considered – thus the technique tricks the unwary into making incorrect interpretations they might not otherwise have made.
The situations where a candidate relies on guessing do not, however, just reveal something about the nature of the test format, but also – and possibly to a greater extent – about the quality of a particular test. We have to consider the possibility of construct- irrelevant variance, which implies that factors outside the construct being measured affect the outcome of the test, thereby decreasing its validity. These factors might for example relate to the fact that there are items that some test-takers who have not understood the spoken text can answer by selecting an option mainly through specific background knowledge, not shared by all test-takers.
Another factor may be specific vocabulary knowledge in the written options that is demanded of the test-takers in order for them to be able to give an answer.
Any selection-test such as the multiple-choice test contains an element of guessing (cf. Haladyna 1994: 152). The interesting points to investigate are, first of all, how this strategy is used and, second, why the candidates use it. These two questions are interrelated and partly dependent on each other.
3 THE STUDY
For this tentative dive into the mysterious and murky waters of guessing, I shall consider the first part of the test under study, consisting of eleven items altogether. The test procedure is as follows. Test-takers first read through the 2–3 written questions (consisting of a stem and three options) relating to a specific spoken text passage. They listen to the text passage and, during a pause, select the option they consider appropriate and read through the following questions. Then they listen to the following text passage.
After having answered all eleven items, they return to item 1, reading through the question and listening to the text a second time.
At this point, they can change their original selection if they wish.
During the short pause, they are asked to briefly justify their opti- on selection writing down their reason in a box next to each item.
With 226 candidates tested, the introspective responses amount to 2486. I have picked out the responses where the candidates have explicitly written down the verb or the noun “to/ a guess”, or some word similar in meaning, in their mother tongues (Finnish or Swedish: arvasin, arvaus, arvoin, veikkaus; gissade, gissning), yielding 648 responses in all to focus on.
The majority of these guesses (ranging from 26 to 93 cases per item, resulting in a total number of 489) consist of just the noun or the verb, not giving away anything else about the nature of the strategy. However, the proportion of correct choices among these guesses is interesting: N= 222, which amounts to 45 % of the cases.
The likelihood of getting the correct choice by random guessing among the three options is 33 %. Consequently, 45 % should be taken to indicate that some of these choices are in fact based on some degree of understanding of the text.
In 158 cases the candidates have given some kind of explanation of why or how they have guessed. I have been able to identify ten different types of guessing altogether. On the basis of these responses, I have set out to establish a continuum of guesses,
from the wildest to the good ones. It is fair to assume that the 489 guesses without any indications of a reason for the use of the guessing strategy could also be placed somewhere on the continuum – many good guesses are likely to belong to the “informed” end of the scale. In the following, each of the ten types of guessing are presented and exemplified.
3.1 TYPE 1 GUESSING: MISSING THE SPOKEN INPUT
From the point of view of test validity, the cases representing non- understanding or missing the spoken input (due to problems in segmenting, parsing, recognizing, grasping or understanding) and resulting in more or less completely random guessing could be said to be acceptable. With three options to select among, this kind of guessing will, from the statistical point of view, lead to the selection of a distractor in two cases out of three. The test-taker has practically nothing to base his or her selection on. A problem from the point of view of test score reliability is, however, the cases where this uninformed candidate happens to select the key option. A construct- irrelevant case occurs if this guessing or selection is not random, but based on some information in the written options giving away the correct response. In an authentic interactive language use situation the listener can in most cases use the interlocutor or some factor other than the auditory information to cover for the lack of catching a message immediately. However, in a non-interactive test situation, if the options give themselves away by their wording, and candidates can rely exclusively on test wiseness, comprehension of the heard message is not assessed by the test.
29 candidates have given non-understanding as a reason for guessing. In more than half of the cases (55 %) they have chosen the key. Have some of these candidates understood more than they believe or have they just been lucky? Examples of the first type of guessing include:
(1) Ihan silkka arvaus, meni vähän ohi… ‘A complete guess, I missed this a little…’ (Candidate code: V7:5)
(2) Ihan pelkkiä arvauksia, liian huono sanavarasto niin ei ymmärrä puhetta ’Just pure guesses, too narrow a vocabulary to understand what is said’ (P1:4)
(3) Arvaus, puhui liian nopeasti ja epäselvästi ’ A guess, he talked too fast and too unclearly’ (X3:5)
(4) En oikein ymmärtänyt mitään, joten arvasin ’I hardly understood anything, so I made a guess’ (S1:2)
(5) Arvasin, en kuullut ‘I made a guess, I didn’t hear’ (Z8:5) (6) En saanut selvää, arvaus ’I couldn’t make it out, a guess’ (T1:1)
3.2 TYPE 2 GUESSING:
MISINTERPRETATION OF THE SPOKEN TEXT
A few candidates (4) have guessed on the basis of a misinterpretation of the text contents. If a test works optimally, such misinterpretations should lead to the selection of a distractor, which is the case with half of these responses. In two cases, the misinterpretations match the contents of the key or at least appear to be based on the key option. This situation can hardly be completely avoided when the multiple-choice format is used, even if the quality of the options is good.
(7) Kaupungissa ihmiset luovat enemmän suhteita kuin maalla (osit- tain arvaus) ‘In a town people establish more relationships than in the country (partly a guess)’ (K14:6)
(8) Arvaus, hänellä oli allergia? ’A guess, she had an allergy?’ (N11:9) (9) Se maksu kai aleni tai jotain, joten arvaan A:n ’The fee went down
or something, so I’m guessing A’ (T1:1)
(10) Arvasin. Sitä paitsi sanottiin vain ettei se asu yhdessä äitinsä kans- sa. Ei mitään siitä näkeekö hän äitiään koskaan ’I made a guess.
Moreover, they only said that she doesn’t live with her mother.
Nothing about if she ever saw her’ (U6:7)
3.3 TYPE 3 GUESSING:
SINGLE-WORD COMPREHENSION
The following type of responses includes the cases where the candidates have caught one or two single words from the spoken
input. They have, however, found this insufficient to base an interpretation of the input and the selection of an option on, and have made use of the strategy of guessing. We have in all 22 cases of this type among the responses to the eleven items under scrutiny.
It is evident that we have already moved some way away from random guessing. The selection is an individual “best guess” of some kind, based on incomplete comprehension but not being random. Unfortunately this has not proven a good strategy for the test-taker: only one fifth of the candidates have selected the key.
From the point of view of the test constructors, the test probably works as intended, since a typical technique for constructing distractors is to pick a word from the text and place it in the wrong context to attract weak candidates. The following examples show this type of response:
(11) Puhuttiin rahasta…arvaus ‘They talked about money... a guess’
(J11:8)
(12) Arvasin. Puhuttiin “sosiaalisesta ongelmasta”. ‘I made a guess.
They talked about a “social problem”’(X2:5)
(13) Puhuttiin turisteista ja valokuvista? osittain arvasin’They talked about tourists and photographs? Partly a guess’ (V2:3)
(14) Arvaus, toisella kerralla puhuttiin poliisista ’A guess, the second time they talked about the police’ (G1:3)
(15) Kuulosti parhaalta (=arvaus). Hän puhui investoinneista ’Seemed the best one (= a guess). He talked about investments’ (G1:3) (16) Kuulin sanan ravintola ja arvasin ’I heard the word restaurant
and I made a guess’ (T3:4)
(17) Puhu jotain postista (arvaus) ’Said something about mail (a guess)’
(Z4:4)
(18) Arvasin. Puhuttiin nukeista ja leikkimisestä ’I made a guess. They talked about dolls and playing’ (X2:5)
3.4 TYPE 4 GUESSING: PARTIAL COMPREHENSION
Despite partial comprehension of the spoken text, some candidates still feel they have to rely on guessing in their selection of an opti- on. The candidates have actually understood bits and pieces of
be able to discard the distractors and pick the key. In fact, another typical item constructing strategy is to have one of the distractors attract candidates that have understood some, but not all of the essential information or message in the text. Among the responses given to the first eleven items, we have thirteen cases of evidenced partial comprehension. Nearly half of these test-takers have arrived at the key – their success probably depending on exactly what part of the text they have understood: some detail in the central message or some secondary piece of information. The following responses show partial comprehension:
(19) Puhuttiin kauppojen sulkemisesta. (Arvaus) ’They talked about closing down shops. (A guess)’ (O4:3)
(20) Han berättar att han jobbar ute. Gissade. ‘He tells that he works outdoors. A guess. (D2:4)
(21) Arvaus, halusi olla rauhassa ‘A guess, wanted to be alone’ (S8:6) (22) Arvasin puoliksi, puhui jotain painon putoamisesta ’A half-guess,
said something about losing weight’ (N8:7)
(23) Ljud störde henne, det lät som någon av dehär, gissning ’Noises bothered her, it sounded like one of these, a guess’ (H7:7) (24) Har två döttrar. Gissning. ‘Has two daughters. A guess.’ (H6:6) (25) Puhuttiin maalauksista ja sitä, että vanhemmat pelkäävät varkaita
(arvasin) ‘They talked about paintings and of the parents being afraid of thieves (I made a guess)’ (F3:5)
3.5 TYPE 5 GUESSING: UNSURE CANDIDATES
There are candidates who, perhaps due to personality factors, have simply not felt certain enough about their selection of an option and have therefore called their strategy “guessing”. Sometimes they have actually understood most of the text but for some unclear details. Their understanding may, nevertheless, be sufficient for selecting the key. Of these 9 cases, one third have chosen the key.
Two choices of a distractor and two correct choices are exemplified by the following:
(26) En ollut yhtään varma, mutta arvasin ’I wasn’t sure at all, but I guessed’ (S1:2)
(27) Arvaus, mutta jotain tietoa sinnepäin ‘A guess, but something like that’ (J3:3)
(28) Arvasin, kun en ollut varma ’I guessed since I wasn’t sure’ (Z7:4) (29) Ei ihan täysi arvaus, mutta ei kovin varma vastauskaan ’Not a
complete guess, nor a sure answer either’ (G6:8)
3.6 TYPE 6 GUESSING:
PROBLEMS WITH THE OPTIONS
Cases where the validity of some of the items is threatened by construct-irrelevant variance occur when the candidates have had to rely on guessing due to problems in understanding either one keyword or the entire meaning of the written stem (question), or (some of) the options. We have a total of 28 such cases. This type of guessing is used by candidates who cannot prove their listening comprehension because of insufficient vocabulary knowledge or deficient reading comprehension. The essential question is whether the targeted listening comprehension ability construct covers this reading ability or written vocabulary knowledge or whether the needed abilities are construct-irrelevant. In a valid test, construct- irrelevant abilities should not affect the test scores. Half of the candidates may have missed the key because of an opaque question or unclear options. There may of course be other factors – like listening comprehension deficiencies – that also affect the process of response selection. In the following cases, the construct-irrelevant variance may be a real threat to the validity of the item:
(30) Jotain kuulin, en ymmärrä vaihtoehtojen sanojaà arvaus ‘I heard something, but I don’t understand the vocabulary of the options’
(Z8:5)
(31) Arvaus (Phénomène?) ‘A guess (Phénomène?)’ (B6:7)
(32) Ymmärsin kyllä mitä nauhalla sanottiin, mutten ymmärrä vastaus vaihtoehdoista B & C kohtia, B vahvempi arvaus ‘I did understand what was said on the tape, but I don’t understand options B and C, B is a stronger guess’ (U7:7)
(33) Aika arvaus ku en täysin ymmärrä ku A-kohdan ja se ei vissiin oo se ’Quite a guess as I only understand option A properly, and I
(34) Arvaus? Pientä tietoa. En ymmärtänyt kunnolla vaihtoehtoja.
’Guess? Little knowledge. I didn’t understand the options properly’
(J3:3)
(35) Arvasin, koska en tiennyt mitä muut vaihtoehdot tarkoittavat ’I made a guess, since I didn’t know what the other options mean’
(J8:6)
(36) Mikä on cambriolages? Täysi arvaus ’What is cambriolages? A complete guess’ (S1:2)
(37) Arvaus. En ymmärrä kysymystä. ‘A guess. I don’t understand the question’ (B7:5)
(38) Arvaus (en tiedä sanaa cambriolages) ‘A guess (I don’t know the word cambriolages)’ (U8:7)
(39) Arvasin…sans frapper…? Ainoa joka tuntu oikeelta ‘I made a guess...sans frapper...? The only option that seemed right’ (N9:8) (40) C:stä en oo ihan varma suomeks, joten arvon vähän ’I’m not sure
about C in Finnish, so I’m guessing a bit’ (U1:3)
3.7 TYPE 7 GUESSING: RULING-OUT OPTIONS
Many candidates have stated that they have been able to eliminate one distractor (or through misunderstanding perhaps the key), but have made a guess between the other two options. Some candidates have explicitly stated that they have used the ruling-out strategy.
This combination of ruling-out and guessing is interesting as this phenomenon is probably only found when the multiple-choice test format is used. Elimination is not a bad strategy as such, and is more often than not based on partial comprehension of the spoken text.
Ebel and Frisbie (1991: 156–157) note that the availability of the process of elimination is sometimes regarded as a weakness of the multiple-choice item format. They conclude, though, that the use of this rather demanding cognitive process is a justified strategy, as the knowledge or ability required to properly eliminate incorrect alternatives can be closely related to the knowledge or ability that would be required to select the correct alternative. They maintain that elimination does not automatically affect the reliability or the validity of the multiple-choice test:
The fact that a student responds in a reflective, problem-solving man- ner, choosing the best answer by rational processes (including the process of elimination) should be applauded rather than deplored. In practice, few multiple choice test items are likely to be answered correctly merely by eliminating incorrect choices. Far more often the process of choice will involve comparative judgements of this alternative against that.
As many as 22 candidates have guessed between two distractors.
In addition, eleven candidates have applied the ruling-out strategy.
Of these 33 candidates one third have been successful. Examples of this strategy emerge from the following:
(41) Ei ainakaan voi olla A joten B tai C. C on arvaus... ’At least it can’t be A so either B or C. C is a guess…’ (B12:8)
(42) Suljin pois vaihtoehtoja ja arvasin kahdesta ’I ruled out options and guessed out of two’ (Å8:7)
(43) Ajattelin, että vastaus ei olekaan C joten veikkasin A:ta ‘I thought the response is not C after all so I made a guess on A’ (Z2:3) (44) Arvaus, ei ainakaan C ‘A guess, at least not C’ (B9:8)
(45) Arvaus, päättelin ettei A koska galleriaa ei ollut ’A guess, I inferred that it was not A since there wasn’t a gallery’ (B9:8)
(46) Pianoa soitettiin illalla à ei B, C arvaus ’The piano was played in the evening à not B, C is a guess’ (J4:3)
(47) En ole varma onko A vai C oikea, siis arvasin ’I’m not sure if A or C is correct, so I guessed’ (T3:4)
3.8 TYPE 8 GUESSING:
INFLUENCE OF THE SECOND LISTENING
The test procedure included listening to the text twice, with a response required after the first listening, but with the possibility to change the response after the second time. As a result, some candidates say that they had to guess at the first listening, but say that the second listening made them more certain as to which opti- on to select. The second listening probably completed the lacking comprehension of the spoken text for these twelve candidates, although they did not necessarily arrive at the correct option. More than a half (seven candidates) arrived at the key. Successful and
(48) Ekalla kerralla arvasin puoliksi, mutta toisella kerralla kuulin vastauksen. En tiedä mitä C tarkoittaa ‘At the first listening I made half a guess, but at the second I heard the answer. I don’t know what C means’ (F5:9)
(49) Ensimmäisellä kerralla en ymmärtänyt mitään, toisella sain kiin- ni ideasta. Vastaukset sen sijaan ovat tod.näk. päin mäntyä.
Arvasin ne kuulemani perusteella. ( P7:4)
(50) Ekaksi arvasin, luulen että puhui kalakaupasta ’At first I made a guess, I think she talked about a fishmonger’s’ (K6:5)
(51) Oli yksinäinen, mutta piti silti lapsuudesta, ensimmäisellä kerral- la ehdin vain arvata ’She was lonely, but liked her childhood, et the first listening I only had the time to guess’ (T5:4)
(52) Eka oli arvaus, luulen että se silti oli A ’The first one was a guess, I think it was A after all’ (J7:5)
3.9 GUESSING OF TYPE 9: UNCLEAR REASONS
There are cases where the candidates are unsure of why they have selected a particular option, and say that they may have guessed. It is understandable that it is not always easy to reflect on one’s own test-taking or comprehension process to know why a particular option is selected. To explain it as guessing feels like an easy way out. These five uncertain candidates probably have some other reason than guessing for selecting an option, a reason that is or is not available for reflection:
(53) Ei nyt ihan arvaus, mutten osaa perustellakaan... ‘Not quite a guess, but I don’t know how to motivate it…’ (B5:6)
(54) Arvasin, tai en muista millä perusteella päädyin vastaukseen ‘I made a guess, or I don’t remember on what basis I arrived at this response’ (P5:3)
(55) En osaa sanoa, arvasin tai päättelin ’I can’t say, I made a guess or I inferred’ (Q2:5)
(56) Puoliksi arvaus, ei kunnon perusteluita ‘Half a guess, no good motivations’ (B11:8)
3.10 GUESSING OF TYPE 10:
NONE OF THE OPTIONS IS GOOD
A few candidates feel that none of the options matches the spoken text and have therefore made guesses. Can we conclude that they have missed something essential in the text contents? Yes, if the options are just and fair, the understanding of the text contents should give away the correct option. Only three candidates have guessed because of a “non-match” situation, two of them have chosen the key option:
(57) Jag tyckte inte att de sa något om någon av dehär, gissning ‘I don’t think they said anything about anyone of these; a guess’
(H7:7)
(58) Mikään ei tuntunut sopivan, arvaus ‘None of these seemed to be good; a guess’ (S9:6)
(59) Arvaus, ei mielestäni sanottu mitään vaihtoehdoista ‘A guess, I don’t think anything was said about the options (Z8:5)
4 CONCLUSIONS ON THE PROCESS OF GUESSING AT A MULTIPLE-CHOICE TEST
What does the continuum from a wild, random guess to a good, informed guess look like then? In Figure 1 all ten types of guessing are included. We can notice that there is a fairly clear correlation between types of guessing strategies and understanding the text.
Random guessing is more likely to occur at the “missing of the input”- end, whereas partial comprehension implies informed guessing. The two other text-related categories (misinterpretation and single word comprehension) are partly overlapping – the responses suggest that the misinterpretations are based on the comprehension of a more covering part of the text than the single word-bound responses. The tendency seems to be that while the single word-bound responses are found among the weakest test- takers, the misinterpretations are made by test-takers on a slightly
guessing related to the ways of applying the guessing strategy (ruling-out and second listening,) or the reasons of applying the strategy (unsure candidates, problems with the options, unclear reasons and no good options) and do not necessarily reveal how much of the text is actually understood.
(4) Partial comprehension
(3) Single word comprehension (2) Mis-
interpretation
(1) Missing the input
(5) Uncertainty
(6) Options not understood (7) Ruling-out
strategy Informed
guessing
Random guessing
(10) No good options (8) Second
listening (9) Unclear
reasons
46 50
18 55
33 80
33
58 50 67
0 10 20 30 40 50 60 70 80 90 100
Partial com prehension
Misinterpretation Sin
gle word grasp ed
Missi ng the
input Unsu
re
Uncl ear reas
ons
Ruling-o ut strategy
Second listening Problem
atic opt ions
No good opt ions
% correct
FIGURE 1. The guessing continuum.
If we also take into consideration the relative success of the candidates having applied these different types of guessing, the continuum is even less linear (see Figure 2).
FIGURE 2. Percentage of correct choices per type of guessing.
As far as the text-based reasons for guessing are concerned, the least successful cases in this sample are those where the choice is based on guessing combined with a reliance on one or two single words: only one fifth of these cases have lead to the key option.
The types representing a misinterpretation or missing of the input as well as partial comprehension have lead to the selection of the key in approximately half of the cases. The uncertain candidates as well as the candidates who have used the ruling-out strategy have been successful in only one third of the cases, which may indicate that these test-takers have not understood a particularly covering part of the text. As opposed to this, the fact that the unclear reasons are combined with the selection of a key in as many as 80 % of the cases may be taken to suggest that if the comprehension and opti- on-selecting task goes smoothly, the test-taker cannot reflect upon the processes very easily. Of course we must keep in mind that the sample in this study is comparatively small, some types being represented only by a handful of cases (see for example types 2, 5, 9 and 10). Still, this result reflects the complicated outcomes of the test-takers’ processes used at a test-taking situation. The reasons for guessing vary from the situation where the spoken text is not understood, through the cases where a test-taker has simply not felt confident enough not to make at least an informed guess, to the situation where the options are unclear. The ways of guessing range from complete random guessing, to cases where an implausible distractor can first be eliminated and to cases where guessing is based on a nearly complete comprehension of the spoken input.
5 CONSEQUENCES FOR TEST CONSTRUCTION
The critique against the multiple-choice format in general, associated with the risk of a lack of reliability and thereby possibly a lack of validity, appears to be legitimate. On the other hand, we
guessing at a multiple-choice test. Guessing is only rarely random guessing, and it is not automatically a bad strategy, always associated with the test format. Rather, it seems that bad test items lead to bad guessing; a good test may lead to the use of the strategies of inferring, reasoning and elimination, ingredients of many communicative language situations. Constructing good multiple- choice items is not an easy task. Piloting items is essential, and the processes activated at particular items and in entire tests need to be studied. Introspection (short written introspection or similar tools) has proved to be useful for this purpose. Bachman and Palmer (1996:
205) recommend that provisions for eliminating or reducing the potential causes of random guessing be included when tests are designed. These provisions are: providing ample time for the majority of the test-takers to complete all the tasks in a test, matching the difficulty of the items with the ability levels of the test takers and encouraging test-takers to make informed guesses on the basis of partial knowledge.
REFERENCES
Alderson, J. C., C. Clapham & D. Wall 1995. Language Test Construction and Evaluation. Cambridge: Cambridge University Press.
Bachman, L. & A. Palmer 1996. Language Testing in Practice. Oxford:
Oxford University Press.
Buck, G. 2001. Assessing Listening. Cambridge: Cambridge University Press.
Ebel, R. L. & D. A. Frisbie 1991. Essentials of Educational Measurement.
New Jersey: Prentice Hall.
Haladyna, T. M. 1994. Developing and Validating Multiple-Choice Test Items. New Jersey: Lawrence Erlbaum Associates.
Linn, R. L. & M. D. Miller 2005. Measurement and Assessment in Teaching. New Jersey: Pearson Education, Inc.
