Lecture 3 Test usefulness

Lecture three Test usefulness: Qualities of language testsI test usefulnessn Our notion of usefulness can be expressed as in the following Figure Usefulness = Reliability + Validity + Authenticity + Interactiveness + Washback + Practicalityn Three principles for usefulnessl Principle 1 It is the overall usefulness of the test that is to be maximized, rather than the individual qualities that affect usefulness.l Principle 2 The individual test qualities cannot be evaluated independently, but must be evaluated in terms of their combined effect on the overall usefulness of the test.l Principle 3 Test usefulness and the appropriate balance among the different qualities cannot be prescribed in general, but must be determined for each specific testing situation.II Qualities of language testsn Reliabilityl Definition of reliabilityReliability is often defined as consistency (accuracy) of measurement (test scores).l Methods of reliability computationTest-retest method1. The simplest computational method2. Calculated by means of product-moment correlation of two sets of scores for the same persons3. The same test is readministered to the same people.4. Following an interval of no more than two weeks5. A formula might be expressed for this method as fellow: rtt = r1,2rtt = the reliability coefficient using this methodr1,2 = the correlation of the scores at time one with those at time two for the same test used with the same personsProduct-moment correlation: also called Pearson product-moment correlation coefficient（皮尔逊积矩相关系数）which is symbolized by r. It‘s value varies fro m – 1 . 00 to + 1 . 00, with the value of zero indicating the absence of any correlation and either a minus or plus one indicating perfect correspondence of scores.For example: if students received quite similar scores on two tests their scores would have a high positive correlation. If their scores on one test were the reverse of their scores on the other, their scores would have a high negative correlation. If their scores on the two tests were not related in any predicable way their scores would have a zero correlation.Parallel forms method1. Two tests are administered to the same people without considering an intervalreliability2. One benefit of this method in comparison with test-retest method is that practice effect is controlled, since the examinees do not encounter the same items twice.3. The tests employed in this method must satisfy rigid requirements of equivalence. These tests must show equivalent difficulty, equivalent language knowledge, equivalent numbers of items,and equivalent time.4. calculated by means of Product-moment correlation5. A formula might be expressed for this method as fellow: rtt = rA,Brtt = the reliability coefficient using this methodrA,B = the correlation of form A with form B of the test when administered to the same people at the same timeSplit half method1. under the condition that it is not possible to administer the test twice and only one rater is involved in the scoring.2. internal consistency reliability – within one administration of the exam itself3. a test is administered once to a group of examinees, it is divided into halves, the scores of each half are correlated with the other half.4. calculated by means of Spearman-Brown Formula5. the test may be split in a variety of ways.If the test is comprised of items, a common technique is to place all of the odd-numbered items in one half and all of the even-numbered items in the other half. These two halves are scored separately and are correlated with each other.For tests without item, some other method of dividing the test may be used. If, for example, the test consisted of a series of compositions, each group of compositions may be scored separately and correlated with the other.6. A formula might be expressed for this method as fellow: rtt = 2rhh /1 + rhh rtt = the reliability coefficient using the split half methodrhh = the correlation of the scores from one half of the test with those from the other halfn ValidityHenning 1987 defines validity as follows:Validity in general refers to the appropriateness of a given test or any of its component parts as a measure of what it is purported to measure. A test is said to be valid to the extent that it measures what it is supposed to measure. It follows that the term valid when used describe a test should usually be accompanied by the preposition for. Note that Henning‘s definition allows for degrees of validity: tests are more or less valid for their purposes: validity is not an all-or-nothing matter. This important point m eans that users will have to use their own, or somebody else‘s, judgement when deciding, on the basis of evidence, on the relative validity of a test.However, if a test is to be used for any purpose, the validity of use for that purpose needs to be established and demonstrated. The validity of a test is the extent to which it measures what it is supposed to measure. Every test, whether it be a short,informal/classroom test or a public examination, should be as valid as the constructor can make it. The test must aim to provide a true measure of the particular skill which it intends to measure. If it measures external knowledge and other skills at the same time, it will not be a valid test.Invalid tests are those that have undesirable contents mixed in the desirable content. Since validity has been defined as the capacity of a test to measure what it is purported to measure, a variety of considerations may bring about a reduction of this validity.A. Invalid application of testsB. Inappropriate selection of contentC. Imperfect cooperation of the examineeD. Inappropriate referentE. Use of invalid constructsTypes of ValidityValidity can be established in a number of different ways, which leads most writers on the topic to talk of different types of validity. This lecture uses the terms internal validity and external validity: the internal validity relates to the studies of the perceived content of the test and its perceived effect, while the external validity relate s to the studies of comparing students‘ scores with their ability gleaned from outside the test.Internal validityl Face validityIngram (1977: 18) defines face validity as ― the test‘s surface credibility or public acceptability‖. A test is s aid to have face validity if it looks as if it measures what it is supposed to measure. Language tests which have been designed primarily for one country and adopted by another country may lack face validity. A test which does not have face validity may not be accepted by candidates, teachers, education authorities or employees. It may simply not be used; and if it is used, the candidates‘ reaction to it may mean that they do not perform on it in a way that truly reflects their ability. Face validity is important in testing. For one thing, tests that do not appear to be valid to users may not be taken seriously for their given purpose; For another, if test takers consider a test to be face valid, they are likely to perform to the best of their ability on the test and respond appropriately to items.Face validity is a kind of impressionistic reaction to the examination on the part of examinees. If a test has good face validity, the students‘ motivation is maintained, for most students will try harder if the test looks sound. If, on the other hand, the test appears to have little relevance in the eyes of students, it will clearly lack face validity. Data on face validity may be gathered by the teachers interviewing students or asking them to complete a questionnaire about their attitudes and reactions to a test they have just taken.In order to know how subjects respond to a test, the following questionnaire is to survey language test anxiety when subjects are engaged in an English test. Designed on the basis of a test-anxiety study conducted by Jones-Madsen, the questionnaire consists of 3 questions.Questionnaire paper1). 我喜欢做这一题型。