Examination System
Setting and Pretesting
1.1 Selection of materials
Supported by the test specifications, item writers select materials
for draft tasks from a variety of sources including magazines,
newspapers, brochures, advertisements and radio programmes. Guidelines
for the selection of materials are related to the following:
• skills to be assessed;
• linguistic complexity;
• subject matter;
• cultural considerations;
• item types.
The materials are submitted for consideration together with draft
items. More source texts are selected than will be eventually
needed for the final forms of tests so that the materials that
are deemed unsuitable by the vetting team can be rejected.
1.2 Test item writing
Item writers are practising language teachers or applied linguists.
They are commissioned to select materials and draft sets of tasks.
The Centre for Applied Language Studies (CALS) is responsible
for training the item writers and for producing and updating test
specifications which guide both item writing and materials selection.
Item writing is followed by an item review process where the members
of the test writing team take the tasks, comment on them, and
propose revisions, after which the original item writer completes
the revisions. Typically, the item review process takes two or
three rounds of reviews and revisions. Final layout for pretesting
and official test administrations is done at CALS. Official test
forms always contain a mix of previously used and analysed tasks
and fresh test items.
1.3 Pretesting
Each operational test form contains previously used and analysed
items together with a set of new items. Subsequent item analysis
indicates which of the items were compatible with the existing
item pool, and these will be used in scoring and in subsequent
test rounds. Regular feedback on the appropriacy of new items
is also gathered from the markers. Item banks based on IRT analyses
are being created, beginning with tests in Finnish, English and
Swedish.
Marking
2.1 Process
Scripts and recordings produced by candidates are sent to CALS
for central marking. The marking process begins with a standardisation
meeting where all the raters participate. This is followed by
a number of days of marking work, which includes both first and
second marking. If necessary, third marking is done at CALS immediately
after the analysis of the centralised marking process is complete.
The move to centralised marking was made after two years of trials
because the results showed that there were significant gains in
quality control and in speed of marking.
The quality of marking is supported by multiple strategies. The
markers must have a university degree in the language they mark,
and most of them are language teachers. Marking begins with a
standardization session, in addition to which, the marking work
is guided by a fairly detailed marking scheme and by sets of benchmark
examples in the productive skills. For double marking, see below.
2.2 Training
Before becoming accredited markers, the applicants are trained
in seminars which are led by staff members of CALS and experienced
item writers and markers. During these seminars the marking schemes
and criteria are discussed, followed by marking and discussion
of sample scripts. Training sessions on marking speaking and writing
start with viewing and discussion of a set of benchmark samples
at different levels of proficiency. This is followed by assessment
and discussion of new scripts and taped performances. A database
of accredited markers is kept centrally at the National Board
of Education.
2.3 Checking
With the basic and intermediate level tests, about 30% of the
subjectively marked components are double marked. First raters
can request certain scripts to be double marked, but this selection
is complemented by random sampling of further scripts from each
rater. If a marker is found to deviate too much from the standard,
all his or her scripts are double marked. In the advanced level
tests, all performances (excluding the objectively marked tasks)
are double marked.
Results
The participants get both an overall grade and a profile of grades
in speaking, listening, reading, writing and vocabulary and grammar.
The grades are given on a six-point scale. The basic level tests
yields grades 1 and 2, the intermediate level tests grades 3 and
4, and the advanced level test grades 5 and 6. Candidates who
fail to reach the lowest grade available from a test section get
a section grade of "below 1" on the basic level test,
"below 3" on the intermediate level test, and "below
5" on the advanced level test. The results and certificates
are issued within two months of the examination date.
Security
Examination sessions are carefully controlled and each examination
centre has a person responsible for the security of the test papers.
Printing and storage of the examination materials takes place
centrally at the University of Jyväskylä.
|
