Examination System

Setting and Pretesting

1.1 Selection of materials

Authentic materials are selected for the four profiles from a wide range of sources including newspapers, magazines, brochures, advertisements, publicity material, collections of short stories, operating instructions and radio items such as interviews, news broadcasts, debates, announcements, informative programmes and speeches. The selection is to be based on an inventory of situations (which is partially based on the needs analysis) that are relevant to the particular profile. We will strive to have about half of the materials of northern origin (the Netherlands) while the other half comes from the Dutch-speaking part of Belgium. The selection is to be carried out by the scientific unit of the CNaVT, which also constitutes the Examination Board. In their choice of materials, they will be guided by the following criteria:

• skills to be assessed
• linguistic complexity
• subject matter (situations relevant to a profile)
• kinds of tasks
• cultural considerations
• the profile description.

1.2 The writing of test tasks

The members of the CNaVT scientific team will write the test tasks. They are highly experienced in this field. New member will be trained and closely monitored. The whole team will keep up to date on developments in testing theory and practice. For each of the profiles, the profile descriptions function as a point of reference for the writing of the test tasks. They are intended to determine the contextual and linguistic complexity and the level of difficulty of the tasks. All tasks will be reviewed by every team member, and several editing rounds will take place. An advisory board for the CNaVT composed of an external group of experts in relevant fields will critique the first version of the exams. On the basis of this advice, the teams in Leuven en Nijmegen will prepare the final version.

1.3 Preliminary testing

Since the profile-related exams are completely new, they will need to be pre-tested at a preliminary stage, but an annual pre-testing is not expected to be necessary later. As an alternative, link items will be included in the actual examinations. This will allow for calibration and, after marking, for obtaining statistics in regard to the discrimination and reliability for each task. In this way, pass standards can be established as related to known levels of ability. For each session, a new set of linking items will be selected as based on the statistical findings and then included in the new examination. This procedure, when combined with the profile descriptions as points of reference, will ensure that the same norm will be maintained for examinations in successive sessions. Examination candidates, teachers and examiners, and raters will be invited to provide feedback so as to get as much information as possible on the quality of the test tasks.

Marking

2.1 Process

Objective components will be marked mechanically (manually and by computer) at the CNaVT. The responses in the oral part will be recorded on audiotape by the local examiner (usually a teacher of Dutch) and will be marked by one rater at Leuven and at Nijmegen. The scripts in the written component will also be marked by one rater at the CNaVT. Raters will be qualified teachers of Dutch, usually with some teaching experience in teaching Dutch as a foreign language. Detailed marking scales pertaining to a number of writing and speaking tasks will be used.

The marks for all examinations will be scaled using IRT-based procedures. Cut-off marks will be determined by reference to fixed points on the ability scales.

2.2 Training

Raters will undergo an extensive six-day training supervised by the CNaVT staff. This will familiarise the raters with the examinations and their underlying principles and will give the raters extensive practice in using the marking scales on sample examinations from previous years. Those marking the same subtest will be encouraged to confer with each other on difficult cases, the aim being to reduce differences in scoring behaviour. Raters will mark all the oral and written exams in Leuven and Nijmegen and will be continuously advised and monitored by the staff.

2.3 Monitoring the raters

The CNaVT research staff will regularly check random samples of marked examinations and provide the raters with feedback. From time to time, raters will be asked to re-mark each other's work, and any significant differences will be discussed with marks being adjusted if necessary. One staff member will be accountable for the quality of the marking done for writing tasks, while another staff member will be accountable for the quality of the marking done for the oral component. Borderline cases will be compared to similar cases from previous years to see how they had been marked.

Examination results

Results will be in the form of a pass or no-pass mark. Cut-off scores will be determined by using IRT-based procedures. Centres and candidates will receive the results within eight to ten weeks of taking an examination. Certificates are to be issued two weeks after that.

Security

During each stage of the examination, precautions are to be taken to ensure maximum security. The examinations are to be dispatched in sealed envelopes that can be opened only in the presence of the examinees at the time the examinations are to be administered. Local examiners are required to follow strict security procedures. All examination papers, even if unused, must be returned to the central agency at Leuven or Nijmegen.