Assessment Guidelines - Scaling

Contents

Background

Scaling Guidelines

1. What is scaling?
2. When may scaling be used?
3. When should scaling NOT be used?
4. Can marks for assessment items be scaled after they are provided to students?
5. Who can make the decision to scale?
6. How might marks be scaled?
7. Can marks be scaled up or down?
8. Other mark adjustment methods.

Background

In its report to Academic Senate (Nov 2002), the Review of Assessment Practices and Processes (RAPP) Committee recommended that the issue of how and when scaling should be used should be revisited.

As part of its comprehensive revision of UOW assessment policies in light of the RAPP recommendations, the Assessment Working Group (chaired by A/Prof Di Kelly and including EPRS and ESS representatives) recommended that a general policy on scaling would not be practical as faculties applying scaling do so by different methods. Instead, it recommended that it be mandatory for students to be informed in subject outlines if scaling is routinely used in subjects. It was, therefore, recommended that the Code of Practice - Teaching and Assessment be amended to provide, in paragraph 2.3.3(iv), that:

The University reserves the right to scale marks in any subject. If scaling is routinely used in a subject, the method of scaling must be clearly stated in the subject outline.

The Education Policy Review Sub-committee endorsed this amendment (on 23 April 2003) and recommended that Best Practice Guidelines be developed to support faculties in making decisions concerning scaling.

Development of Guidelines

A working group was established by EPRS to develop guidelines for scaling.

Viewed broadly, both the creation and variation of marking schemes involve scaling. However, for the purposes of the guidelines, the working group defined 'scaling' more narrowly to cover 'the adjustment of a group of marks of an entire class or a subset of that class'. The draft guidelines specify when scaling may be used and by whom, and provide examples of how marks might be scaled.

The University affirms the approach of using best practice guidelines to support the implementation of codes of practice, rather than including detailed, prescriptive requirements in the codes. It supports the use of similar guidelines to cover other assessment issues, such as online assessment and in-session testing. An appropriate approval process will need to be agreed upon.

Guiding Assessment Principles

The Guidelines for Scaling specifically state that all decisions regarding scaling must be justifiable in terms of the following core guiding principles identified by the Assessment Working Group:

• collegiality
• transparency
• equity
• consistency

Best Practice - Scaling Guidelines

The following should be used as a guide to academic staff in deciding whether it is appropriate to scale marks and, if so, what method of scaling should be employed. Other mark adjustment processes are also briefly considered.

Any decision to scale or otherwise adjust marks must be justifiable in terms of the core principles or collegiality, transparency, equity and consistency, identified in the Teaching and Assessment Policy.

1. What is scaling?

For the purposes of these guidelines, 'scaling' is defined as the adjustment of a group of marks of an entire class or a subset of that class, for example, a tutorial group. The term is not used to cover the adjustment of marks for individuals or the variation of marking schemes, although these are discussed briefly in section 8 below.

Scaling may involve all of the marks for the subject or just the marks for a particular component of the assessment, for example, the final exam.

2. When may scaling be used?

Scaling may be used when the marks of a group are affected (positively or adversely) by the assessment regime of the subject in an unplanned way.

Some examples of aberrations or unexpected outcomes, which may point to an underlying problem justifying scaling, are:

• the average mark for the cohort is considerably higher or lower than the performance demonstrated by the cohort in other assessment for that subject (or other subjects) or compared to cohorts in previous years;
• external forces unrelated to student performance have caused inappropriate variations between cohorts undertaking the same assessment (e.g. students from different campuses of the University or from different tutorial groups);
• marks are highly concentrated in a narrow band around the median;
• the shape of the distribution of marks is unusual (e.g. highly skewed or bimodal);
• a single assessment task or examination question proves to be problematic.

However, scaling should always be used with caution. Judgement must be exercised about the cause of any aberration or unexpected outcome before adopting scaling to correct a perceived problem. For example:

• Variations between the performance of cohorts may be caused by differences in effectiveness of teaching and/or the quality of students. In such cases non-standard results should be accepted.
• Although many people look for a uni-modal, bell-shaped distribution, there is no universally correct shape. The nature and/or mixture of students doing a subject may validly result in a bi-modal or moderately (or even highly) skewed distribution.

3. When should scaling NOT be used?

Compensating for breakdowns in learning and teaching process

Scaling is not an appropriate method to compensate for serious breakdowns in the learning and teaching process. Other strategies need to be used to address such situations.

Using quotas and normalising

Some Universities require the proportion of specific grades to lie within defined bands (e.g. High Distinctions to be within a band of 5-12% of the cohort), except for small enrolment groups. Such quotas are not part of preferred UOW practice, although academics marking assessment items may use such reference points to inform their decision-making.

Scaling to a normal distribution (or other preconceived model) is also not a preferred UOW practice.

However, as part of their quality audit role, Faculty Assessment Committees (FACs) are expected to compare distributions of grades and investigate any apparent problem areas.

4. Can marks for assessment items be scaled after they are provided to students?

Once marks for individual components of assessment have been approved by the subject coordinator and released to students they CANNOT be scaled down, unless the subject outline explicitly states that this may occur: see Code of Practice - Teaching and Assessment, section 7.3.

5. Who can make the decision to scale?

The Unit Assessment Committee and/or the FAC can review the distribution of grades for any given subject and make a decision to scale final marks. The FAC must be informed where results are scaled by the Unit Assessment Committee.

For assessment occurring throughout the session, routine scaling can be undertaken by the subject coordinator, provided this is done as stated in the subject outline, as required by section 7.1 of the Code of Practice - Teaching and Assessment.

6. How might marks be scaled?

Scaling might involve:

1. widening the range of marks about the mean, but not shifting the mean (used to correct for concentration in a narrow band);
2. shifting the mean (used to adjust for an unreasonably low or high average);
3. adjusting the shape of the distribution (for a clearly defined and valid reason, if one exists); or
4. a combination of any of the above

The algorithm for doing this may be based on linear or non-linear transformations. Other methods, such as scaling to normality, or scaling to achieve specified percentages within bands are not recommended practice at the University of Wollongong. Unless there are clearly identified goals, the simpler and more transparent the scaling method the better. Advice on how to achieve particular scaling goals is available from the Statistical Consulting Service at the University.

Any method of scaling of an individual assessment component or a final mark must preserve rank order within the relevant cohort: see Code of Practice - Teaching and Assessment, section 7.2. However, the scaling of an individual component of assessment for a sub-cohort (such as a tutorial group) may result in changes to the rank order in the larger cohort of students studying a particular subject.

7. Can marks be scaled up or down?

It may be appropriate to scale marks either up or down, although particular caution is advised when scaling down.

8. Other mark adjustment methods

There are other methods of adjusting marks which are not strictly scaling, and which also need to be approached with caution or avoided altogether.

Modifying marking schemes

The subject coordinator may need to modify assessment methods or weightings in unexpected circumstances. In such cases students must be informed in writing.

If changes are made after the second week of session, the consent of every student enrolled in the subject must be obtained and the subject coordinator must seek approval from the head of the academic unit (as required by section 4.2.5 of the Code of Practice - Teaching and Assessment).

Adjusting marks or weightings in individual cases

Where a student has demonstrated uneven performance over the session (e.g. strong performance in assignments and poor performance in exams), it is NOT APPROPRIATE to:

• adjust the marks of the student, or
• give added weight to a particular type or item of assessment completed by the student.

If the final exam is considered paramount, it should be awarded a greater percentage of the overall assessment. Students may also be required to perform to a satisfactory standard in the final exam or other significant assessment task (e.g. to achieve 40% in the final exam or to satisfactorily complete a laboratory component) in order to achieve a pass in a subject overall. Such a policy must be clearly communicated to students in the subject outline.

The consideration of students' composite results in special consideration cases is covered by the Special Consideration policy. It is inappropriate to regard such circumstances as a basis for scaling, but it may be appropriate to modify the weightings for the various assessment components to reflect the circumstances of the special consideration granted to the student.

Where there has been an error in a mark awarded to a student, the matter should be referred to the Chair of the Faculty Assessment Committee and the Academic Registrar or their nominees.

Last Updated: March 16, 2004