How to Create Fair, Valid, and Reliable Exams
Review validity, reliability and fairness with practical checks for coverage, scoring, access and alternate papers. Understand the evidence and limits.
Creating valid and reliable exams means gathering evidence that the scores support the decisions you intend to make. Fairness also requires attention to access, task design, administration and scoring. A blueprint, a rubric or a question bank can support that work, but none automatically makes an exam trustworthy.
This guide gives a classroom review framework, not a certification method. Consequential uses such as placement, selection or qualification decisions may need formal measurement evidence and specialist advice beyond the checks described here.
Validity, reliability and fairness are different questions
| Concept | Main question | Important limit |
|---|---|---|
| Validity | Does the evidence support this interpretation and use of the scores? | A paper is not valid for every purpose simply because it covers taught topics. |
| Reliability / precision | How consistent or precise are the scores under the relevant conditions? | Agreement between markers does not establish consistency across time or alternate forms. |
| Fairness | Can students demonstrate the intended learning without irrelevant barriers or inappropriate treatment? | Giving everyone identical conditions is not always equitable or accessible. |
Good consistency does not prove that the intended skill is being assessed. Equally, strong coverage does not remove uncertainty from a short or inconsistently scored assessment. Review these questions together without treating one checklist as proof of all three.
Start with the intended decision and learning outcomes
Specify what a score will be used for and what learning the assessment should reveal. A quick feedback quiz and a selection exam do not require the same depth of evidence or support the same conclusions.
Use a table of specifications to plan content and skill coverage. Consider the applicable specification, outcome importance and opportunities to learn; teaching time is one input, not a universal weighting formula.
The University of Michigan’s exam-design guidance links tasks to intended learning and realistic conditions. A grid helps make the plan explicit, but its labels do not prove what the actual questions measure.
Check what the task really asks students to do
- Content: does the paper sample the intended domain adequately for this decision?
- Demand: does each task require the knowledge or thinking you intended, rather than merely using a suitable-looking verb?
- Language and context: are reading demands or assumed background knowledge relevant to the intended skill?
- Format: does the response show the evidence you need, including reasoning or practical performance where required?
- Conditions: are time, materials, instructions and required adjustments appropriate?
Removing irrelevant wording can improve a question, but simplification should not remove the skill itself. If interpreting technical language is part of the outcome, that demand may be relevant. Make the distinction deliberately and test the final version.
Use the paper accessibility checklist to review layout, diagrams, languages and access requirements. Check the actual output, including any accessible format, rather than assuming a template works for everyone.
Make scoring decisions explicit and review their application
Prepare the marking guidance with the paper. Include acceptable alternatives and partial-credit decisions where appropriate. Choose an appropriate rubric for extended work and discuss a suitable common sample before wider marking.
A rubric does not guarantee agreement within half a mark or eliminate judgement. Check how markers apply the guidance, document differences and follow the required moderation process. Do not treat a new scoring rule as a private exception for one script.
Cambridge International’s assessment-quality standards describe repeated question/key review, examiner standardisation and monitoring. They illustrate several layers of quality assurance, not a claim that using this guide reproduces an awarding body’s process.
Use score evidence carefully
Reliability evidence depends on the question being asked. Marker agreement, internal consistency, stability over time and alternate-form evidence concern different sources of variation. One is not a substitute for all the others.
There is no universal number of questions that makes a test reliable. Adding suitable items may improve sampling, but adding ambiguous, redundant or irrelevant items can introduce new problems and increase fatigue. Review the purpose, coverage, time and scoring instead of simply making every paper longer.
The item analysis calculator can help examine selected response statistics when you have suitable data. A difficult item or an unusual discrimination value is a prompt to investigate, not an automatic reason to change marks or delete the item. Small samples, multiple skills and the cohort affect interpretation.
A visual scan of results does not establish a reliability coefficient, and coefficient alpha alone would not prove validity or fairness. Use evidence appropriate to the intended decision and seek specialist review where the consequences require it.
Review alternate papers before comparing scores
Matching a blueprint, marks and difficulty tags does not establish equal difficulty or score equivalence. Different tasks can vary in reading load, response time, prior exposure and the reasoning required.
Check each form’s questions, context, instructions and key. For consequential comparisons, a common specification may need to be supplemented by empirical evidence and a suitable comparability process. See the alternate-forms guide; do not assume software selection makes the review unnecessary.
A practical review record
| Stage | Evidence to retain |
|---|---|
| Before assembly | Purpose, outcomes, applicable specification and the coverage plan |
| Before release | Reviewed paper/key versions, access checks, unresolved actions and required approvals |
| During marking | Calibration decisions, scoring clarifications and the required moderation record |
| After use | Queries, appropriate response evidence, investigated errors and corrections before reuse |
The moderation checklist can organise this record. Keep confidential scripts and student data in approved institutional storage, not a public web worksheet.
What Examiar supports
Examiar organises questions and creates papers and answer keys. Those functions support preparation; they do not automatically analyse student responses, enforce academic approvals or certify validity, reliability or fairness.
Follow the documentation, review a small first output and retain the academic decision with the responsible people. Try Examiar free with a non-confidential sample if you want to evaluate the preparation workflow.
Frequently asked questions
Can a consistent exam still be unsuitable?
Yes. Consistent scoring can still reflect an unintended skill or an inaccessible task. Reliability evidence does not establish validity or fairness on its own.
Can I check reliability without statistics?
You can review design and scoring practices without calculating a statistic. That is useful, but it does not establish formal score precision or replace evidence needed for a consequential decision.
Does a shared pattern make two papers equivalent?
No. A common pattern specifies some requirements. The actual forms still need review and, where appropriate, additional evidence before their scores are treated as comparable.