Bias and Fairness Review
Pubrica provides bias and fairness review of assessment items and learning content for awarding bodies, examination boards, publishers and education providers. Our reviewers identify content that disadvantages a group of learners for reasons unrelated to the construct being measured, such as a question about a regional sport, a scenario that assumes a particular household or a reading passage with regional vocabulary. We carry out sensitivity review by a panel representing your candidates, run differential item functioning (DIF) analysis where response data exist, and deliver evidenced findings with proposed revisions.
- Focus on construct-irrelevant difficulty: we check whether content makes an item harder for reasons unrelated to what it measures, not only whether it is offensive
- Sensitivity review before the test runs, by a panel representing the groups who take the assessment
- Differential item functioning analysis where response data exist, because panel review and statistical analysis identify different problems
- Evidenced, specific findings, so each item is revised for a stated reason
- Learning content as well as assessment items, where the same assumptions can affect learners across a whole course
Sensitivity review of items and contentIncluded
Construct-irrelevance assessmentIncluded
Differential item functioning analysisWhere data are available
Evidenced findings with revisionsIncluded
Setting your content policyYours
Items → reviewed → tested → evidenced → revised
- Construct-irrelevance as the review criterion
- Panel representing the candidate population
- DIF analysis where data allows
- Findings supported by evidence
- Revisions proposed for each finding
Service Scope
Four Components of Every Bias and Fairness Review
Each review covers one assessment and its candidate population, which defines what counts as construct-irrelevant.
From the reviewA mathematics word problem set at a baseball game.The mathematics is correct, but the context assumes familiarity with innings and scoring. Candidates unfamiliar with the sport must interpret the context before reaching the mathematics. We revised it to a neutral context with the same mathematics.
From the panelA panel drawn from one background is likely to miss the assumptions of that background. This reflects how assumptions work, not a shortcoming of individual reviewers.
From the DIF analysisPanels and data identify different issues. DIF analysis flags items that reviewers did not suspect, while panels identify problems in items that have not yet been administered.
From the reportRemoving items without revision reduces the item bank. Most flagged items can be fixed by changing the context.
The samples shown are illustrative and anonymised. Hover over a sample to replay it.
Compliance and Standards
Four Fairness and Testing Standards
The testing standards treat fairness as a validity requirement, and our review follows the same approach.
Standards for Educational and Psychological Testing (AERA, APA, NCME — fairness chapter)
Treats fairness as a validity issue: whether score differences reflect the construct or an irrelevant factor.
Differential item functioning (Mantel-Haenszel, IRT-based)
Statistical methods that detect items functioning differently for matched groups of equal ability, which distinguishes bias from genuine differences in performance.
Web Content Accessibility Guidelines (level AA)
Covers captions, contrast, keyboard operation and reading order, and is a procurement requirement for most institutional and public-sector buyers.
Quality management (cert. H3CR9TCU20251212INDQM10N)
This certification covers how the work is briefed, performed, reviewed and released.
Responsible Use of AI
Our Approach to AI Use and Disclosure
AI tools support specific, checkable tasks. Editorial judgement, final wording and sign-off remain with Pubrica subject specialists.
Why We Do Not Publish AI Accuracy Figures
Accuracy figures for AI tools are usually reported by the companies that sell them.
Pubrica therefore does not quote its own accuracy figures, as they would carry the same conflict of interest. Instead, we tell you which tool was used on your project, at which stage, and what a specialist changed afterwards.
Methodology
How Our Bias and Fairness Review Process Works
Panel review takes place before the test runs and data analysis afterwards. Each method complements the other.
Define the population
We establish who takes the assessment, which determines which assumptions count as construct-irrelevant for this test.
Sensitivity review
A panel representing that population reviews items and content for construct-irrelevant difficulty as well as for offensive material.
DIF analysis where data exists
We test items for differential functioning between matched groups. Flagged items are reviewed by a specialist and are not removed automatically.
Revisions and report
Each finding is evidenced and paired with a proposed revision, and the review is documented for your technical file.
Our Experts
Your Bias and Fairness Review Team
A named team of subject specialists stays with your project from scoping to delivery. Your project coordinator introduces each team member by name before work begins.
Reviews the scientific content for accuracy and confirms that each conclusion is supported by the evidence. Each expert is matched to your field.
Reviews content for accuracy, pedagogy and conformance with the applicable curriculum, assessment or accessibility standard.
Checks the finished work independently against the agreed scope and the applicable checklist. Findings are logged and resolved before delivery.
Manages the timeline, keeps you informed at each stage and coordinates the team so that questions are answered and deadlines are met.
Packages
Bias and Fairness Review Packages
Pricing depends on item volume and on whether response data are available for analysis.
Sensitivity review
Items and content reviewed before use
Quoted per projectBased on item or content volume
Delivery By volume
- ✓Population and construct defined
- ✓Panel review by reviewers reflecting the population
- ✓Construct-irrelevant difficulty identified
- ✓Proposed revisions for each finding
- ✓Documented for your technical file
Review before the test runs allows items to be revised rather than retired.
Service Scope Request a quoteReview with DIF analysis
Live assessments with response data
Quoted per projectBased on item volume and sample size
Delivery By scope
- ✓Everything in sensitivity review
- ✓Differential item functioning analysis
- ✓Flagged items reviewed rather than auto-removed
- ✓Bias distinguished from genuine differences
- ✓Bank-level patterns identified
Patterns across the item bank often point to an item-writing convention, so we report them alongside individual findings.
Service Scope Request a quoteSend fifty items and we will review ten free of charge. Most item sets include two or three items where the context, rather than the construct, affects difficulty.
FAQs
Bias and Fairness Review FAQs
If your question is not answered here, ask an expert — no quote is required.
No, although any offensive content is removed. The review focuses on construct-irrelevant difficulty: content that makes an item harder for a group for reasons unrelated to what is being measured.
For example, a mathematics item set in an unfamiliar sporting context is not offensive, but it still measures the wrong thing for some candidates.
Not necessarily. Differential functioning means the item behaves differently for matched groups of equal ability, which is an indicator to investigate rather than a conclusion.
It sometimes reflects a genuine difference in what was taught. Flagged items are therefore reviewed by a specialist and are not removed automatically.
The panel should include reviewers from the groups who take the assessment, because reviewers are less likely to notice assumptions they share.
For this reason, panel composition is part of our review method.
Yes, although it is less often carried out. An assessment item affects a candidate once, while a course that assumes a particular background affects learners throughout the course.
We apply the same review method to the scenarios, examples, names and contexts used in learning content.
Related Services
Related Assessment Services
Fairness is part of validity, so this review is often combined with other assessment services.
1ASSESSMENT EVIDENCEsame technical file
Psychometric servicesItem analysis and standard setting. Assessment and exam reviewItem quality and outcome alignment.2LEARNING CONTENTscenarios and examples
Educational content developmentContent developed to learning objectives. Revisioning and localisationContent adapted for new audiences.3ACCESSIBILITYaccess for all candidates
Accessibility complianceWCAG 2.2 across content. Standards alignmentDefining what the test should measure.Request a Free Bias and Fairness Review of Ten Items
The sample review identifies items where the context, rather than the construct, may be affecting responses, and states whether each item can be revised.