Skip to main content
Tokenline

Line 3 · Station 2 of 6

Bias & Fairness: Whose Patterns Does AI Learn?

How unfair patterns get into AI systems, why they matter when AI makes decisions about people, and what fairness checks look like.

  • 8 min
  • +50 XP
  • 2 knowledge checks

By the end of this station, you can:

  • Identify four places bias can enter an AI system
  • Explain why biased data can lead to unfair outcomes even when no one intends harm
  • Describe practical ways to test an AI system for fairness

AI learns the world it's shown

Machine learning systems find patterns in examples. If those examples reflect unfair patterns from the past — or leave some groups of people out — the system can learn and repeat those patterns, often at large scale.

That's why an AI tool can be unfair even when nobody who built it meant it to be. Bias in AI is usually not a single villain. It's a chain of choices about data, labels and design that adds up.

Key terms

Algorithmic bias
Systematic errors in an AI or computer system that create unfair outcomes, such as consistently favoring or disadvantaging certain groups of people.
Representative data
Data that includes the full range of people and situations the system will be used on, in reasonable proportions.
Proxy variable
A piece of data that isn't about a sensitive trait directly but closely tracks it — for example, a zip code that strongly relates to income or race.

Four places bias can sneak in

  • The data: some groups are missing, under-represented, or shown in stereotyped ways.
  • The labels: the "right answers" in training data came from past human decisions, which may themselves have been unfair.
  • The design: what the system is told to optimize, and which inputs it uses (including proxies), shape who it serves well.
  • The use: a tool built for one setting can perform badly — or be used unfairly — in another.

Knowledge check 1

+10 XP on first try
A company trains a résumé-screening tool on ten years of its own hiring decisions. In that decade, managers rarely hired people from certain schools. What is the most likely result?

Interactive lab

Biased Screener

A fictional summer-internship screener was trained on 10 years of past picks — most came from Northside High. Each pair of applicants below has identical qualifications. Try the fixes and watch the scores.

Try a fix
Screener scores for six fictional applicants
ApplicantSchool · ZipQualificationsModel scoreDecision
Ava R.Northside · 401107085Interview
Jordan M.Southside · 402207060Rejected
Priya K.Northside · 401107893Interview
Marcus T.Southside · 402207868Rejected
Lena O.Northside · 401106479Interview
Diego S.Southside · 402206454Rejected

Interview cutoff: score of 75 or higher.

Interviews: Northside 3/3 · Southside 0/3

The model copies the pattern in its training data: past picks were mostly from Northside, so Northside applicants get a boost even with identical qualifications.

What to notice: Models copy patterns in their training data, including unfair ones. Deleting a column isn't enough when proxies like zip code carry the same information — better data and testing outcomes by group are what fix it.

A real example: testing face analysis

Researchers Joy Buolamwini and Timnit Gebru tested commercial facial analysis systems that classified gender from photos. Their Gender Shades study found the systems were much less accurate for darker-skinned women than for lighter-skinned men.

The study made an important point: a system's overall accuracy can look strong while hiding big differences between groups. You only see the problem when you break the results down.

What fairness checks look like

  1. Break results down by group

    Measure accuracy and error rates separately for different groups instead of trusting one overall score.

  2. Audit the training data

    Ask who is included, who is missing, and where the labels came from.

  3. Remove or question proxies

    Check whether inputs like location or school quietly stand in for protected traits.

  4. Keep humans accountable

    For decisions about people, give them a way to question the result and have a person review it.

Knowledge check 2

+10 XP on first try
A team reports that their new AI tool is "95% accurate overall." What's the most important follow-up question from a fairness point of view?
Learn moreIs "removing race and gender from the data" enough?

Not usually. Other inputs — like zip code, school name or even word choice — can act as proxies that carry the same information. A system can still produce unequal outcomes without ever seeing a sensitive trait directly.

That's why fairness work focuses on testing outcomes, not just deleting columns. Frameworks such as the NIST AI Risk Management Framework treat bias as a risk to be measured and managed throughout a system's life.

Sources for this lesson

  • Joy Buolamwini and Timnit Gebru. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, 2018. Full citation
  • National Institute of Standards and Technology (NIST). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023. Full citation

Finish this station

Answer every knowledge check (right or wrong) to unlock completion.

  • Knowledge check 1 — not answered yet
  • Knowledge check 2 — not answered yet

0/2 checks answered