Are Algorithms the New Prejudice? Unveiling Hidden Bias in Rating Systems

Soraya Malik in Tech & Innovation July 2025 • 5 min read.

"Explore how seemingly fair algorithms in online marketplaces can perpetuate discrimination, and what it means for the future of fairness in the digital age."

In today's digital age, discrimination remains a persistent challenge. Despite advancements in technology, inequalities based on race, gender, ethnicity, and other social identities continue to surface in online marketplaces and social media platforms. You might think that algorithms and rating systems would eliminate bias, but studies show that discrimination persists on platforms like Airbnb, freelancing websites, and even online communities. This raises an important question: How can discrimination occur even when algorithms are designed to be fair?

At first glance, online marketplaces seem like an ideal setting for fairness. User-generated rating systems are designed to provide accurate information about individuals, which should reduce biased inferences based on group identities. In a perfect world, more information and social learning should lead to less discrimination. However, real-world marketplaces are far from perfect, and it's not always clear whether these mechanisms actually reduce discrimination.

The key lies in understanding how social learning works in these environments. Social learning involves a feedback loop between two processes: data sampling (or experience gathering) and informing (or recommending user decisions. While the latter can be designed to be unbiased and fair, the former is inherently non-random and potentially biased. Data sampling occurs when transactions take place, driven by the economic interests of the parties involved. Users naturally seek high-value partners with positive ratings, not random or representative ones. This selective sampling can lead to unexpected and unfair outcomes.

How Can Fair Rating Systems Still Lead to Discrimination?

To understand this paradox, let's delve into a recent study that examines how statistical discrimination can arise in ratings-guided markets, even when the algorithms themselves are unbiased. The researchers developed a model that incorporates the feedback process between data sampling and user decisions to examine the implications for statistical discrimination. The model features directed search and matching between buyers and sellers, guided by user-contributed ratings.

Imagine a marketplace where sellers are indexed by their social group identity (e.g., Group 1 or Group 2) and their productivity type (high or low). While a seller's group identity remains constant, their productivity can change over time. Buyers aim to match with high-productivity sellers, but they rely on imperfect information in the form of binary ratings: "good" or "bad."

The Model Setup: Sellers have a group identity (Group 1 or Group 2) and a productivity type (High or Low), ratings are binary ("good" or "bad").
The Twist: Ratings can be updated after each transaction, reflecting a seller’s actual type.
The Key Parameter: The effectiveness of social learning (how well ratings reflect actual types) is captured by a parameter α.
Strategic Buyers: Buyers direct their search based on ratings and group identities.

One critical assumption is that group identity is independent of a seller's productivity. This means that a seller's group affiliation has no direct impact on their ability to perform well. However, buyers may still use group identity as a signal when making decisions. The researchers found that this can lead to a "non-discriminatory" equilibrium, where sellers from both groups are treated identically. In this scenario, all sellers receive the same level of attention from buyers, regardless of their group identity. G-rated sellers enjoy a higher match rate than B-rated sellers, thanks to the positive signal associated with a good rating. This equal treatment ensures unbiased sampling across groups, leading to identical belief updates, and maintaining non-discrimination in the steady state.

The Path Forward: Ensuring Fairness in the Digital Economy

This research highlights the importance of considering the broader context of social learning when designing algorithms for online marketplaces. While algorithmic fairness is essential, it's not enough to achieve true fairness. We must also address the potential for discriminatory sampling and biased interpretations of ratings. By understanding these subtle mechanisms, we can work towards creating more equitable and inclusive digital environments for everyone.

About this Article -

This article was crafted using a human-AI hybrid and collaborative approach. AI assisted our team with initial drafting, research insights, identifying key questions, and image generation. Our human editors guided topic selection, defined the angle, structured the content, ensured factual accuracy and relevance, refined the tone, and conducted thorough editing to deliver helpful, high-quality information.See our About page for more information.

This article is based on research published under:

DOI-LINK: https://doi.org/10.48550/arXiv.2004.11531,

Title: Statistical Discrimination In Ratings-Guided Markets

Subject: cs.gt econ.th

Authors: Yeon-Koo Che, Kyungmin Kim, Weijie Zhong

Published: 24-04-2020

Everything You Need To Know

How can discrimination persist in online marketplaces even when algorithms are designed to be fair?

Discrimination can persist even with fair algorithms because of how social learning works. Social learning involves a feedback loop between data sampling (who interacts with whom) and informing (how decisions are made based on available information). While the informing part can be designed to be unbiased, the data sampling part is often driven by economic incentives, leading to non-random and potentially biased interactions. For instance, buyers might preferentially select sellers with positive ratings, leading to skewed data sampling that affects the outcomes for different groups.

What role does 'data sampling' play in perpetuating unfair outcomes in rating systems?

Data sampling is a critical element in how discrimination arises within rating systems. It refers to the process where users interact with each other, driven by their economic interests. For example, buyers, aiming for high-value partners, naturally seek sellers with positive ratings, not randomly or representatively. This selective sampling can lead to unfair outcomes because it doesn't provide equal opportunities for all groups or individuals to be evaluated fairly. Biased sampling can result in skewed rating distributions and the reinforcement of pre-existing stereotypes.

In the context of the study, what are 'Group 1' and 'Group 2' and how do they relate to seller productivity?

In the study's model, 'Group 1' and 'Group 2' represent a seller's social group identity. Sellers are categorized into these groups, but their group identity is assumed to be independent of their 'productivity type' (high or low). This means a seller's group affiliation doesn't inherently determine their performance. The model explores how buyers might still use group identity as a signal when making decisions, potentially leading to statistical discrimination if the sampling process is biased.

How does the model's use of binary ratings ('good' or 'bad') influence the potential for discrimination?

The binary nature of the ratings ('good' or 'bad') in the model simplifies the information buyers have about sellers. This simplification creates room for biased interpretation. Buyers use these ratings to guide their decisions about who to interact with. If the sampling is skewed (e.g., due to buyers' preferences for certain groups or those with higher ratings), it affects the information available to the buyers. This can lead to a situation where a group's reputation becomes unfairly associated with 'bad' ratings, even if the underlying productivity distribution is similar to another group, perpetuating statistical discrimination.

What is the significance of the 'alpha' parameter in the model, and how does it relate to fairness in rating systems?

The 'alpha' parameter captures the effectiveness of social learning in the model, specifically, how well the ratings reflect the seller's actual 'productivity type'. A higher alpha means that ratings are more accurate indicators of performance. In a scenario with high alpha (effective social learning), the model might reach a 'non-discriminatory equilibrium,' where sellers from both 'Group 1' and 'Group 2' receive equal treatment. This is because good ratings are associated with high productivity regardless of group identity. The parameter indicates the impact of feedback between data sampling and how the buyer's make decisions. If the Alpha value is low, the market can reinforce pre-existing biases.