Human brain intertwining with computer circuitry, representing human-AI collaboration.

AI Decision-Making: Can Algorithms and Humans Truly Collaborate?

Author's portrait.
Nico Varela in Tech & Innovation May 2025 • 4 min read.

"Explore the pitfalls and potential of combining AI predictions with human judgment for better decisions."


In today's world, decisions are more complex than ever. To help, algorithms are being used in important areas like criminal justice, healthcare, and lending. These algorithms analyze data and make predictions, which are then used to make decisions either automatically or with the help of human agents.

While these human agents may have valuable knowledge, they can also have biases or limitations that don't line up with the algorithm. Take, for instance, a child protection service using an algorithm to assess risk. A social worker might have useful intuition, but also personal biases that affect their judgment. This raises the question: How should we design these algorithms to work effectively with human decision-makers?

New research offers insights into this challenge, revealing how to design algorithms and delegate decisions for better outcomes. The findings suggest that simply providing more information isn't always the best approach, and that policies like always having a 'human-in-the-loop' can sometimes make things worse. By understanding these dynamics, we can create more effective collaborations between humans and AI.

The Delicate Balance: When Should AI Decisions Be Handled by Humans?

Human brain intertwining with computer circuitry, representing human-AI collaboration.

The study models a scenario where a principal (like a manager or policymaker) designs an algorithm to predict a binary outcome (like whether a loan will be repaid or a patient will respond to treatment). The principal must then decide whether to act directly on the algorithm's prediction or delegate the decision to an agent who has private information but may be misaligned with the principal's goals.

The research reveals that delegation is only beneficial if the principal would make the same decision as the agent if they had access to the agent's private information. If the principal's and agent's interests aren't aligned, the principal might prefer to keep control of the decision. This highlights the importance of understanding the incentives and potential biases of human decision-makers when designing AI systems.

  • Delegation Improves Outcomes If: The principal would make the same decision as the agent with the agent's information.
  • Misalignment Risks: Delegation can be counterproductive if the agent's interests diverge from the principal's.
  • Understanding Human Factors: Incentives and potential biases of human decision-makers are crucial when designing AI systems.
Interestingly, the study also finds that providing the most comprehensive algorithm isn't always the optimal strategy, even if the principal can directly act on the algorithm's prediction. Instead, the ideal algorithm might focus on providing more information about one specific outcome while limiting information about others. This suggests that strategic information design is key to effective AI-assisted decision-making.

Avoiding the Pitfalls: Toward Better Human-AI Collaboration

The study cautions that well-intentioned policies, such as always including a human in the decision-making loop or aiming for maximum prediction accuracy, can sometimes worsen the quality of decisions. This is particularly true when there's a misalignment between the algorithm's goals and the human agent's incentives. The findings offer a possible explanation for why human-machine collaborations often underperform in real-world scenarios. By understanding these potential pitfalls and carefully designing algorithms and delegation strategies, we can harness the power of AI to make better, more informed decisions.

Everything You Need To Know

1

What exactly is AI decision-making and why is it being used in important areas?

AI decision-making involves using algorithms to analyze data and make predictions, often in areas like criminal justice, healthcare, and lending. These predictions can be used to make decisions automatically or with the help of human agents. The goal is to improve the quality and efficiency of decisions, but it's important to consider the potential biases and limitations of both algorithms and humans.

2

What does 'delegation' mean in the context of AI-assisted decisions, and when is it actually helpful?

Delegation in AI decision-making refers to the act of assigning the decision-making authority to an agent (usually human) after an algorithm has provided a prediction. Delegation is beneficial when the principal (the one designing the algorithm) would make the same decision as the agent if the principal had access to the agent's private information. Delegation can be counterproductive if the agent's interests diverge from the principal's.

3

I've heard about 'human-in-the-loop' for AI. What does that mean and why might it not always be a good idea?

Human-in-the-loop refers to the practice of always including a human in the decision-making process when using AI. It's a policy aimed at preventing errors or biases in AI decisions. While seemingly beneficial, it can worsen decision quality, especially when there's a misalignment between the algorithm's goals and the human agent's incentives. This happens because human biases or misaligned incentives can override the algorithm's predictions, leading to suboptimal outcomes.

4

What is meant by 'strategic information design' in the world of algorithms, and why does it matter?

Strategic information design means creating algorithms that provide specific information about certain outcomes while limiting information about others. This is important because providing the most comprehensive algorithm isn't always the best strategy. By strategically designing the information provided, decision-makers can avoid information overload and focus on the most relevant factors for making better decisions. This approach acknowledges the cognitive limitations of decision-makers and aims to present information in a way that enhances their judgment.

5

What does it mean when there is a 'misalignment' between what the algorithm wants and what the human wants, and why is that important?

Misalignment between the algorithm's goals and the human agent's incentives refers to a situation where the objectives of the AI system are not in harmony with the motivations or biases of the human decision-maker. This is significant because it can lead to suboptimal or even counterproductive decisions. If a human agent has personal biases or incentives that conflict with the algorithm's predictions, they may override the AI's recommendations, resulting in poorer outcomes. Recognizing and addressing these misalignments is essential for successful human-AI collaboration.

Newsletter Subscribe

Subscribe to get the latest articles and insights directly in your inbox.