Crossroads of AI and human skills, symbolizing the choice between technology and traditional knowledge.

ChatGPT Isn't Always the Answer: Discover When AI Falls Short and How to Adapt

Beau Callahan in Tech & Innovation June 2025 • 4 min read.

"Uncover the surprising limitations of ChatGPT in writing, analysis, and education, and learn strategies to maximize its potential while safeguarding against its pitfalls."

In recent years, generative AI technologies like ChatGPT, Gemini, and MidJourney have advanced rapidly, making significant impacts across various sectors. While literature often highlights the positive influence of ChatGPT on productivity—particularly in areas with extensive training data like English language tasks and Python/SQL programming—a closer look reveals that these tools aren't universally beneficial. In certain contexts, their capabilities may be limited, and their effectiveness can vary substantially among users.

A groundbreaking study by researchers at Chulalongkorn University sheds light on these nuances. Their experiment, which involved economics students performing writing and data analysis tasks, uncovers critical insights into when ChatGPT truly enhances productivity and when it falls short. By understanding these limitations, stakeholders can make informed decisions about implementing AI in education and the workplace.

This article synthesizes the key findings of the research paper, offering a practical guide to navigating the complexities of AI integration. We'll explore the specific scenarios where ChatGPT struggles, discuss the human skills that remain essential, and provide actionable strategies to optimize your approach to generative AI.

ChatGPT's Unexpected Limitations: Why It Doesn't Always Increase Productivity

Crossroads of AI and human skills, symbolizing the choice between technology and traditional knowledge.

The Chulalongkorn University study challenged the assumption that ChatGPT uniformly improves productivity. Economics students were tasked with writing and data analysis assignments, some using ChatGPT and others without. The writing tasks were in Thai, and the data analysis used the statistical software Stata.

The findings revealed a mixed bag. On average, students using ChatGPT performed better in terms of scores and completion time. However, a deeper dive unveiled a critical detail: not everyone benefited. In fact, 34% of participants saw no improvement in writing analysis tasks, and 42% experienced no gain in math and data analysis when using ChatGPT.

Varied outcomes: While some participants saw gains in scores and speed, a significant portion experienced no improvement.
Task-dependent effectiveness: ChatGPT's success hinged on the specific nature of the task, with some areas showing less consistent benefits.
The 'human factor': Individual skills and prior knowledge played a crucial role in determining whether ChatGPT proved helpful.

These findings highlight the importance of considering individual capabilities and task characteristics when integrating AI tools. A one-size-fits-all approach is unlikely to succeed, and careful planning is essential to maximize the benefits of generative AI.

The Enduring Importance of Human Skills in the Age of AI

The Chulalongkorn University study underscores that ChatGPT and similar AI technologies are tools, not replacements for human skills. While AI can augment certain tasks, critical thinking, digital literacy, and domain expertise remain essential. Stakeholders in education and the workplace should prioritize these skills to harness AI's potential effectively and mitigate its limitations. By understanding the interplay between human capabilities and artificial intelligence, we can create a future where technology empowers individuals and drives innovation.

About this Article -

This article was crafted using a human-AI hybrid and collaborative approach. AI assisted our team with initial drafting, research insights, identifying key questions, and image generation. Our human editors guided topic selection, defined the angle, structured the content, ensured factual accuracy and relevance, refined the tone, and conducted thorough editing to deliver helpful, high-quality information.See our About page for more information.

This article is based on research published under:

DOI-LINK: https://doi.org/10.48550/arXiv.2403.0177,

Title: Experimenting With Generative Ai: Does Chatgpt Really Increase Everyone'S Productivity?

Subject: econ.gn q-fin.ec

Authors: Voraprapa Nakavachara, Tanapong Potipiti, Thanee Chaiwat

Published: 04-03-2024

Everything You Need To Know

What are the primary limitations of ChatGPT as highlighted by the Chulalongkorn University study?

The Chulalongkorn University study revealed that ChatGPT's effectiveness is not uniform. While some economics students showed improvement in writing and data analysis tasks using ChatGPT, a significant portion did not. Specifically, 34% of participants saw no benefit in writing tasks, and 42% saw no improvement in math and data analysis when utilizing ChatGPT. This varied outcome suggests that ChatGPT's utility is heavily dependent on the task and the user's existing skill set.

How did the Chulalongkorn University research assess the performance of students using ChatGPT?

The research involved economics students who were assigned writing and data analysis tasks. Some students used ChatGPT to assist them, while others completed the tasks without AI assistance. The study evaluated their performance based on scores and completion time. The tasks included writing assignments in Thai and data analysis using the statistical software Stata, providing a real-world context to assess ChatGPT's impact on productivity.

What skills, as emphasized by the Chulalongkorn University research, remain essential despite the advancements in AI like ChatGPT?

The study underscores the enduring importance of human skills such as critical thinking, digital literacy, and domain expertise. The research indicates that ChatGPT is a tool, not a replacement for these skills. Effective use of AI requires a foundation of human capabilities to interpret, analyze, and validate the AI's output, ensuring meaningful and accurate results.

In what specific areas did the Chulalongkorn University study find ChatGPT's effectiveness to be task-dependent?

The study showed that the success of ChatGPT varied based on the task. While some students experienced gains in both scores and speed, others saw no improvement. The research indicated that the benefit of using ChatGPT depended on the nature of the task itself. Areas like writing, specifically in Thai, and data analysis using Stata software demonstrated varied outcomes. This highlights that ChatGPT's usefulness is not consistent across all types of assignments.

Why is it crucial to consider individual capabilities when integrating tools like ChatGPT, as highlighted by the Chulalongkorn University study?

The research emphasized the importance of individual skills, specifically, that not every student benefited from using ChatGPT. A significant percentage of participants experienced no improvement in their writing and data analysis tasks when using ChatGPT. This points to the 'human factor,' where a user's existing skills and prior knowledge play a crucial role in determining the tool's helpfulness. Therefore, understanding the individual capabilities is essential for maximizing the benefits of generative AI and avoiding a one-size-fits-all approach that may not be effective for everyone.