AI transforming historical documents into data.

Unlock the Past: How AI is Revolutionizing Historical Occupational Data

Jordan Keane in Tech & Innovation June 2025 • 4 min read.

"Discover how the OccCANINE tool automates HISCO classification, saving researchers time and unlocking new insights into historical trends."

For researchers delving into social and economic history, understanding what people did for a living is crucial. The Historical International Standard Classification of Occupations (HISCO) provides a standardized way to categorize this data, but manually classifying vast datasets is incredibly time-consuming and prone to errors. Imagine spending countless hours poring over census records, marriage certificates, and other historical documents, trying to assign the correct HISCO code to each occupation.

This is where artificial intelligence steps in to revolutionize the process. OccCANINE, a new AI-powered tool, automates the transformation of occupational descriptions into the HISCO classification system. This innovation promises to save researchers significant time and effort while improving the accuracy and scalability of their work.

The AI model simplifies access to historical occupational data, enabling researchers to conduct more extensive and diverse studies. This breakthrough has the potential to unlock new insights into occupational trends and shifts over time, contributing valuable knowledge to economics, sociology, political science, history, and many related fields.

What is OccCANINE and How Does It Work?

AI transforming historical documents into data.

OccCANINE is a transformer language model fine-tuned on 14 million observations of occupational descriptions with associated HISCO codes in 14 different languages. Think of it as an AI that has learned to understand the nuances of historical occupations, capable of recognizing variations in spelling, typos, and even different languages.

The model is trained on vast amounts of data contributed by 22 different research projects, making it highly accurate and versatile. The result is a tool that can take a straightforward textual description of an occupation and accurately determine the most applicable HISCO codes associated with it in seconds or minutes, a job that previously took days or weeks.

Here's what makes OccCANINE a game-changer:

No String Cleaning Required: The model can handle raw text directly, without the need for tedious pre-processing.
High Accuracy: The model is as accurate, if not more so, than a human labeller.
General Understanding: The model understands historical occupations, generalizing well to different settings with little or no fine-tuning.
Fully Replicable: Given the same inputs, OccCANINE will always deliver the same HISCO codes.

The traditional approach to HISCO coding involves classical string matching and cleaning using regular expressions. This process is complex and error-prone due to negations, changing spelling conventions, typos, and transcription errors. OccCANINE replaces all of that with a single step: inputting a raw occupational description and language as context, and receiving HISCO codes as outputs.

The Future of Historical Data Analysis

OccCANINE represents a significant leap forward in historical occupational data processing, effectively breaking down the HISCO barrier. By automating the translation of occupational descriptions into HISCO codes with high accuracy, the model streamlines research in historical social science and paves the way for answering important research questions. This frees up researchers to focus on higher-level analysis and gain deeper insights into the past.

About this Article -

This article was crafted using a human-AI hybrid and collaborative approach. AI assisted our team with initial drafting, research insights, identifying key questions, and image generation. Our human editors guided topic selection, defined the angle, structured the content, ensured factual accuracy and relevance, refined the tone, and conducted thorough editing to deliver helpful, high-quality information.See our About page for more information.

This article is based on research published under:

DOI-LINK: https://doi.org/10.48550/arXiv.2402.13604,

Title: Breaking The Hisco Barrier: Automatic Occupational Standardization With Occcanine

Subject: cs.cl econ.em

Authors: Christian Møller Dahl, Torben Johansen, Christian Vedel

Published: 21-02-2024

Everything You Need To Know

What is OccCANINE, and what problem does it solve for historical researchers?

OccCANINE is an AI-powered tool designed to automate the classification of historical occupational data. It directly addresses the time-consuming and error-prone process of manually assigning HISCO codes to occupational descriptions. Researchers previously spent countless hours poring over documents to determine the correct HISCO code for each occupation. OccCANINE eliminates this tedious manual labor, significantly saving time and improving accuracy, thus freeing researchers to focus on deeper analysis and insights.

How does OccCANINE work, and what makes it different from traditional HISCO coding methods?

OccCANINE employs a transformer language model fine-tuned on a massive dataset of 14 million observations with associated HISCO codes in 14 different languages. It has learned to understand the nuances of historical occupations, including variations in spelling, typos, and different languages. This allows it to accurately determine the most applicable HISCO codes from a raw occupational description in seconds or minutes. Traditional methods involve string matching and cleaning using regular expressions, which are complex and prone to errors due to various inconsistencies in historical data. OccCANINE simplifies this by directly processing raw text input and providing the HISCO codes as outputs.

What are the key advantages of using OccCANINE for historical occupational data analysis?

OccCANINE offers several key advantages. It requires no string cleaning, meaning it can process raw text directly. It boasts high accuracy, often matching or exceeding human accuracy in labeling. It demonstrates general understanding of historical occupations, performing well in different settings without needing extensive fine-tuning. Furthermore, it's fully replicable, always producing the same HISCO codes for the same inputs. These features collectively make OccCANINE a powerful, efficient, and reliable tool for researchers.

What is HISCO, and why is it important for historical research?

HISCO, or Historical International Standard Classification of Occupations, is a standardized system for categorizing historical occupational data. It is crucial for researchers in fields like social and economic history because it provides a consistent framework for understanding what people did for a living across different time periods and regions. By using HISCO, researchers can analyze occupational trends, shifts, and their impact on society, economics, and other related fields.

How will OccCANINE transform historical research, and what kind of new insights might it unlock?

OccCANINE revolutionizes historical research by automating the HISCO coding process, saving researchers significant time and effort. This allows researchers to conduct more extensive and diverse studies, as they can process larger datasets more efficiently. This efficiency can unlock new insights into occupational trends and shifts over time, contributing valuable knowledge to economics, sociology, political science, and history. It enables a deeper understanding of the past and its impact on the present by facilitating research into social and economic changes, labor markets, and the evolution of different industries.