More Efficient Manual Review of Automatically Transcribed Tabular Data
Any machine learning method for transcribing historical text requires manual verification and correction, which is often time-consuming and expensive. Our aim is to make it more efficient. Previously, we developed a machine learning model to transcribe 2.3 million handwritten occupation codes from...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
International Institute of Social History
2024-04-01
|
| Series: | Historical Life Course Studies |
| Subjects: | |
| Online Access: | https://hlcs.nl/article/view/15456 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832569382577897472 |
|---|---|
| author | Bjørn-Richard Pedersen Rigmor Katrine Johansen Einar Holsbø Hilde Sommerseth Lars Ailo Bongo |
| author_facet | Bjørn-Richard Pedersen Rigmor Katrine Johansen Einar Holsbø Hilde Sommerseth Lars Ailo Bongo |
| author_sort | Bjørn-Richard Pedersen |
| collection | DOAJ |
| description |
Any machine learning method for transcribing historical text requires manual verification and correction, which is often time-consuming and expensive. Our aim is to make it more efficient. Previously, we developed a machine learning model to transcribe 2.3 million handwritten occupation codes from the Norwegian 1950 census. Here, we manually review the 90,000 codes (3%) for which our model had the lowest confidence scores. We allocated these codes to human reviewers, who used our custom annotation tool to review them. The reviewers agreed with the model's labels 31.9% of the time. They corrected 62.8% of the labels, and 5.1% of the images were uncertain or assigned invalid labels. 9,000 images were reviewed by multiple reviewers, resulting in an agreement of 86.4% and a disagreement of 9%. The results suggest that one reviewer per image is sufficient. We recommend that reviewers indicate any uncertainty about the label they assign to an image by adding a flag to their label. Our interviews show that the reviewers performed internal quality control and found our custom tool to be useful and easy to operate. We provide guidelines for efficient and accurate transcription of historical text by combining machine learning and manual review. We have open-sourced our custom annotation tool and made the reviewed images open access.
|
| format | Article |
| id | doaj-art-3df0fc3fd40f45c8966b91efe1c76b74 |
| institution | Kabale University |
| issn | 2352-6343 |
| language | English |
| publishDate | 2024-04-01 |
| publisher | International Institute of Social History |
| record_format | Article |
| series | Historical Life Course Studies |
| spelling | doaj-art-3df0fc3fd40f45c8966b91efe1c76b742025-02-02T21:48:29ZengInternational Institute of Social HistoryHistorical Life Course Studies2352-63432024-04-011410.51964/hlcs15456More Efficient Manual Review of Automatically Transcribed Tabular DataBjørn-Richard Pedersen0Rigmor Katrine Johansen1Einar Holsbø2Hilde Sommerseth3Lars Ailo Bongo4Norwegian Historical Data Centre, UiT The Arctic University of NorwayDepartment of Health and Care Sciences, UiT The Arctic University of NorwayDepartment of Computer Science, UiT The Arctic University of NorwayNorwegian Historical Data Centre, UiT The Arctic University of NorwayDepartment of Computer Science, UiT The Arctic University of Norway Any machine learning method for transcribing historical text requires manual verification and correction, which is often time-consuming and expensive. Our aim is to make it more efficient. Previously, we developed a machine learning model to transcribe 2.3 million handwritten occupation codes from the Norwegian 1950 census. Here, we manually review the 90,000 codes (3%) for which our model had the lowest confidence scores. We allocated these codes to human reviewers, who used our custom annotation tool to review them. The reviewers agreed with the model's labels 31.9% of the time. They corrected 62.8% of the labels, and 5.1% of the images were uncertain or assigned invalid labels. 9,000 images were reviewed by multiple reviewers, resulting in an agreement of 86.4% and a disagreement of 9%. The results suggest that one reviewer per image is sufficient. We recommend that reviewers indicate any uncertainty about the label they assign to an image by adding a flag to their label. Our interviews show that the reviewers performed internal quality control and found our custom tool to be useful and easy to operate. We provide guidelines for efficient and accurate transcription of historical text by combining machine learning and manual review. We have open-sourced our custom annotation tool and made the reviewed images open access. https://hlcs.nl/article/view/15456Population census dataMachine LearningHistorical dataManual reviewOccupation codesNorway 1950 |
| spellingShingle | Bjørn-Richard Pedersen Rigmor Katrine Johansen Einar Holsbø Hilde Sommerseth Lars Ailo Bongo More Efficient Manual Review of Automatically Transcribed Tabular Data Historical Life Course Studies Population census data Machine Learning Historical data Manual review Occupation codes Norway 1950 |
| title | More Efficient Manual Review of Automatically Transcribed Tabular Data |
| title_full | More Efficient Manual Review of Automatically Transcribed Tabular Data |
| title_fullStr | More Efficient Manual Review of Automatically Transcribed Tabular Data |
| title_full_unstemmed | More Efficient Manual Review of Automatically Transcribed Tabular Data |
| title_short | More Efficient Manual Review of Automatically Transcribed Tabular Data |
| title_sort | more efficient manual review of automatically transcribed tabular data |
| topic | Population census data Machine Learning Historical data Manual review Occupation codes Norway 1950 |
| url | https://hlcs.nl/article/view/15456 |
| work_keys_str_mv | AT bjørnrichardpedersen moreefficientmanualreviewofautomaticallytranscribedtabulardata AT rigmorkatrinejohansen moreefficientmanualreviewofautomaticallytranscribedtabulardata AT einarholsbø moreefficientmanualreviewofautomaticallytranscribedtabulardata AT hildesommerseth moreefficientmanualreviewofautomaticallytranscribedtabulardata AT larsailobongo moreefficientmanualreviewofautomaticallytranscribedtabulardata |