Digitizing records transforms them from texts to be read to data to be mined (Moss et al., 2018; Mordell, 2019). The increasing volume of unstructured, uncategorized, and highly diverse data has challenged manually performed archival classification and evaluation tasks (Sousa, 2022; Vellino and Alberts, 2016). For Greene and Meissner (2005), the accumulation of unprocessed records is a problem exacerbated by traditional approaches to handling collections. After reviewing the literature and archival practices in the United States and the United Kingdom, they conclude that a lack of professional consensus on the minimum components and work metrics results in a waste of resources and an accumulation of unprocessed records. Thus, they propose the concept of “more product, less process” (MPLP) to allocate resources most efficiently for the benefit of the end users of the collection. Trace (2022) points out that the loss of backlog accumulation interrupts the distribution and final consumption of the research process, which constitutes a knowledge infrastructure problem with clear harm to society. Problems include the obstruction of research, the loss of contextual information from records, a limited understanding of the past, increased preservation costs, a loss of public trust, and reduced visibility of the social and cultural impact of collections.

The challenges we identified can be categorized into three distinct orders. The first challenge is the lack of engagement by archivists with the conceptual principles of new information technology (Bunn, 2019), which has consequently limited the application of computational analysis in support of archival practices. The second challenge lies in the technical ignorance among archivists; this lack of knowledge often results in computational tools being perceived as opaque and insufficiently transparent regarding their actual functioning (Mordell, 2019; Amozurrutia et al., 2023). The third challenge involves promoting the proper use of computational thinking within archives to ensure that archival requirements are met and that such endeavors contribute to the emerging transdisciplinary field known as Computational Archival Science (Marciano et al., 2018).

In the literature, several researchers advocate for the use of computational analysis tools, such as machine learning (ML) and natural language processing (NLP), which identify patterns in data, make predictions, automate repetitive tasks, and speed up processing (Rolan et al., 2019; Colavizza et al., 2021; Bunn, 2019; Jaillant, 2022a; Sousa, 2022; Alaoui, 2024). As presented, automatic records classification appears to help manage the accumulation of records processing tasks, with additional benefits such as identifying and signaling confidential and sensitive information and improving data quality. Additionally, Chabin (2020) demonstrates how the use of archival knowledge and diplomatic analysis improves the performance of artificial intelligence (AI) models and thus enriched the corpus of records from the large-scale “yellow vest” social movement in France in 2019. Shabou et al. (2020) combines an archival model with a data mining model to create an automated method for evaluating the relevance of data recorded in different formats and content. In this context, we identified a lack of research that comparatively analyzes the results and methodologies used in experiments that apply ML methods to the automatic classification of records, especially if they use knowledge and instruments from archival science and information science. Therefore, we formulated two research questions:

(1) What are the guidelines outlined in the literature for developing artificial intelligence projects in records and archives?

(2) Do the automatic records and archives classification experiments reported in the literature meet the guidelines identified in objective 1?

This research is fundamentally rooted in Knowledge Organization (KO). We argue that automatic record classification is not merely a technical data processing challenge but a complex KO problem. Traditional archival practices, such as establishing provenance, maintaining original order, and creating hierarchical descriptions, are sophisticated, human-driven methods of classifying and organizing knowledge, that impose intellectual order on chaos, reveal context, and facilitate access. This paper examines how well the fundamental KO principles of archival science are being translated, preserved, or potentially threatened with the intrusion of data-driven machine learning algorithmic classification. Our goal is to provide a critical lens for the KO community to assess the impact of AI on one of its oldest domains of practice.

This article addresses archival principles and their articulation with classification in section 2, describes and defines artificial intelligence and its subdisciplines in section 3, section 4 addresses the application of AI in records management, and section 5 summarizes the methodological procedures and the literature. The results are discussed in section 6 and the conclusions in section 7.

The management, preservation, and accessibility of records and archival rely on core principles developed over centuries of practice. Before examining how Computational Archival Science (CAS) transforms archival paradigms, we must establish how fundamental archival practices embody Knowledge Organization (KO) principles. This chapter explores key archival tenets—provenance, original order, record group conceptualization, and archival description—demonstrating their intrinsic connections to classification theory and practice.

Contemporary archival theory distinguishes between records and archives. Records are documents created or received during organizational activities and preserved as evidence, characterized by their organicity and primary value (administrative, legal, fiscal). Archives comprise records that, having fulfilled their immediate administrative purpose, are selected for permanent preservation due to their secondary value—evidential, informational, and historical (Pearce-Moses, 2005). This distinction is crucial for automatic classification, as context and metadata differ substantially between active records (associated with workflows) and historical archives (associated with series and fonds). This study encompasses records and archives, applying experiments across the continuum—from current records management to permanent archive organization—demonstrating how Knowledge Organization principles apply distinctly at each stage.

Since the nineteenth century, provenance has been archival science’s theoretical and practical foundation (Tognoli and Guimarães, 2020). The International Council on Archives defines provenance as the relationship between records and the organizations or individuals that created, accumulated, maintained, and used them during personal or corporate activities (ICA, 2007). The principle’s codification through the French respect des fonds (in 1841) established that documents sharing common origin must remain together and be organized systematically (Tognoli and Guimarães, 2020).

Provenance operates as the primary classification mechanism in archival knowledge organization. By establishing that records from a single creator constitute a distinct fond, this principle creates the fundamental classificatory level within archives (Tognoli and Guimarães, 2020). Based on creator-record relationships, this grouping exemplifies core KO practice, delineating knowledge spheres corresponding to creators’ functions and activities. The principle’s classificatory power generates meaningful intellectual boundaries reflecting the administrative realities of record creation and the epistemological frameworks structuring original knowledge production.

Original order (Registraturprinzip in German, respect de l’ordre primitif in French) is intrinsically linked to provenance and emerged from European archival traditions (Tognoli and Guimarães, 2020). This principle mandates maintaining records in the sequence and establishing configuration creators during operational activities, revealing how creators actually operated—their working methods, administrative priorities, decision-making processes, and information management approaches.

The principle of the original order is articulated with the organization of knowledge by being responsible for validating the classification system of its creator, in which it recognizes the original classification scheme as a legitimate and significant structure essential to delimit the context and integrity of the documents. These arrangements represent indigenous classification schemes and filing methodologies developed to meet specific operational requirements (Tognoli and Guimarães, 2020). Archival respect for original order acknowledges existing knowledge organization systems emerging from particular functional contexts. Analyzing creator-imposed structures yields insights into organizational activities, decision-making frameworks, and intended intellectual relationships among documents—central concerns in KO theory and practice.

The scale and complexity of fonds often necessitate subdivision for effective management and access. Archivists typically organize fonds into series, which are records structured according to a filing system or maintained as a unit due to originating from the same accumulation, filing process, or activity (ICA, 2007). The series formation may reflect the creators’ original arrangements or archivists’ subsequent organization based on function, subject, or format.

Delineating fonds and series constitutes explicit classification and hierarchical knowledge organization. The progression from fonds through series (and potentially sub-series, files, and items) creates a classification system organizing records into intellectually coherent units. Classification criteria—administrative function, operational activity, subject matter, or documentary form—derive from the creation context and functional use. Subdividing fonds requires systematic analysis to identify logical groupings and structural relationships, representing fundamental KO activity that reveals internal architecture while maintaining provenance-based integrity.

Archival description creates comprehensive representations that facilitate record identification, management, understanding, and use (ICA, 2007). Duranti (1993) characterizes description as analysis, identification, and organization for “control, retrieval and access”, producing representations that illuminate “archival material, its provenance and documentary context, interrelationships and how it can be identified and used”.

According to Vital and Brascher (2016), archival description and classification are central and inextricably linked activities within information organization and representation. Description structures and presents documentary information and acts as a mediating mechanism between the holdings and their users, enabling efficient management and retrieval. The researchers contend that organizing archival materials fundamentally depends on systematic observation, analytical examination, and synthesizing activities designed to discern commonalities and distinctions among documentary records. At its core, this methodology represents a classification-based approach that creates significant connections and meaningful differentiations throughout archival collections.

Description is the primary mechanism for communicating archival classification and organization to users, demonstrating profound KO integration. Finding aids—description’s primary output—typically mirror provenance and original order principles, reflecting hierarchical classification while articulating each unit’s intellectual and functional basis. Complex knowledge organization systems embedded within archives become navigable and comprehensible through descriptive apparatus.

These foundational principles reveal deep integration with knowledge organization theory. Provenance establishes primary classificatory boundaries; original order preserves organization systems; hierarchical structuring creates manageable intellectual units; description renders organizational systems accessible. This intrinsic relationship between archival principles and KO provides theoretical grounding for understanding how computational approaches simultaneously build upon and transform traditional organizational frameworks—a transformation examined in subsequent sections.

AI is a discipline that researches and develops mechanisms and applications of AI systems, considered those that generate content outputs, predictions, recommendations or decisions to meet human-defined objectives (ISO/IEC, 2022). The evolution of AI has been marked by the emergence of the subdisciplines illustrated in Fig. 1.

Fig. 1.

AI concepts. Reprinted from Banh and Strobel, 2023, p. 63. AI, Artificial intelligence.

The first AI systems were expert systems and knowledge bases that depended on humans to provide established rules to generate their results (Banh and Strobel, 2023). The advent of ML as a subdiscipline of AI has profoundly changed the development of algorithms capable of solving tasks autonomously, as they rely on exposure to data without the need for explicit programming, as shown in Fig. 2.

Fig. 2.

Differences between Traditional Programming and Machine Learning. Reprinted from Shyam and Singh, 2021, p.19.

For automatic text classification, model training data is collected and undergoes cleaning and pre-processing steps to then be submitted to feature engineering. Feature engineering (1) processes the raw data into features that can be understood by the algorithms and (2) transforms them into resources that help algorithms achieve better results in terms of prediction and/or interpretability (Verdonck et al., 2021). The Fig. 3 ilustrates the feature engineering for classification.

Fig. 3.

A general framework of feature engineering for classification. Reprinted from Rawat and Khemchandani, 2017, p. 170.

Mosqueira-Rey et al. (2023) reviewed the literature on new interactions between humans and machine learning algorithms, referred to as Human-in-the-loop machine learning (HITL-ML), where the goal is not solely to achieve higher accuracy or speed, but also to make humans more effective and efficient. The main approaches of Human-in-the-loop machine learning share interactivity as a key factor but have different degrees and purposes of this interactivity (Mosqueira-Rey et al., 2023). Active Learning, Interactive Machine Learning, and Machine Teaching focus on controlling the learning process, while Curriculum Learning and Explainable AI concentrate on improving the efficiency and transparency of learning.

3.1 Trustworthy AI and Explainable AI

The rapid rise of AI has raised concerns about its potential risks and negative impacts. The increasing complexity of AI systems with deep learning algorithms and more recently Generative AI makes it difficult to understand their internal workings and ensure their reliability. Cases like the investigation surrounding the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), which assessed the risk of reoffending and was used by judges in decision-making, revealed that the software generated higher false positive rates for African Americans compared to Caucasian offenders (Mehrabi et al., 2022).

It is in this context that Trustworthy AI emerges, a framework created to ensure that an AI system is worthy of trust based on evidence related to its declared requirements, so that user and stakeholder expectations are verifiably met, as defined by the ISO/IEC 24027:2021 standard (ISO/IEC, 2021). Trustworthy AI studies encompass aspects such as human agency and oversight, robustness and safety, privacy and data governance, transparency, diversity and fairness, social and environmental well-being, and accountability (Chamola et al., 2023).

To ensure Trustworthy AI, human participation is essential at all stages of the AI lifecycle, in what can be called a “Human-Centered Approach to Trustworthy AI (Human + AI)” (Kaur et al., 2023). This collaboration between humans and machines aims to combine human cognitive capacity with the computational power of machines, creating more robust, ethical, and efficient systems.

The explainability of an AI system consists of its ability to express the important factors influencing the outcomes it produces in a way that humans can understand (ISO/IEC, 2022). It is linked to reliability (the property of consistent behavior and results), robustness (the system’s ability to maintain its performance level under any circumstance), transparency (the property of the system to provide due information to stakeholders), and predictability (which should allow stakeholders to make reliable assumptions about the outputs) (Kale et al., 2023).

Provenance in the AI context provides a detailed record of the origin, processing, and transformations of data in a system, answering the “who, what, when, where” questions about each step of a process (Kale et al., 2023). In Fig. 4, the topics of provenance, Explainable AI and Trustworthy AI are presented and their reciprocal relationships.

Fig. 4.

Similarity of topics involving AI provenance, explainable AI, and trustworthy AI. Reprinted from Kale et al., 2023, p. 151. XAI, Explainable AI.

3.2 Methodologies for Developing Projects Using Data Science and AI

The development of projects to extract useful knowledge from large volumes of data had its initial landmark with the creation of the term Knowledge Discovery in Databases by Piatetsky-Shapiro (1990). However, it was only five years later that the Knowledge Discovery in Databases Process was published, a non-trivial and iterative process with the objective of identifying valid, new, potentially useful, and understandable patterns in data. The process converts raw data into knowledge by selecting relevant data, preprocessing it for quality, transforming it into suitable formats, applying data‑mining techniques to extract patterns, and finally evaluating these patterns to derive meaningful insights.

Martinez-Plumed et al. (2019) developed a graphical representation of the evolution of data science methodologies in Fig. 5.

Fig. 5.

Evolution of the most important models and methodologies of Data Mining and Data Science. Reprinted from (Martinez-Plumed et al., 2019), p. 4. KDD, Knowledge Discovery in Databases. CRISP-DM, CRoss-Industry Standard Process for Data Mining. SEMMA, Sampling, Exploring, Modifying, Modeling and Assessing. RAMSYS, RApid collaborative data Mining SYStem. $D^{3}M$, Domain-Driven Data Mining. ASUM-DM, Analytics Solutions Unified Method for Data Mining. CASP-DM, Context-Aware Standard Process for Data Mining. FMDS, Foundational Methodology for Data Science. TDSP, Team Data Science Process. 5A’s, Assess, Access, Analyze, Act and Automate.

According to Fig. 5, the KDD Process had a very relevant role, but with the advent of the Cross-Industry Standard Process for Data Mining (CRISP-DM), most of the major methodologies since then have been directly influenced by the latter. CRISP-DM organizes the process into six phases.

The main contributions to CRISP-DM were based on practical experience, as it was developed by industry leaders from real projects, which is why it became a robust and effective model in the real world (Shearer, 2000), as illustrated in Fig. 6. The model also encourages the adoption of best practices by describing the tasks and subtasks within each phase, highlighting the importance of business understanding, data quality, and evaluation of results.

Fig. 6.

Phases of CRISP-DM. Reprinted from Martinez-Plumed et al., 2019, p. 2. CRISP-DM, Cross-Industry Standard Process for Data Mining.

On the other hand, the TDSP (Team Data Science Process) is an agile and iterative methodology created by Microsoft from CRISP-DM to support solutions for predictive analysis and intelligent applications (Microsoft, 2024). Its main components are the Data Science Lifecycle, the standardized project structure, the recommended infrastructure and resources for data science projects, and recommended tools and utilities for project execution, as illustrated in Fig. 7.

Fig. 7.

TDSP data science lifecycle. Reprinted from (Microsoft, 2024). TDSP, Team Data Science Process.

The Domino DS Lifecycle is an agile methodology developed by Domino Data Lab in 2017 based on CRISP-DM, which adopts a holistic approach covering the entire lifecycle from ideation to delivery and monitoring (Domino Data Lab, 2017).

Finally, the Agile Data Science Lifecycle is a framework created by Jurney (2017) for rapid prototyping, exploratory data analysis, interactive visualization, and applied machine learning, from the perspective of utilizing data science through web applications.

The author elaborates the Agile Data Science Manifesto with principles that support the framework. The first is “Iterate, iterate, iterate”, which emphasizes the need for data to be analyzed, formatted, classified, aggregated, and summarized before being understood (Jurney, 2017).

Silva and Alahakoon (2022), researchers at the Centre for Data Analytics and Cognition, created the CDAC AI Life Cycle to cover all stages of AI development, from conception to production. Due to the increasing sophistication and incorporation of AI in all forms of digital systems and services, CDAC aims to address risk assessment issues related to: Privacy; Cybersecurity; Trust, interpretability, explainability, and robustness; Usability; and Social Implications.

The CDAC is composed of three main phases that unfold into 19 stages, as shown in Fig. 8.

Fig. 8.

CDAC AI life cycle. Reprinted from Silva and Alahakoon, 2022, p. 6. CDAC, Centre for Data Analytics and Cognition AI life cycle. ML, Machine Learning.

The high volume of digital records makes so-called “close reading” unfeasible, which requires archivists to adopt a macroscopic approach supported by computational methods to automate the processing, organization, and analysis of digital records (Ranade, 2016; Moss et al., 2018).

ML automatically associates records with archival categories, and for this the archivist does not need to specifically define the characteristics of each category was done is in a “capstone project” in which emails were categorized according to functional rules and user accounts (NARA, 2014).

However, the statistical modeling inherent in ML, while presenting the advantage of simplifying relationships between input variables and responses (Edmond et al., 2022), is not an exact emulation of nature and remains far from objective. Furthermore, the application of NLP in the organization and description of digital files, as noted by Mordell (Mordell, 2019), often involves opaque interfaces in computational tools, which may lead users to perceive these tools as neutral, disregarding their historically conditioned nature.

The judicious selection of computational tools, considering their functionality, scope, customization options, and data types, is crucial (Underwood and Marciano, 2019). Problems must be reformulated to align with existing computational tools, involving decomposition, rephrasing into known problems, and simplification. A fundamental understanding of how a computational model represents a phenomenon is essential, including the aspects faithfully modeled, those simplified, and the underlying assumptions made. It is also critical to acknowledge that algorithms are formalized opinions in mathematics; they reflect the choices, biases, and intentions of their creators, which can perpetuate and even amplify social injustices (O’Neil, 2020). Consequently, increased transparency and accountability in their development and implementation are paramount.

The consensus, as highlighted by Amozurrutia et al. (2023), is that automated tools should not replace human activity, but rather support it. While AI systems may be capable of generating administrative metadata, as suggested by Cushing and Osti (2023), the necessity for human intervention remains in providing context and understanding events. Therefore, the role of AI lies in supporting human work through a human-machine partnership, where humans retain the ability to verify AI technology outputs for potential issues before making collections available to users. Furthermore, initiatives such as crowdsourcing, as proposed by Ranade (2018), are important for accommodating different, and potentially conflicting, interpretations of documents. This innovative descriptive practice aims to present raw data alongside derived data, qualified with essential confidence measures, promoting a shift in archival thinking based on technical advancements.

The “Traces Through Time” project at The National Archives of the United Kingdom demonstrates how computational techniques can transform access to documents, by recognizing connections between individuals in the collections and automatically transcribing paper documents (Ranade, 2016). This effort highlights the challenge of enhancing computational techniques and addressing the implications of their probabilistic nature and inherent uncertainty for archives, which requires a fundamental shift in archival thinking.

New approaches should focus on improving communication and transparency. Another critical challenge is to ensure that AI usage aligns with core archival concepts, such as archival bonds (Sullivan, 2023). The French government’s use of AI tools to organize information from over 1.5 million contributions in the 2019 “Great National Debate” illustrates how incorporating archival knowledge and diplomatic analysis, particularly the formal elements of contributions, can enrich the document corpus and enhance AI model performance (Chabin, 2020). Indeed, according to Chabin (2020), the diplomatic approach allows for demonstrating the reliability of the material, offering a critical assessment of its representativeness, understanding what is expressed beyond words, and considering the production context including place, time, silences, repetitions, and discourse organization.

The implementation of responsible practices to manage biases is crucial, through symposiums, best practice exchanges, the formation of committees, and the auditing of methods used (Padilla et al., 2019). Moreover, it is vital to ensure the participation of underrepresented and marginalized sectors of society, as proposed by Punzalan and Caswell (2016) as a key aspect of social justice in archival studies.

Computational Archival Science

The CAS is a transdisciplinary field that applies computational methods and resources, design standards, socio-technical constructs, and man-machine interaction to the processing, analysis, storage, long-term preservation, and access problems of large-scale documents and archives (big data) (Marciano et al., 2018). CAS originated from a series of workshops held between 2013 and 2015 on Big Humanities Data in collaboration with King’s College London at the IEEE Big Data Conference (Marciano et al., 2024).

According to Marciano et al. (2018), the definition of CAS presented is still provisional and in constant development due to its recent origin. Transdisciplinarity demands a bidirectional exchange of knowledge among the foundational disciplines, an aspect that the current definition does not yet address satisfactorily. The goals of CAS are to enhance and optimize efficiency, authenticity, veracity, provenance, productivity, computation, informational structure and design, accuracy, and man-machine interaction in support of acquisition, appraisal, arrangement and description, preservation, communication, transmission, analysis, and access decisions. As an important contribution, the authors emphasize the importance of examining the “computational theories and methods that dominate document practices” (Marciano et al., 2018, p. 179).

For Wing (2006), computational thinking is a fundamental skill and a form of thought that involves multi-layered abstractions, the decomposition of a complex task into smaller parts, heuristic reasoning in the discovery and resolution of various problems, even outside the field of computer science. Computational thinking is characterized by its reflection on multiple levels of abstraction, in a way that resembles human rather than computer thinking.

Building on the contributions of Weintrop et al. (2016), CAS seeks to integrate computational thinking with archival thinking in academic teaching and archival practice (Marciano et al., 2018; Underwood et al., 2018). According to the characteristics of each research project, relevant computational thinking practices illustrated in the taxonomy in Fig. 9 are selected and applied, in an effort to structure computational thinking applied to digital archives (Marciano, 2022).

Fig. 9.

Taxonomy of computational thinking practices in mathematics and sciences. Reprinted from Weintrop et al., 2016, p. 9.

Weintrop et al. (2016) consider the growing trend of research in science and mathematics being strongly influenced by computing; they seek to contribute a theoretical foundation for computational thinking to be incorporated into teaching according to the taxonomy presented in Fig. 7.

From the work presented at the 2017 CAS Workshop, Marciano (2022) mapped archival concepts with computational methods in Table 1.

Table 1. Archival concepts and computational methods.

Archival concepts	Computational methods
Transitioning from paper catalog entries to digital catalogs, matching records in distributed databases	Graph and probabilistic databases
Technology-assisted review accessibility for presidential and federal emails accessed in National Archives	Analysis, predictive coding to address personal data
Provenance in terms of why, who, and how	Abstraction and ontology construction
Appraisal	File format characterization, file format policies, bulk extractor (identifies personal data), content visualization, tagging
Classification of archival images	AI, line detection, image segmentation
Document Management	Auto-categorization, auto-classification, e-discovery, machine learning
Personally Identifiable Information (PII)	NLP, entity recognition, sentiment analysis
Structured data interfaces for archival materials	APIs for cultural heritage materials, graph databases
Decentralized recordkeeping	Blockchain, secure computing, reliability

Source: adapted from Marciano, 2022.

AI, artificial intelligence; NLP, natural language processing; APIs, Application Programming Interfaces.

For Mordell (2019), CAS represents a crucial junction that unites two trajectories of archival theory and practice that have so far operated on segregated planes, which are social justice issues and technology. And Pajares et al. (2023) highlight the potential of Computational Archival Science (CAS) with the use of NLP, including creating new datasets, detecting problems in digitized documents, enriching data management, producing quality information for decision-making, and improving document management systems.

The present study adopts a qualitative research design that combines literature review and comparative analysis to address two research questions: (1) identifying and synthesizing existing guidelines for AI projects in archives from diverse literature sources and (2) assessing adherence to reported automatic records classification experiments to these synthesized guidelines.

Comparative analysis examines relationships between macro-level cases to explain differences, similarities, and contextual relations beyond single instances (Esser and Vliegenthart, 2017). This method enables reflexive understanding through contextual comparison, discovers variations across units, and provides means to test and develop causal theories by assessing empirical limits (Azarian, 2011).

Within this research, comparative analysis serves dual functions:

- Guideline Development (RQ1): A comparative synthesis of literature from computer science, archival science, and information science identified recurring themes, critical concerns, and best practices. These elements were analyzed and integrated to formulate coherent guidelines (Section 5.1), establishing literature-grounded benchmarks for evaluation.

- Experiment Evaluation (RQ2): The developed guidelines served as an analytical framework to examine ML experiments for automatic record classification. Each experiment was analyzed according to the guidelines and respective verification rules to assess its adherence. This evaluation reveals how the theoretical considerations translate into practice, which allows identifying strengths, limitations and opportunities for improvement.

The methodological choice addresses the research gap highlighted in the literature on comparative evaluations by structuring the investigation, synthesizing diverse information, and, thus, allowing the systematic evaluation of theory and practice (Esser and Vliegenthart, 2017).

The analysis of the literature enabled the elaboration of the six guidelines for using AI in automatic records and archives classification, which were selected based on the relevance manifested by their occurrence among the majority of the authors researched. Although categorized in different groups, the guidelines have a strong reciprocal influence, as will be addressed below. Next, the analysis of the experiments made it possible to verify how they were developed in comparison with the elaborated guidelines, which enabled us to discuss the results in the sequential order of the guidelines, with a joint analysis of their incidence or non-incidence in the researched experiments. At the end of this chapter, we will discuss the results in general to formulate broader proposals.

The challenge of enabling the use of machine learning in archives finds its way into Computational Archival Science (CAS), a transdiscipline that seeks not only to apply technology to enhance archival practices, but also to transform the founding disciplines into a movement in which the theories and methods of both are integrated and mutually influence each other (Marciano et al., 2018). Table 3 reveals an initial convergence in some of the guidelines originating from the archival literature with the computation methodologies for the use of AI, since in 5 of the guidelines there is a provision for steps in the analyzed methodologies. It should be noted that on the one hand, the guidelines that we have elaborated are specific (Integrate professionals or professors in archives or records management), while on the other hand, the steps of the methodologies are more generic (Business understanding) due to their enormous scope of application, as is evidenced in the very expression “Cross-Industry” of CRISP-DM, one of the studied methodologies.

Table 8 presents the adherence to the guidelines. For guideline no. 1 of “Integrate professionals or professors in archives or records management”, we identified that most of the experiments meet the criteria (54.1%), of which four were projects within the scope of public archives (NARA, NSWSA, National Archives of Korea and Victoria). The form of this collaboration was quite varied, with the prevalence of authors (8), followed by focus groups and interviews with specialists (4), and with archival institutions as project managers (3). The sum is greater than the number of experiments because in some of them, more than one form of collaboration was observed.

In the computer science literature, domain knowledge appears as a fundamental basis to be captured during the experiment, since the ML paradigm requires quality data and pre-existing outputs to train the algorithms (Banh and Strobel, 2023). The human participation of professionals or professors in archives or records management appears as essential in the perspective of Human-in-the-loop machine learning (Mosqueira-Rey et al., 2023), which values human interaction in different types:

$\bullet$ Active Learning: values human annotation to label unmarked data and increase the model’s accuracy, which will also have a strong relationship with guideline no. 3;

$\bullet$ Interactive Machine Learning: values human collaboration to guide learning and perform various tasks, where human specialists are better able to explain how tasks should be performed and with what quality;

$\bullet$ Machine Teaching: values human expertise to transfer knowledge to the machine learning model, as human experts can develop strategies on how to teach the machine according to the peculiarities of the way it learns;

$\bullet$ Curriculum Learning: values human organization of learning to guide the machine efficiently, starting with the presentation of examples in increasing levels of difficulty, which only a human specialist is able to elaborate;

$\bullet$ Explainable AI: values the human capacity to understand and trust AI models based on the generation of explainable models, which depend on the evaluation of human specialists to be improved, which also has a direct relationship with guideline no. 6;

$\bullet$ Usable and Useful AI: values the design of human-computer interaction to create practical and accessible solutions, in which human experts can contribute based on the knowledge they have of how users interact with the documents and thus assist in the design and evaluation of the AI system.

To maximize the contribution of specialists, Marciano (2022) suggests that the development of AI solutions should be iterative, involving the prototyping, testing, and adaptation of the solutions. In this way, the refinement and continuous improvement of the model is allowed based on the feedback from archivists and the analysis of results (Amozurrutia et al., 2023).

As Tognoli and Guimarães (2020) emphasize, provenance operates as the primary classification mechanism in archival knowledge organization, creating meaningful intellectual boundaries that reflect the administrative realities of record creation. However, in the experiments reviewed, provenance consideration frequently manifests as simple metadata extraction rather than the deep contextual understanding that archival theory demands. For instance, while Wang et al. (2021) included “issuing institution” and “receiving institution” as variables, and Kim et al. (2017) incorporated “metadata of related records”, these approaches fail to capture what Duranti (1993) characterizes as the full scope of “archival material, its provenance and documentary context, interrelationships and how it can be identified and used”. The experiments demonstrate technical acknowledgment of provenance without achieving the interpretive depth that Chabin (2020) achieved in the French Great National Debate, where diplomatic analysis enriched the AI model’s understanding of documentary context.

The much lower adherence to guidelines that reflect original order (implicit in preserving context) and the complete absence of algorithmic accountability (the new form of archival description) are telling. This suggests that the ML experiments analyzed succeed at a rudimentary level of content-based categorization but largely fail at the more sophisticated KO task of preserving and representing the complex, evidence-based relationships that define an archive. They classify records as isolated items, not interconnected components of a larger, organic knowledge system. This trend points to a significant risk: the efficiency of automatic classification may come at the expense of the very contextual richness that archival KO practices were designed to protect.

In stark contrast, the work of Payne (2023) with the STACC (State, Temporality, Attachment, Context, and Content) framework exemplifies a deep approach. Rather than merely using authorship metadata, STACC attempts to model archival reasoning itself, operationalizing complex concepts like the “archival bond” and the “transactional/supporting” nature of a document into computationally tractable features. This distinction is fundamental: whereas the superficial approach performs content categorization assisted by metadata, the deep approach aims for true archival classification that preserves context. The prevalence of the former over the latter explains the central paradox of this study: a high formal respect for provenance coexisting with a systemic failure to preserve the archive’s relational integrity, effectively treating it as a mere collection of data to be mined, not an organized knowledge system.

The computer science literature seems very convergent with that of archival science regarding guideline no. 3 “Taking care of data quality”. The data science and AI methodologies CRISP-DM, RAMSYS, DST, and TDSP expressly provide for steps to deal with data quality, as expressed in Table 3. And Trustworthy AI requires reliable data beyond the traditional criteria of precision, completeness, and consistency, to also encompass requirements related to equity and the prevention of biases (Kaur et al., 2023), which reveals a strong relationship with guideline no. 5 to be addressed below.

A little more than half of the experiments meet guideline no.3 (54.1%), which was manifested in different forms, such as the need for human review of the data set used in model training to ensure the production of better results (Binici, 2019). In five experiments (20.8% of the total), the Kappa indicator was used, a statistical measure that measures the degree of agreement or agreement between two or more evaluators in the range of –1 to 1, thus offering a more precise assessment of consistency between assessments. However, the greater investment in data quality has the side effect of reducing the set of documents suitable for use. The experiment with the largest number of documents was Liu et al. (2017) with 3600 documents, which is far from being a representative sample of the diversity of most document collections of institutions today. The need for prior annotation of all documents is linked to the use of ML in supervised learning (Emmert-Streib and Dehmer, 2022), which was the option of practically all the experiments analyzed (95.8%). Therefore, a possible path may be the use of semi-supervised learning, which uses the combination of a small set of annotated documents that will serve as a basis for the automatic annotation of a large volume of documents. This path is advocated by Shu et al. (2020), who consider the need to elaborate large-scale annotated training data for model learning, which has proven to be unfeasible given the high costs and a lot of time to be invested, as well as situations related to the privacy of personal data.

The problems highlighted by Kim et al. (2017) and Binici (2019) in the classification plan regarding hierarchical structure, lack of specific classes, and absence of scope notes would have influenced the quality of the training data due to greater divergence between annotators. And in turn, the problems accumulated in these two groups may end up damaging the results obtained by the classification model. Records classification plans, as knowledge organization systems, can provide an explicit representation of the relationships between records, in order to facilitate the understanding of the context and importance as evidence (Frendo, 2007), which we consider an essential stage of projects. A classification plan with clear and mutually exclusive classes helps train the prediction model, as the algorithms are trained from records that fall into each category and from iterative reviews of additional records (NARA, 2014).

Regarding guideline no. 4 “Include different perspectives”, which was verified in 33.3% of the experiments, the ISO/IEC TR 24027:2021 standard (ISO/IEC, 2022) states the need for the project team to have a diverse composition, including contributions from experts in ethics, social sciences, quality, and privacy. One of the risks is social bias, which consists of the unfair treatment shared in a society, which may be encoded and perpetuated through organizational policies and ML models that learn historical patterns. Hence, the need to establish strategies for treating bias at all stages of the life cycle of an AI system, from the analysis of requirements to deployment and monitoring, which involves consideration of external and internal requirements, acceptance criteria, training data, adjustment, fairness metrics, among other points.

The forms of inclusion of different perspectives in the experiments were done in different ways. We can mention the meeting of a multidisciplinary advisory board composed of cybernetics scholars and technologists and the test with a focus group to understand the needs of different users done by Anderson (2021). This experiment is in line with Padilla’s suggestion (2019) to form committees and audit the methods used as a proposed measure to mitigate these risks. Alberts and Forest (2012) included the perspective of users of email classification systems, and Franks (2022) also included researchers. In five experiments, groups of annotators were integrated for the same documents as a measure to capture different views and calculate the inter-annotator agreement indicator (Cohen et al., 2004; Goldstein and Evans Sabin, 2006; Lampert et al., 2010; Vellino and Alberts, 2016; Payne, 2023).

For guideline no. 5 of “Ensuring algorithmic accountability”, the literature has advanced significantly, but it is a very recent topic, in which the definition of Trustworthy AI was developed only in the early 2020s (ISO/IEC, 2021). This seems to be a determining reason why none of the experiments managed to meet the minimum requirements of the guideline, which range from data protection principles such as privacy and ethical principles to the tracking and auditing of results. Of all the data science and AI methodologies, only CDAC, which is the most recent of all, contemplates the “Review data and AI ethics” phase (Silva and Alahakoon, 2022). This is a phase that goes far beyond the simple technical verification of the data because it represents a deep commitment to responsibility, inclusion, and the mitigation of risks inherent in the implementation of ethical AI systems that are fair and aligned with the values of society, in addition to being effective.

To face these challenges, Kaur et al. (2023) propose the “Human-Centered Approach to Trustworthy AI (Human + AI)”, which provides for human participation in all stages, from the design of the AI system, through development and use, and to subsequent supervision, which is close to the understanding of Cushing and Osti (2023). As a way to materialize this approach, Kaur et al. (2023) propose several techniques and methods to meet the requirements of Trustworthy AI in relation to equity, explainability, accountability, privacy, and acceptance. Due to the complex and probabilistic nature of many AI systems, the authors understand that traditional software testing techniques may be insufficient and, therefore, consolidated some approaches for the verification and validation of Trustworthy AI which are the Metamorphic Test (verification of the system based on the relationships between its inputs and outputs, without depending on an oracle to define the expected result), Expert Panels (human experts evaluate the performance and safety of AI systems), Benchmarking (comparison of the AI system’s performance with established benchmarks, using standardized data sets to assess their accuracy and robustness), Field Testing (evaluation of the AI system in real environments with the observation of its behavior in practical scenarios and with real users), and Comparison with Human Intelligence (performed in specific tasks to assess the ability of the AI system to replicate or surpass human intelligence).

An important concept for Trustworthy AI is the provenance of AI systems, which seeks to provide a detailed record of the origin, processing, and transformations of data about each step of the process (Kale et al., 2023; Rolan et al., 2019). AI provenance brings as benefits increased transparency, support for explainability, promotion of reproducibility, help in identifying biases, and facilitation of auditing and debugging, which are valued by Jaillant (2022a). The models, languages, and tools indicated to document AI provenance are the PROV (Provenance Data Model) of the W3C, ProvStore, Prov Viewer, and initiatives such as OpenML and ModelDB, in addition to the Renku and WholeTale platforms for practical application in research environments. We can perceive a strong link between guideline no. 5 and guidelines no. 3 (data quality), no. 4 (inclusion of different perspectives), no. 6 (explainability), and no. 7 (privacy), which reveals that this is the guideline with the largest number of dependencies with different work fronts, which helps to understand its high complexity and, on the other hand, the challenges in applying it in experiments.

In guideline no. 4 “Provide explainability for predictions”, 16.7% managed to meet the minimum requirements. Burkart and Huber (2021) indicate that the use of self-interpretable (“interpretable by nature”) supervised machine learning models, such as Decision Tree, Naive Bayes, and kNN, can be considered a research decision that favors explainability. There were nine experiments that used “interpretable by nature” models as shown in Table 6. However, the use of such models was not accompanied by the researchers’ justification that the choice was made to enable the explanation of the generated predictions. It is so certain that only four experiments adopted measures to favor the explainability of the results. In the research project of Anderson (2021), explainability is favored by making the documents publicly available, as well as open access to all the code and application used.

The other three experiments favored explainability through the qualitative evaluation of the generated predictions, which proved to be very important for understanding the functioning of automatic classification, with its gaps to support actions to improve the results. Alberts and Forest (2012) developed rankings of discriminant features with the chi-square value for each of the categories, which help the reader to interpret the model results and analyze the document collection itself. However, it should be noted that this analysis was exploratory since the authors did not use the conclusions of the most important characteristics to eliminate the less important ones and thus optimize the results and the model’s own explainability. Next, the authors discovered that the most discriminating non-lexical characteristics were related to the sender and the addressee (e.g., social network, hierarchy, mailing lists). The presence of formatting (bold, capital letters) and attachments also proved to be relevant. Concerning the lexical characteristics, the keywords specific to each category were identified (e.g., “approve” for action-authorization, “meeting” for action-meeting), and the lexical analysis revealed nuances in the language and tone of each category.

Kim et al. (2017) identified the types of errors in the model results as class coverage, classification error due to a lack of correspondence between the content and the class indicated by the AI, classification errors in the learning data, class ambiguity, overfitting, and situations in which it was not possible to determine the cause. Binici (2019) lists reasons related to the classification plan that cause the algorithm to make mistakes, such as the hierarchical and content structure of the plan, including the lack of proper classes. He points out that the uncertainty caused by the simplicity of class titles and the lack of additional explanations in the filing system creates uncertainty, the use of similar subtitles in different hierarchies causes confusion, the presence of more than one subject in the document creates uncertainty in the notation, and carelessness in the annotation can motivate errors. In these two experiments, we can see that the focus was on explaining the reasons for the wrong predictions, which brings important conclusions by detecting errors in the quality of the data and in the classification schemes used, which can be later corrected so that the model can be trained again and thus generate better results. However, the focus of these two experiments only on the errors ends up bringing a partial explanation about the model’s predictions. One problem with this approach is that it does not analyze whether the correct classifications were made based on discriminant features that are not based on archival theory.

As warned by Ranade (2018), we cannot sacrifice the nuances in the confusing data of the archives through its improper cleaning or the imposition of structure or order where they did not exist. This is because archival data are probabilistic in nature, and dealing with this reality should require transparency regarding this uncertainty with the adoption of tools and approaches that embrace it.

Once the individual analysis of each guideline has been completed, from this point we will address general aspects of the research results. A very significant challenge of this research resided in the diversity of methodological procedures used in the analyzed experiments. On the other hand, we did not locate methodologies or frameworks that could help in the comparison of automatic classification experiments of archival documents. Therefore, based on a review of the theoretical literature in the area, it was possible to elaborate the six guidelines based on theories in the area that allowed us to first identify the main research questions related and, second, to serve as solid parameters for evaluating the experiments carried out. Another important challenge to be highlighted is that the technical knowledge of computer science required a dual effort in the research to (i) understand the concepts, methods, and techniques of AI and (ii) identify points and forms of articulation with archival science.

Moving forward, the field requires what Wing (2006) describes as true computational thinking: multi-layered abstractions that reflect human rather than computer thinking. In the archival context, this means developing algorithms that can represent and process the complex relationships Duranti (1993) identifies—not just between documents but between documents and their contexts, creators, and communities. It means creating systems embodying what Marciano et al. (2018) envision for CAS: genuine transdisciplinary integration that transforms archival and computational practice.

It seems to us that a solution to advance studies on automatic classification in records and archives is to structure a methodological framework, which can be defined as a structured practical guideline or a tool to guide the user through a process, which is organized in stages or in a step-by-step approach (McMeekin et al., 2020). This is because the data-formatted documents demand the need to create tools, methodologies, and approaches for using and analyzing files as data, as proposed by Moss et al. (2018). The methodological framework includes a “set of structured principles” and a “sequence of methods” (McMeekin et al., 2020), as well as an “approach to make explicit and structure how a certain task is used” (Andrade et al., 2009), and a “body of methods, rules, and postulates employed by a specific procedure or set of procedures” (Rivera et al., 2017, used by Rivera et al., 2017). The methodological framework would bring as benefits improving the consistency, robustness, and documentation of activities (Rodgers et al., 2016), increasing the quality of research, standardizing approaches (Squires et al., 2016), as well as maximizing the reliability of results (Kallio et al., 2016).

This article sought to bridge the gap between classification theory and AI practice by answering two guiding questions: (1) What are the fundamental guidelines for applying AI to records and archives? Furthermore, (2) How closely do current experiments in automatic classification adhere to them? We begin by grounding our investigation in the fundamental archival principles for classifying and organizing knowledge, as detailed in Section 2, establishing that provenance, original order, and hierarchical description are not mere administrative procedures but rather sophisticated forms of knowledge organization (Tognoli and Guimarães, 2020; Duranti, 1993). From this foundation, we synthesize a multidisciplinary literature review into six evidence-based guidelines for implementing AI and conduct a comparative analysis of 24 experiments based on this new framework.

Our analysis reveals a critical paradox. From one perspective, the experiments demonstrated significant respect for the principle of provenance (75.0% adherence to Guideline no. 2), which recognizes the creator-record relationship fundamental to classification within the KO framework. This suggests that the core of archival thinking has permeated computational practice. However, this approach has remained superficial for the most part, failing to capture the deeper contextual integrity preserved by the principles of the original order or the rich, hierarchical relationships articulated by archival description. The result is a “datafication” of records that respects their provenance but often strips them of their organic, narrative, and evidentiary structure. It is as if the treatment accorded to records and archives fails to distinguish them from mere data collection.

This study aimed to synthesize guidelines for developing AI projects in records and archives and to assess whether existing automatic records classification experiments adhere to these guidelines. Through a comprehensive literature review spanning computer science, archival science, and information science, we identified six critical guidelines for the application of ML in the automatic classification of records.

The core findings emphasize the need for a collaborative, context-aware, and ethically grounded approach to AI implementation in archival settings. Our analysis of 24 experiments applying machine learning to automatic records classification revealed a mixed landscape. While some guidelines, such as considering provenance and context (75.0%) and integrating archival expertise (54.1%), were reasonably well-addressed, others, particularly algorithmic accountability (0.0%) and explainability (16.7%), were largely neglected. This highlights a significant gap between the theoretical understanding of ethical AI principles and their practical implementation in records and archival contexts. The relative strength in considering provenance suggests a recognition of the inherent value of archival principles, but the weakness in accountability and explainability points to a need for greater awareness and application of Trustworthy AI frameworks.

These findings have significant implications for both theory and practice. Theoretically, this research contributes to the emerging field of Computational Archival Science (CAS) by providing a structured set of guidelines for developing AI-driven archival systems. The guidelines synthesize knowledge from diverse disciplines and offer a framework for integrating computational methods with core archival principles. Practically, our findings can inform the design and implementation of future AI projects in records and archives, promoting more responsible and effective use of these technologies. They also serve as a checklist for evaluating existing systems and identifying areas for improvement.

This study has limitations. First, the assessment of compliance with the guidelines was based on published articles, which may not fully capture the methodological nuances of each experiment. Second, the binary assessment of compliance did not allow for distinguishing levels of adherence to the verification rules. Future research could address these limitations by developing a comprehensive methodological framework specifically designed for evaluating AI applications in archival contexts is paramount. This framework must have as elements the objectives, key concepts, principles (the basic requirements for trustworthy AI), guidelines (the standards to follow for developing AI models), work processes (the necessary steps to the implementation of AI projects), tools and models (the AI frameworks and algorithms), and metrics and indicators (quantitative and qualitative measurements for the assessment of results).

Future research should also focus on developing concrete strategies for implementing the guidelines identified in this study. This includes developing tools and techniques for data quality assessment, bias detection and mitigation, and explainable AI. Furthermore, research is needed to explore the ethical implications of AI in archives, including issues of privacy, access, and representation.

In conclusion, our findings underscore the importance of a collaborative, context-aware, and ethically grounded approach to AI, ensuring that these tools are used to enhance, rather than undermine, the core values of archival science and the integrity of the historical record. This work serves as a foundation for future research and practice, promoting the development of more transparent, accountable, and equitable AI systems for records and archives.

There is no data and material beyond what is mentioned in the article.

EW and RTBS designed the research study, performed the research and analyzed the data. EW drafted the manuscript. Both authors contributed to critical revision of the manuscript for important intellectual content. Both authors read and approved the final manuscript. Both authors contributed to editorial changes in the manuscript. Both authors have participated sufficiently in the work and agreed to be accountable for all aspects of the work.

Not applicable.

This research received no external funding.

The authors declare no conflict of interest.