Research Data Guide: Text Data

Research data vary considerably, and therefore the solutions used to manage them also differ. Data management is a form of creative problem-solving and an essential component of responsible and ethical research. The aim of data management is not to apply a single model to all research projects. Rather, guidance and support services are intended to help researchers identify solutions that are appropriate for their data, research design, and research environment. 

Well-designed data management supports the reliability, transparency, reproducibility, and potential reuse of research. Data management encompasses the entire lifecycle of research data, from research planning through data collection or acquisition, processing, analysis, documentation, and storage, to the sharing, archiving, or deletion of data after the completion of the research project. 

Research data may vary, for example, with respect to how they are generated, their data protection and information security requirements, contractual obligations, technical solutions, and opportunities for future reuse. This data-specific guide examines key data management issues from the perspective of text data.

Not all research data can or should be made available for reuse. Data sharing may be restricted, for example, by the protection of personal data, copyright, contractual obligations, or research ethics considerations. Even when the data themselves cannot be published, metadata, documentation, analytical methods, or software code used in the research can often be made available to other researchers. This supports research transparency, the evaluation of research findings, and the broader goals of open science. 

The purpose of this guide is to help researchers identify the key data management issues associated with their particular type of research data and to find practical solutions for different stages of the research process. The guide complements the organisation's general guidance on research data management, data protection, information security, and open science. 

This guide addresses data management issues relating to the preparation and processing of text data. It focuses particularly on datasets that are processed programmatically and transformed into structured formats for research use and publication. Text data may originate from sources outside the research group, such as memory institutions, or they may be collected by the research group itself, for example through the digitisation of books and other documents. While the guide focuses primarily on data obtained from external sources, the solutions presented are also applicable to data collected or generated by individual researchers and projects. 

The guide focuses in particular on transforming data into RDF format (Resource Description Framework, a standard model for describing and exchanging information) and on developing Semantic Web interfaces, such as portals similar to . 

The following key terms are used in this guide: raw data (which may consist of either structured or unstructured textual data), source organisations (organisations from which raw data are obtained), processed data (text data transformed into a structured format), processing pipeline (software through which raw data are transformed into processed data), and portal (the end-user interface). 

Published data should be treated as a research output equivalent to an article or a book. At the planning stage, the research group should consider whether the data generated in the project could be structured and published as research data. Particularly in projects involving large textual datasets, processed data may constitute a significant research output alongside research articles and other traditional publications. 

Structured data may be published directly as a data publication or made available through a dedicated user interface such as a portal. Considering these options requires that data management decisions be planned and anticipated from the beginning of the project, including the documentation of the data and the processing workflow (see Documentation). At the earliest possible stage, researchers should record how the data are collected and processed, what challenges arise during processing, and other relevant information. 

Interdisciplinary collaboration and cooperation between organisations. Research involving large textual datasets is often conducted through interdisciplinary and inter-organisational collaboration. For example, computer scientists may be responsible for processing the data, while historians use the processed data in their research. 

Careful planning at an early stage is particularly important in such collaborative settings. The better the research group structures both the project and its data management practices at the outset, the fewer resources will be spent later on repeating data processing, programming tasks, and other workflows. 

The following questions may support the planning process: 

• Is the data self-collected or obtained from a source organisation? 
• If the data are obtained from a source organisation, what agreements are required and what restrictions do they contain? 
• Which research organisations are involved in the project, and how are data management responsibilities divided among institutions and researchers? 
• How will version control be implemented so that it is unambiguous and understood by all parties? 
• How should the processing workflow be designed to ensure that its stages are reproducible and documented? 
• Into what format will the data be processed, and how will they be described? 
• Which processed datasets will be published, where, and under what conditions? 
• What resources are available for data processing and further development? How should those resources be allocated?

Agreements and legislation form the basis of data processing. In research involving large textual datasets, both contractual agreements and legislation govern the use and publication of data. Where data are obtained from another organisation, agreements concerning the use of the data are of central importance. In contrast, where data are collected by the research group itself, the main constraints on research activities derive from legislation, such as data protection regulations. 

When data are obtained from external sources, it is in the interests of both the research group and the source organisation to establish a clear written agreement concerning the use and publication of the data. Preparing agreements relating to data processing should become a routine practice. 

The agreement need not be complex, but it must be clear and unambiguous. At a minimum, it should specify what data are being provided, to whom, under what conditions, and what data may be published and under which licence or repository. Ambiguous agreements may result in the publication of incorrect data, unnecessary work, and interpretation disputes that waste resources. and available in the University of Helsinki intranet Flamma may be useful when preparing such agreements. 

Constraints arising from agreements. In research involving large textual datasets, agreements with source organisations are particularly important because these organisations provide text or metadata corpora to the research group, and the research cannot proceed without their permission. It is important that the research group is also aware of any restrictions arising from agreements previously concluded by the source organisation concerning the data. Typical requirements may include, for example, filtering out information relating to living persons or removing specific sections of the data prior to publication. 

Constraints arising from legislation. The General Data Protection Regulation (GDPR), in particular, determines how personal data relating to living individuals may be processed and published. Individuals whose data are processed have the right to know what information is collected about them, for what purposes it is used, and who processes it. Before publishing data or a portal based on those data, it is essential that the research group understands the requirements imposed by the GDPR and other relevant legislation. Failure to do so may result in unlawful publication. 

How should researchers address GDPR requirements? If the data consist exclusively of personal data relating to deceased individuals (for example, the ) or are clearly public in nature (for example, ), EU’s General Data Protection Regulation (GDPR) generally does not restrict the production or publication of processed data. 

More complex situations arise when the data contain information about both deceased and living persons or their relatives (for example, ). In these cases, particular attention should be paid to the processing of personal data. A privacy notice should be prepared for the portal and made easily accessible to users. A clear and accessible privacy notice is also an important component of good and responsible data management and publication practices. 

When publishing datasets, researchers must ensure that there is a lawful basis for processing personal data and that data subjects have been informed appropriately. This may mean informing individuals directly or relying on data protection measures implemented by the source organisation. The data protection practices of the source organisation may also define the conditions under which the research group may operate. In addition, researchers are encouraged to consult their organisation's data protection officer and, where necessary, legal experts at an early stage to ensure that the research, data publication, and portal satisfy all legal requirements. 

Transforming data into structured formats may reveal sensitive personal data that would otherwise remain hidden. In the case of , for example, individual pieces of information may not be particularly sensitive or high-risk on their own, but they nevertheless constitute personal data. A key question is whether structuring and processing the data introduces a new informational layer to existing personal data. Structured formats may make it possible to combine or interpret information in ways that are not readily apparent in the original material. If the processing is considered to produce such new and potentially sensitive information, for example information belonging to special categories of personal data under the GDPR, particular care must be taken in handling the data. 

Collect more metadata fields rather than too few. Since text data are often obtained from multiple source organisations, they are typically heterogeneous. Consequently, they almost always require harmonisation and transformation into a structured format suitable for research use. To facilitate this process, it is generally advisable to request more data fields from source organisations rather than too few. This helps avoid situations where the same processing stages need to be repeated later and saves both time and resources during the research process. 

Data structure and file formats. When preparing large textual datasets, attention should be paid both to the structure of the data and to the suitability of the file format for processing. Structure refers to how clearly data elements and records are organised, whereas file format refers to the technical format in which the data are stored (for example, .csv, .txt, .json, or .docx). For programmatic processing, transparent and straightforward formats such as .csv and .json are generally easier to work with than closed or more complex formats such as .docx. At the same time, increasing the level of structure may require transformations that make the data less immediately human-readable. 

In practice, researchers often need to balance structure and usability. It is not always appropriate to pursue maximum structure, particularly if resources are limited. Nevertheless, source organisations should be asked to provide data in the most easily processable form possible, preferably using open and transparent file formats. 

The absence of established procedures may slow down data acquisition. Obtaining data from source organisations often involves lengthy negotiations and clarification, particularly where established practices do not yet exist. Such discussions can delay the start of research. One way to streamline the process is to prepare a standard template for contacting source organisations. This template should describe the data being requested, the planned processing methods, and any intended forms of publication, such as a portal, data publication, or research outputs. Particular attention should be given to the information source organisations require to evaluate the request. 

Source organisation requirements take time. Source organisations may impose requirements regarding which parts of textual datasets may be used or published through a portal or data publication. A common example is the removal of information relating to living persons in order to comply with data protection legislation. Implementing these requirements can be time-consuming. They should therefore be considered during the planning stage and sufficient time should be allocated to this work. 

Even with careful planning and clear agreements, raw data are rarely error-free. Typical issues include errors and inaccuracies resulting from optical character recognition (OCR). In principle, researchers may correct these errors themselves, but in the long term it is often preferable to contact the source organisation and request corrections to the original data. This avoids having to repeat the same corrections whenever the data are processed again through the processing pipeline. However, source organisations may not always have the resources to make corrections, in which case the research group must be prepared to undertake the work itself. It is therefore advisable to allow flexibility in project timelines. 

Reproducibility and documentation of the processing pipeline. The processing pipeline should be designed so that its operation is reproducible and adequately documented. Where appropriate, the process may also be automated, particularly when the goal is to maintain a long-lived user portal. In practice, reproducibility means that when the source organisation updates its raw data, the processed data can be updated by running the material through the pipeline again. Adequate documentation of the processing pipeline and the tools used is essential. Such documentation supports maintenance and troubleshooting, including situations where project personnel change over time. 

Clear communication within the research group. Research involving large textual datasets is frequently conducted through collaboration between multiple organisations. In such situations, communication within the research group is important from a data management perspective as well. Clear and unambiguous communication helps ensure that researchers do not perform the same processing tasks on the same data and that version control remains consistent and understandable. 

Documenting the research process. Research involving large textual datasets often draws on data provided by source organisations. In such cases, it may not always be necessary to document the raw data entirely from scratch. In practice, however, source organisations rarely provide sufficiently detailed documentation, meaning that at least part of this work falls to the research group. Where additional documentation or metadata are required, it is often most efficient to create them as part of the processing workflow. The central focus of this guide, however, is the documentation of the research process itself. This helps ensure research transparency and enables methods and tools to be reused in future projects. Relevant documentation includes information on tools used, software versions, program libraries, source code, and version control practices. 

Publishing metadata separately. The processing pipeline is often described in research articles, but from the perspective of open science it may also be published separately as a metadata record, for example through . Metadata may contain descriptions of the processing pipeline, tools used, and software versions. Such documentation facilitates the sharing and reuse of established methods. 

When selecting a storage solution, researchers should consider factors such as: 

• how many people require access to the data, 
• the size of the dataset, 
• whether the data contain sensitive information, 
• whether the research is conducted in collaboration with other universities. 

Researchers are strongly encouraged to familiarise themselves with available storage solutions. The storage services provided by the University of Helsinki are described in this . Particular caution should be exercised when using commercial cloud services or portable storage media such as external hard drives and USB drives, especially where personal data are involved. 

In projects involving large textual datasets, version control is particularly important when selecting a storage solution. When raw data are repeatedly processed through a workflow, researchers must be able to ensure that changes made during processing can also be reversed. For example, if data become corrupted and the latest backup is outdated, effective version control can save considerable time and resources. 

The following storage solutions are recommended for projects involving large textual datasets. The recommendations are based on storage services available at the University of Helsinki and assume that the data contain at most low-risk personal data. 

• For source code, the University of Helsinki's GitLab service, , is recommended. 
• If a researcher works independently at the University of Helsinki and accumulates less than 100 GB of data, the home directory () is a suitable option, as it provides an adequate level of information security together with granular access control, versioning, and automatic backups. 
• If a researcher works independently or in a group consisting only of University of Helsinki researchers and the amount of data remains below 10 TB, the shared group directory () is a suitable option, as it provides an adequate level of information security together with granular access control, versioning, and automatic backups. 
• If the research group includes collaborators from outside the University of Helsinki, suitable solutions include , , and, where appropriate, . It should be noted that not all of these services are suitable for storing personal data. If personal data must be stored in these services, the data should be encrypted using tools such as Cryptomator or 7-Zip. A commonly used storage solution is . However, researchers should note that data cannot be processed directly within Allas; instead, the data must be downloaded to another storage environment for processing. 

The sharing, archiving, and disposal of research data must comply with the agreements made during the research project, the information provided to research participants, and any legal and ethical obligations governing the data. Even where the research data themselves cannot be shared, metadata, documentation, or methodological descriptions may be published to support research transparency and reproducibility.  

The University of Helsinki Databank, Data archive, and Data catalogue services are available to researchers for the preservation, publication, and sharing of research data, datasets, and metadata after the active phase of research or once a research project has been completed (see the University of Helsinki , , and ). The suitability of these services depends on the nature of the data, access rights, and data protection requirements.  

Textual datasets prepared for research use are often published in two forms in addition to research articles: as a data publication intended for further reuse and as a user-facing portal. A data publication may be made available through services such as , , or . A portal, in turn, is published as a separate website through which the data are presented in a form that is easily accessible to a broader audience. In both cases, the underlying data are the same, but their presentation and intended use differ: the portal serves a broader user community, whereas the data publication is intended primarily for research use. There may be differences between published versions, for example due to requirements imposed by source organisations. In such cases, some information may need to be removed from the data publication, while the portal may direct users to the original source. 

Long-term preservation and maintenance. Long-term sustainability should be considered already during the early stages of a research project. In projects involving textual datasets, published portals and datasets may remain in use long after the project itself has ended. For this reason, it is important to ensure how the functionality of the portal will be maintained over the long term. This requires, for example, a plan for what will happen to the portal once the project ends, how maintenance costs will be covered, and how software updates will be managed. 

Data that are to be deleted must be disposed of securely. Not all research data can or should be retained after a project has ended. If data are to be deleted, they must be deleted in a manner that prevents the files from being recovered and reused. The organisation's information security guidelines should be followed with particular care when disposing data containing personal information or other types of sensitive data (see the guidelines by ). Physical storage media, such as external hard drives, USB drives, and optical discs, should also be disposed of securely. Researchers may seek assistance from their organisation's IT services or information security specialists where necessary (see ). Local IT support () may also be requested to carry out the deletion if the researcher is uncertain about the correct procedure or if the data are highly sensitive. 

Lou, B., O'Brien O'Keeffe, K. & Unsworth, J. (toim.) (2005). Electronic Textual Editing. New York: Modern Languages Association.

Nyhan, J. (2012). Text encoding and scholarly digital editions. Teoksessa C. Warwick, M. Terras & J. Nyhan (toim.), Digital Humanities in Practice (s. 117–138). Facet Publishing.

Oldman, D., Doerr, M. & Gradmann, S. (2015). Zen and the Art of Linked Data: New Strategies for a Semantic Web of Humanist Knowledge. TeoksessaS. Schreibman, R. Siemens & J. Unsworth (toim.), A New Companion to Digital Humanities (s. 251–273). Wiley.

Pitti, D. V. (2004). Designing Sustainable Projects and Publications. Teoksessa S. Schriebman, R. Siemens & J. Unsworth (toim.), A Companion to Digital Humanities. Oxford: Blackwell Publishing. 

Baker, J. (2014). Preserving Your Research Data. Programming Historian.

The data management handbooks are designed as living resources that are continuously updated to meet the evolving practices of the field and the needs of researchers.

You can give feedback on the Text Data guide, such as additional information, comments, or corrections, via this .