Research Data Guide: Social Media Data

Research data vary considerably, and therefore the solutions used to manage them also differ. Data management is a form of creative problem-solving and an essential component of responsible and ethical research. The aim of data management is not to apply a single model to all research projects. Rather, guidance and support services are intended to help researchers identify solutions that are appropriate for their data, research design, and research environment. 

Well-designed data management supports the reliability, transparency, reproducibility, and potential reuse of research. Data management encompasses the entire lifecycle of research data, from research planning through data collection or acquisition, processing, analysis, documentation, and storage, to the sharing, archiving, or deletion of data after the completion of the research project. 

Research data may vary, for example, with respect to how they are generated, their data protection and information security requirements, contractual obligations, technical solutions, and opportunities for future reuse. This data-specific guide examines key data management issues from the perspective of social media data.

Not all research data can or should be made available for reuse. Data sharing may be restricted, for example, by the protection of personal data, copyright, contractual obligations, or research ethics considerations. Even when the data themselves cannot be published, metadata, documentation, analytical methods, or software code used in the research can often be made available to other researchers. This supports research transparency, the evaluation of research findings, and the broader goals of open science. 

The purpose of this guide is to help researchers identify the key data management issues associated with their particular type of research data and to find practical solutions for different stages of the research process. The guide complements the organisation's general guidance on research data management, data protection, information security, and open science. 

This guide concerns content collected from social media platforms for research purposes (social media data) as well as other forms of digital traces. From the researcher's perspective, it is important to understand the operating principles of the social media platform that is the subject of the research. It is also important to actively monitor changes in social media platforms, as the field evolves exceptionally rapidly. The distinctive characteristics of social media data mean that there are many details that require attention. This should not discourage researchers, however; social media data can provide valuable opportunities for research. See also the . 

In social media data collection, previously agreed principles facilitate the conduct of research. Since social media data collection usually begins quickly after an interesting phenomenon has been identified, successful data management depends on having agreed principles and procedures within the research group in advance. By planning data collection beforehand, a research group can save considerable time, effort, and resources. The key questions listed below are matters for which it is advisable for the research group to establish standard solutions in advance. Otherwise, decisions may need to be made under time pressure, potentially causing immediate problems (for example, collected data may not support the aims of the study) or longer-term difficulties (for example, data processing may be complicated by poorly compatible tools). These issues should be addressed as early as possible (see also Venturini et al. 2018). 

The most important matters to plan and clarify in advance include: 

• Which platforms will the data be collected from? 
• How do the terms of service of the platform under study enable or restrict data collection? 
• What data will be collected? For example, will the collection focus on particular keywords or users? 
• Which tools will be used for data collection, and who is responsible for the collection process? 
• Where will the data be stored, and who will have access to it? 
• How will the data be processed and analysed? Which tools will be used? In a collaborative project, who is responsible for processing which parts of the data? 
• How will the data collection process be documented? 

Does the planned processing of data create a high risk for research participants? At the University of Helsinki, researchers can use the  questionnaire to support this assessment. 

Familiarise yourself with the platform's End-User License Agreement (EULA). The End-User License Agreement (EULA), terms of services and platform policies of social media platforms (alongside developer agreements, discussed below) also shape research activities, even if the EULA does not specifically mention research use. In such cases, the licence agreement and platform policies should be considered in relation to applicable legislation and principles of fair use, citation rights, or private-use provisions in Finnish legislation (compare to fair use in US copyright law, and quotation rights or private-use provisions in Finnish legislation) (see, for example, Laaksonen & Salonen 2018; Obar & Oeldorf-Hirsch 2020). 

The collection of social media data may also be governed by a developer agreement. Social media data collection largely takes place on the terms set by platform providers, and researchers typically need to accept a developer agreement that defines the conditions under which data may be collected and retained. By accepting a developer agreement, a researcher gains access to social media data through an application programming interface (API) or another research interface. API access may involve costs, which should be considered when preparing the project budget. 

The terms of service and agreements made with social media platforms may restrict the collection of online content or data harvesting, that is, computer-assisted or automated data collection from publicly available websites (commonly referred to as web or data scraping). Many social media platforms direct researchers towards their official APIs and may impose restrictions on such harvesting through their terms of service. However, terms of service and technical interfaces change continuously, and researchers should therefore verify the current requirements before collecting data. 

In practice, research-related data harvesting is not a straightforward issue. In addition to platform terms of service, copyright law, data protection regulation, and special provisions relating to research, such as those concerning text and data mining, may also affect the situation (see for example the (only available in Finnish)). Consequently, both legal and research ethics considerations should be assessed when planning data collection. 

There is also an ongoing discussion within the research community concerning the extent to which researchers should comply with platform terms of service when these limit the study of socially significant phenomena such as public debate, election campaigns, or misinformation (see, for example, Bruns 2019; Rogers 2018; Sandvig 2017). Researchers should therefore be aware that platform terms of service, research objectives, and the opportunities provided by legislation are not always fully aligned. 

Social media data purchased from commercial providers is subject to separate contractual arrangements. Social media data may also be acquired for research purposes by purchasing datasets collected by commercial providers. This requires an agreement with the provider, and such agreements may impose restrictions comparable to those found in developer agreements. Researchers should therefore review these agreements carefully before signing them. 

Fulfilling the obligation to inform participants in social media research. Under the General Data Protection Regulation (GDPR), individuals participating in research must be informed about the collection and processing of their personal data. In research based on social media data, fulfilling this obligation is not always straightforward. Users have generally accepted the platform's terms of service and privacy policies, which describe the processing of personal data within the service. However, this does not necessarily mean that users have been informed about the specific research project or the associated processing of personal data. 

In practice, it may be impossible for a researcher to inform every individual whose content is included in a research dataset. Social media data are often collected in multiple languages from hundreds, thousands, or even millions of users, and some user accounts may no longer exist by the time the research is published. 

In certain limited circumstances, the obligation to inform participants may be waived. This may be the case, for example, when personal data have been acquired from a source other than the data subjects themselves and providing individual notifications would be impossible or would require disproportionate effort. In such situations, researchers may provide information about the study publicly, for example through the project website, on the social media platform being studied, or through other communication channels used by researchers. A researcher or research group may, for example, publish messages describing the project using hashtags relevant to the study. 

Even when individual notification is waived or information about the research is provided only at a general level, a privacy notice must still be prepared for the study. The privacy notice also serves as a document supporting accountability (the ). At the University of Helsinki, privacy notices are submitted to the Data Protection Officer for archiving (at the University of Helsinki, via ), and guidance and templates for preparing them are available through Flamma.

Social media data often constitute low-risk personal data but may also contain sensitive information, such as political opinions or information about religious beliefs, particularly when shared within closed groups. At the same time, social media data are often public (visible to anyone on the internet) or semi-public (visible to all users of a service), which may reduce their risk level. In theory, every social media user has accepted the platform's EULA and terms of service, which define the visibility and use of content. However, relatively few users are familiar with these documents in detail (see for example Obar & Oeldorf-Hirsch 2020). Consequently, it cannot be assumed that users understand their posts to be as public as, for example, letters to the editor in newspapers. This raises research ethical concerns and complicates the provision of appropriate participant information (see, for example, the  

Anonymisation of social media data, or of direct quotations extracted from social media data, can often be undermined through phrase searching. When publishing research findings, researchers should recognise that the original source and author of publicly available online content (for example, posts on X or Reddit) can frequently be identified using a simple search engine query. In such cases, the data cannot be considered truly anonymised within the meaning of the GDPR. Complete anonymisation would also require the removal of linked or referenced user accounts. In studies of highly specific or rare social media phenomena, the risk of identifying individuals is further increased. 

Researchers can strengthen participant anonymity, for example by translating social media content into another language (where the language of the data differs from the language of publication) and not providing the original-language version. Another possibility is to use paraphrases, meaning that the content is represented faithfully but not word-for-word (see, for example, Markham 2012). These approaches can enhance participant anonymity and support ethically responsible research practices.

The transparency of social media research is enhanced through careful documentation. APIs, software, and other tools used for social media data collection change continuously. For this reason, it may not be possible to reproduce a data collection process at a later time in exactly the same way as when the original collection took place. However, transparency and reproducibility can be improved by documenting key information about the collection process, including the APIs used, their versions, and any software and scripts employed. Reproducibility is also affected by the fact that social media content disappears when users delete posts or leave platforms, and users may edit content retrospectively. For this reason, it is important to document the timing of data collection as precisely as possible. 

An efficient way to collect social media data is to use established data collection tools. A variety of open-source tools are available online for collecting social media data for research purposes. Many are based on Python or R libraries and make use of platform APIs. Examples include open-source collection tools developed for specific platforms, such as , which has been widely used for collecting data from X. 

The use of these tools often requires access to the platform's API and at least basic technical expertise. Researchers are therefore advised to familiarise themselves carefully with the documentation of the tools they use and, where necessary, their source code (for example, the and ). This helps researchers understand what data the tool collects, how it processes the data, and what limitations are associated with the collection process. 

Developing custom collection tools provides greater control over the data collection process. Creating custom collectors requires substantial programming expertise and a significant investment of time. Widely used collection tools have often undergone extensive development and testing, reducing the likelihood of errors compared to tools developed by individual researchers. 

However, a key advantage of custom-built tools is that researchers have detailed knowledge of the technical implementation of data collection and can exercise precise control over how the data are created. Researchers may also share their collection tools, scripts, and documentation through services such as GitLab or GitHub (for example, the University of Helsinki GitLab service, ). This supports transparency and facilitates the evaluation and reuse of methods in other projects. 

The data collection process behind commercially acquired social media data is not always transparent. When obtaining data from a commercial provider, researchers may not know how the data were collected or what decisions were made in compiling the dataset. Particularly in the case of large datasets, it may be difficult to assess how well the data represent the phenomenon under study. 

The selection of data provided to researchers may be based on algorithms or other methods whose operation is not fully disclosed. In some cases, these may be protected as trade secrets. As a result, researchers may have only limited opportunities to evaluate how the data were generated and what biases may be present. This can create significant blind spots that researchers may be unable to identify or take into account when interpreting their findings (Joseph et al. 2014; Morstatter et al. 2013). 

Considerable time may be required to preprocess raw social media data. Raw social media data often contain large amounts of information that are either irrelevant or difficult to use for research purposes. Consequently, the data frequently need to be transformed before analysis. This may involve processing JSON files with scripts, selecting relevant fields, or restructuring the data into a format better suited to analysis. When planning preprocessing activities, it is important to consider the type of analysis that will be conducted. Decisions made during preprocessing directly affect what can later be studied and what findings can be obtained. Inadequate planning may result in work having to be repeated, leading to unnecessary expenditure of time and resources. The amount of work required for preprocessing can be difficult to estimate in advance and may in practice take weeks or even months, particularly in large projects or where research objectives are not yet clearly defined. 

Documenting data preprocessing supports transparency and reproducibility. Transforming raw data into a form suitable for research may reduce transparency and reproducibility if the processing steps are not documented carefully. Documentation should describe, for example, how the data were processed, what information was removed or modified, and which scripts, software, and tools were used. Careful documentation facilitates both the evaluation and replication of research. It also supports collaboration within the research team, as understanding the work of other researchers may otherwise be difficult and time-consuming. Documentation of the research process can be regarded as metadata describing the research. Such documentation can often be published even when the underlying data cannot be shared. 

In the context of social media data, it is often more meaningful to focus on documenting the research process than on describing the data themselves. Social media data are generated independently of the researcher and for purposes unrelated to research. As a result, researchers cannot comprehensively document all factors that influence the creation of the data. For example, a JSON dataset collected through the X API may contain a large number of fields and variables, many of which are rarely relevant for research purposes (such as the background colour of a user's profile page). Nor can researchers reasonably be held responsible for how a social media platform collects data or determines what information to provide through its interfaces. Researchers should, however, remain aware of how platform features and changes influence data collection and the content of datasets. Familiarity with critical platform studies can support this understanding. In social media research, the researcher's primary responsibility therefore concerns documenting the research, collection, and analysis processes rather than providing a comprehensive description of the data themselves. 

Documentation of research processes supports transparency and reproducibility. The processing and analysis of social media data often involve multiple stages that transform the original data. It is therefore important to document processing procedures, software, scripts and their versions, and key analytical decisions. Careful documentation facilitates evaluation, replication, and reuse of research. It also supports collaboration within research teams, as other researchers can understand previously completed work without needing to reconstruct the rationale for particular decisions afterwards. Documentation of the research process can be considered research metadata. It can often be made publicly available even when the data themselves cannot be shared. 

There is no single storage solution that suits every research project. When selecting a storage solution, researchers should consider factors such as: 

• How many people require access to the data? 
• How much data will be stored? 
• How sensitive are the data? 
• Is the research being conducted in collaboration with other organisations? 

Researchers are encouraged to familiarise themselves carefully with available storage options. The storage solutions provided by the University of Helsinki are summarised in a dedicated storage comparison . 

When selecting a storage solution, particular care should be taken with general cloud services and portable storage media (such as external hard drives and USB memory devices), especially where the data contain personal information or other protected information. 

The following recommendations are based on social media data that may contain personal data but do not involve highly sensitive or high-risk information. 

• If a researcher works independently at the University of Helsinki and accumulates less than 100 GB of data, the home directory () is a suitable option, as it provides an adequate level of information security together with fine-grained access control, versioning, and automatic backups.  
• If a researcher works independently or in a group consisting only of University of Helsinki researchers and the amount of data remains below 10 TB, the shared group directory () is a suitable option, as it provides an adequate level of information security together with fine-grained access control, versioning, and automatic backups.  
• If the research group includes collaborators from outside the University of Helsinki, suitable solutions include , , and, where appropriate, . It should be noted that not all of these services are suitable for storing personal data. The suitability of these services should be assessed on a case-by-case basis with regard to their intended use, data protection requirements, and terms of service. 
• In projects involving personal data, researchers must ensure that the chosen service is suitable for processing the data concerned. Where necessary, data may be protected, for example through encryption. Information security may also be enhanced by separating different components of the data, for example by storing pseudonymisation keys separately from the research data and documentation. 

The sharing, archiving, and disposal of research data must comply with agreements made during the research project, information provided to participants, and applicable legal and ethical obligations. Even where research data themselves cannot be shared, it is often possible to publish metadata, documentation, methodological descriptions, or software code used in the analysis to support transparency and reproducibility. 

The University of Helsinki Databank, Data archive, and Data catalogue services are available to researchers for the preservation, publication, and sharing of research data, datasets, and metadata after the active phase of research or once a research project has been completed (see the University of Helsinki , , and ). The suitability of these services depends on the nature of the data, access rights, and data protection requirements.   

The terms of service and agreements made with social media platforms may restrict data sharing and long-term preservation. In many cases, social media data collected for research cannot be shared or archived in their original form. Instead, researchers must comply with conditions established by the platform. Because these restrictions vary across platforms, they should be assessed during the planning stage of the research. 

On some platforms, data may be shared in the form of identifiers or other reference information rather than the original content itself. In such cases, future use of the data may require the content to be re-collected through the platform's APIs. However, platform practices, technical solutions, and service conditions change continuously, which may complicate both reproducibility and long-term preservation. 

Even when the data themselves cannot be shared, researchers can often publish the methods, scripts, and software code used in their research through services such as GitLab or GitHub. To support reuse, code should be adequately documented and commented. 

After the completion of a project, responsibility for curating social media data may remain with the researcher. Some platform terms of service or agreements require researchers to maintain the currency of their datasets after collection. This may involve removing posts from the dataset if the original author subsequently deletes them from the platform. A user's decision to delete content may be interpreted as indicating a changed expectation of privacy, highlighting the importance of research ethics and data protection considerations in the continued use of the data. 

In practice, curation requirements may create challenges for researchers. Compliance may require regular updating or checking of datasets through platform APIs, increasing workload and potentially generating additional costs. At the same time, the available dataset changes over time as content is removed and users leave platforms. This may reduce reproducibility, as it may no longer be possible to reconstruct the original dataset exactly. 

Curation requirements also affect the object of study itself. In practice, researchers often analyse the content that is available at the time of research or at later points of inspection, rather than all content ever published. For this reason, historical research based on social media data is inherently more limited than research in which the original data can be preserved unchanged. 

When planning long-term preservation, it is important to assess the suitability of archiving solutions carefully. In particular, where data contain personal information, researchers must ensure that the chosen service meets appropriate information security and data protection requirements. Where data must be stored in services or on media with limited protection, security can be improved through measures such as encryption and maintaining multiple backups. 

Deletion of data is also an option. Not all research data need to be retained after the completion of a project. Data may be destroyed, provided there are no research, contractual, or legal reasons requiring their continued retention. 

Data that are to be deleted must be disposed of securely. If data are to be deleted, they must be deleted in a manner that prevents the files from being recovered and reused. The organisation's information security guidelines should be followed with particular care when disposing data containing personal information or other types of sensitive data (see the guidelines by ). Physical storage media, such as external hard drives, USB drives, and optical discs, should also be disposed of securely. Researchers may seek assistance from their organisation's IT services or information security specialists where necessary (see ). Local IT support () may also be requested to carry out the deletion if the researcher is uncertain about the correct procedure or if the data are highly sensitive.  

Bruns, A. (2019). After the ‘APIcalypse’: social media platforms and their fight against critical scholarly research. Information Communication and Society, 22(11), 1544–1566. 

Joseph, K., Landwehr, P. M., & Carley, K. M. (2014). Two 1%s Don’t make a whole: Comparing simultaneous samples from Twitter’s Streaming API. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8393 LNCS, 75–83. 

Laaksonen, Salla-Maaria & Salonen, Margareta 2018. Kuka saa päättää, mitä dataa tutkijalla on käytössään? Ei ainakaan amerikkalainen suuryritys. Rajapinta.

Markham, A. (2012). Fabrication as Ethical Practice. Information, Communication & Society, 15(3), 334–353.

Morstatter, F., Ave, S. M., Ave, S. M., & Carley, K. M. (2013). Is the Sample Good Enough ? Comparing Data from Twitter ’ s Streaming API with Twitter ’ s Firehose.

Obar, J. A., & Oeldorf-Hirsch, A. (2020). The biggest lie on the Internet: ignoring the privacy policies and terms of service policies of social networking services. Information, Communication & Society, 23(1), 128–147.

Rogers, R. (2018). Social media research after the fake news debacle. Partecipazione e Conflitto: The Open Journal of Sociopolitical Studies, 11(2), 557–570. doi:10.1285/i20356609v11i2p557

Sandvig, C. (2017). Heading to the courthouse for Sandvig v. Sessions.

Venturini, T., Bounegru, L., Gray, J., & Rogers, R. (2018). A reality check(list) for digital methods. New Media and Society, 20(11), 4195–4217. 

.

.

.

The data management handbooks are designed as living resources that are continuously updated to meet the evolving practices of the field and the needs of researchers.

You can give feedback on the Social Media Data guide, such as additional information, comments, or corrections, via this .