M.Sc. Sezin Gizem Yaman defends her doctoral thesis Initiating the Transition towards Continuous Experimentation: Empirical Studies with Software Development Teams and Practitioners on Friday the 25th of October 2019 at 12 o'clock noon in the University of Helsinki Athena building, Room 302 (Siltavuorenpenger 3 A, 3rd floor). Her opponent is Professor Brian Fitzgerald (University of Limerick, Ireland) and custos Professor Tomi Männistö (University of Helsinki). The defence will be held in English.
Initiating the Transition towards Continuous Experimentation: Empirical Studies with Software Development Teams and Practitioners
Software experiments are presently often used by big technology pioneers, such as Microsoft, Facebook and Google, in order to learn about their users and to guide their research and development activities. Continuous experimentation (CE) is reported to be an integral part of software development in these organisations, however, how they transitioned to the approach is not publicly shared. Therefore, there is a lack of guidance for other organisations that are willing to adopt CE. In the current competitive markets, investing time and money in a new approach might be risky for these organisations, especially if they do not know how to initiate this transition process.
This dissertation focuses on how organisations can initiate the transition towards CE, i.e., an approach to enhance development decisions by running experiments in an iterative and sustainable fashion. The dissertation was designed to acquire descriptive and observational knowledge through empirical studies and was con- ducted in three main phases. First, we designed and ran multiple-case studies to investigate how CE can be introduced to existing software company development teams, who want to run their first systematic experiments. We extracted descriptive knowledge from the introduction process and composed lessons learned to act as guidelines. In the second phase, we conducted a survey study with practitioners from four Nordic software companies, in order to better understand their attitudes and perception towards experiment-driven development, user involvement and ethics. Examining the results at role-to-role levels gave us an understanding of commonalities and distinctions stemming from different job functions. Furthermore, we identified patterns from the data that describe what trends exist across the dataset with respect to experiment-driven software development. Finally, in the last phase of the study, we conducted a single-case study with a mobile gaming company to investigate how CE functions as an organisational mechanism throughout the development life-cycle.
The findings show that transitioning towards CE is a learning process that can be facilitated well by guidance, utilising existing resources and starting with small experiments with potentially enormous impact. Furthermore, by investigating the point of view of practitioners, we observed that software experiments represent different concepts, for instance, A/B tests and user interviews. We also observed that the role of the practitioner has a big impact not only on how experiments are understood, but also how individuals perceive the ethics involved in the experiments. For example, while managers are more cautious about company-customer relationships, UX designers were found to allow exceptions to user notification during experiments. In addition, we discovered that companies might understand and adopt experiment-driven development differently, for in- stance, influenced by their business contexts. Lastly, by examining a company’s CE practices, we found that experiments can take different forms given the development stage, and the organisational mechanism can be established to fit both the needs of the business domain and organisational goals. One of the biggest challenges of adopting CE, inaccessible real users, can be overcome with alter- native methods, such as proxy users, especially early in the development, when experiments are important in determining product value.
Highly competitive markets can put pressure on organisations to avoid risks and costs when adopting a new approach. In this dissertation, we learned that by and large, software organisations and development teams can initiate their transition towards CE in an efficient and economical way. Furthermore, we conclude that the transition is a learning process that improves with practice and has to adapt to the organisational goals and contexts. The influence of human factors, such as the finding that individual perception of experiments and ethics is correlated with job functions indicates that CE is a multi-disciplinary research field, where individuals should be studied as well as experimentation processes. Software engineering research needs further studies to validate the findings in different contexts.
Availability of the dissertation
An electronic version of the doctoral dissertation is available on the e-thesis site of the University of Helsinki at http://urn.fi/URN:ISBN:978-951-51-5543-6.
Printed copies will be available on request from Sezin Gizem Yaman: sezin.yaman@helsinki.fi.