LN BI P 28 3 18th International Conference, XP 2017 Cologne, Germany, May 22–26, 2017 Proceedings Agile Processes in Software Engineering and Extreme Programming Hubert Baumeister Horst Lichter Matthias Riebisch (Eds.) Lecture Notes in Business Information Processing 283 Series Editors Wil M.P. van der Aalst Eindhoven Technical University, Eindhoven, The Netherlands John Mylopoulos University of Trento, Trento, Italy Michael Rosemann Queensland University of Technology, Brisbane, QLD, Australia Michael J. Shaw University of Illinois, Urbana-Champaign, IL, USA Clemens Szyperski Microsoft Research, Redmond, WA, USA More information about this series at http://www.springer.com/series/7911 http://www.springer.com/series/7911 Hubert Baumeister • Horst Lichter Matthias Riebisch (Eds.) Agile Processes in Software Engineering and Extreme Programming 18th International Conference, XP 2017 Cologne, Germany, May 22–26, 2017 Proceedings Editors Hubert Baumeister Technical University of Denmark Kongens Lyngby Denmark Horst Lichter RWTH Aachen University Aachen Germany Matthias Riebisch University of Hamburg Hamburg Germany ISSN 1865-1348 ISSN 1865-1356 (electronic) Lecture Notes in Business Information Processing ISBN 978-3-319-57632-9 ISBN 978-3-319-57633-6 (eBook) DOI 10.1007/978-3-319-57633-6 Library of Congress Control Number: 2017937714 © The Editor(s) (if applicable) and The Author(s) 2017. This book is an open access publication. Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Printed on acid-free paper This Springer imprint is published by Springer Nature The registered company is Springer International Publishing AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland http://creativecommons.org/licenses/by/4.0/ Preface The 18th XP conference was held 2017 in the wonderful city of Cologne, Germany. In the spirit of past XP conferences, XP 2017 was a place where researchers and practitioners met to exchange new ideas and present their work. These proceedings contain the full research papers, short research papers, and doctoral symposium papers presented at the conference. In all, 46 research papers were submitted (39 full and seven short papers). All submitted papers went through a thorough review process, with each paper receiving at least three reviews. Finally, the Program Committee accepted 14 papers as full research papers (an acceptance rate of 35%). Moreover, six papers — submitted as short or full research papers — were accepted as short research papers. The selected papers cover a wide range of agile techniques and approaches. Many of them present results of empirical studies aiming to systematically evaluate successful agile practices, others are technology studies that are relevant to both researchers and practitioners. In the tradition of former XP conferences, the XP 2017 conference program offered many different session topics. Besides the scientific program, i.e., the research track, doctoral symposium, and scientific workshops, the conference featured an industry and practice track, experience reports, and Open Space sessions. Materials from all of these sessions are available on the conference website at www.xp2017.org. Moreover, three keynotes were given by highly renowned speakers. Andrea Goulet from Corgibytes presented a talk on “Makers and Menders: Putting the Right Devel- opers on the Right Projects” focusing on a group of developers called “menders” – people who love taking an existing project and making it better over time. In his keynote “End-to-End Agility at GitHub” Alain Hélaïli talked about the organization and the efficient workflows at GitHub. Finally, Claes Wohlin from Blekinge Institute of Technology answered the question “Evidence-Driven Change in Software Develop- ment: Is It Feasible?” Many people contributed to the success of the XP 2017 conference. We would like to thank everyone, especially the authors and presenters of all papers, the Program Committee members, the volunteers, and sponsors. Furthermore, we want to express our gratitude to the XP 2017 organizers; they did a great job. March 2017 Hubert Baumeister Horst Lichter Matthias Riebisch http://www.xp2017.org Organization Organizing Committee General Chair Jutta Eckstein IT communication, Germany Conference Chairs Marc Clemens codecentric AG, Germany Nils Wloka codecentric AG, Germany Academic Program Committee Academic Program Chairs Hubert Baumeister Technical University of Denmark Horst Lichter RWTH Aachen University, Germany Matthias Riebisch University of Hamburg, Germany Scientific Workshops Roberto Tonelli University of Cagliari, Italy Poster Chair Ademar Aguiar University of Porto, Portugal PhD Symposium Chair Stefan Wagner University of Stuttgart, Germany Industrial Program Committee Tutorials/Workshops Nancy Van Schooenderwoert Lean-Agile Partners, USA Working Software Aslak Hellesøy Cucumber, UK Individuals and Interaction Diana Larsen FutureWorks Consulting, USA Customer Collaboration Ken Power Cisco Systems, Ireland Responding to Change Jan Coupette codecentric AG, Germany Experience Reports Rebecca Wirfs-Brock Wirfs-Brock Associates, USA Avraham Poupko Cisco Systems, Israel Open Space Alexander Kylburg Paragraph Eins, Germany Program Committee (Research Papers) Ademar Aguiar University of Porto, Portugal Mikio Aoyama Nanzan University, Japan Leonor Barroca The Open University, UK Hubert Baumeister Technical University of Denmark, Denmark Jan Bosch Chalmers University of Technology, Sweden Steve Counsell Brunel University, UK Torgeir Dingsøyr SINTEF, Norway Christof Ebert Vector Consulting Services, Germany Hakan Erdogmus Carnegie Mellon University, USA Juan Garbajosa Technical University of Madrid, Spain Alfredo Goldman University of São Paulo, Brazil Des Greer Queen’s University Belfast, UK Peggy Gregory University of Central Lancashire, UK Rashina Hoda The University of Auckland, New Zealand Helena Holmström Olsson Malmö University, Sweden Casper Lassenius MIT, USA Horst Lichter RWTH Aachen University, Germany Lech Madeyski Wroclaw University of Science and Technology, Poland Michele Marchesi University of Cagliari, Italy Sabrina Marczak Pontifícia Universidade Católica do Rio Grande do Sul, Brazil Tommi Mikkonen University of Helsinki, Finland Alok Mishra Atilim University, Turkey Nils Brede Moe SINTEF, Norway Juergen Muench Reutlingen University and University of Helsinki, Germany/Finland Sridhar Nerur University of Texas at Arlington, USA Maria Paasivaara Helsinki University of Technology, Finland Kai Petersen Blekinge Institute of Technology/Ericsson AB, Sweden VIII Organization Matthias Riebisch University of Hamburg, Germany Pilar Rodríquez University of Oulu, Finland Knut H. Rolland Westerdals Oslo School of Arts, Communication and Technology, Norway Bernhard Rumpe RWTH Aachen University, Germany Kurt Schneider Leibniz Universität Hannover, Germany Helen Sharp The Open University, UK Darja Smite Blekinge Institute of Technology, Sweden Roberto Tonelli University of Cagliari, Italy Rini Van Solingen Delft University of Technology, The Netherlands Stefan Wagner University of Stuttgart, Germany Xiaofeng Wang Free University of Bozen-Bolzano, Italy Hironori Washizaki Waseda University, Japan Agustin Yague Universidad Politecnica de Madrid, Spain Reviewers (Industry and Practice) Giovanni Asproni Asprotunity, UK Emily Bache Bache Consulting, Sweden Filipe Correia Uphold, Portugal Aino Corry Metadeveloper, Denmark Lisa Crispin Pivotal, USA Jutta Eckstein IT communication, Germany Sallyann Freudenberg Sallyann Freudenberg Consulting, UK Steve Holyer Steve Holyer and Associates, Switzerland Lise Hvatum Schlumberger, USA Allan Rennebo Jepsen Core Agile, Denmark Jason Kerney Hunter Industries, USA David Kramer Agile New England, USA Casper Lassenius Aalto University, Finland Olaf Lewitz trustartist.com, Germany Ralph Miarka sinnvollFÜHREN, Austria Maria Paasivaara Alto University, Finland Dana Pylayeva Hudson’s Bay Company, USA Seb Rose Cucumber, UK Johanna Rothman Rothman Consulting Group, USA Aki Salmi Ambientia, Finland Andreas Schliep Das ScrumTeam, Switzerland Irina Tsyganok Yoox Net-A-Porter Group Nils Wloka codecentric AG, Germany Joseph Yoder The Refactory, USA Joe Wright Arnold Clark Automobiles, UK Organization IX Additional Reviewers Adam, Kai Bjørnson, Finn Olav Britto, Ricardo Butting, Arvid Da Silva, Tiago Silva Díaz, Jessica Edison, Henry Fernández-Sánchez, Carlos Fögen, Konrad Kautz, Oliver Kusmenko, Evgeny Raco, Deni Santana, Célio Santos, Viviane Stray, Viktoria Vestues, Kathrine Wang, Yang Sponsors “Cologne Cathedral” Sponsor REWE digital “Albertus Magnus” Sponsor Accenture Interactive “River Rhine” Sponsors DATEV EPLAN Software & Service OPITZ CONSULTING XebiaLabs “Kölsch” Sponsor Hänneschen and Bärbelchen Organizer codecentric X Organization Contents Improving Agile Processes Reflection in Agile Retrospectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Yanti Andriyani, Rashina Hoda, and Robert Amor What Influences the Speed of Prototyping? An Empirical Investigation of Twenty Software Startups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Anh Nguyen-Duc, Xiaofeng Wang, and Pekka Abrahamsson Key Challenges in Agile Requirements Engineering . . . . . . . . . . . . . . . . . . 37 Eva-Maria Schön, Dominique Winter, María José Escalona, and Jörg Thomaschewski Eeny, Meeny, Miny, Mo...: A Multiple Case Study on Selecting a Technique for User-Interaction Data Collecting . . . . . . . . . . . . . . . . . . . . 52 Sampo Suonsyrjä Comparing Requirements Decomposition Within the Scrum, Scrum with Kanban, XP, and Banana Development Processes . . . . . . . . . . . . . . . . 68 Davide Taibi, Valentina Lenarduzzi, Andrea Janes, Kari Liukkunen, and Muhammad Ovais Ahmad Effects of Technical Debt Awareness: A Classroom Study . . . . . . . . . . . . . . 84 Graziela Simone Tonin, Alfredo Goldman, Carolyn Seaman, and Diogo Pina Agile in Organizations Don’t Forget to Breathe: A Controlled Trial of Mindfulness Practices in Agile Project Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Peter den Heijer, Wibo Koole, and Christoph J. Stettina Enhancing Agile Team Collaboration Through the Use of Large Digital Multi-touch Cardwalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Martin Kropp, Craig Anslow, Magdalena Mateescu, Roger Burkhard, Dario Vischi, and Carmen Zahn Knowledge Sharing in a Large Agile Organisation: A Survey Study . . . . . . . 135 Kati Kuusinen, Peggy Gregory, Helen Sharp, Leonor Barroca, Katie Taylor, and Laurence Wood http://dx.doi.org/10.1007/978-3-319-57633-6_1 http://dx.doi.org/10.1007/978-3-319-57633-6_2 http://dx.doi.org/10.1007/978-3-319-57633-6_2 http://dx.doi.org/10.1007/978-3-319-57633-6_3 http://dx.doi.org/10.1007/978-3-319-57633-6_4 http://dx.doi.org/10.1007/978-3-319-57633-6_4 http://dx.doi.org/10.1007/978-3-319-57633-6_5 http://dx.doi.org/10.1007/978-3-319-57633-6_5 http://dx.doi.org/10.1007/978-3-319-57633-6_6 http://dx.doi.org/10.1007/978-3-319-57633-6_7 http://dx.doi.org/10.1007/978-3-319-57633-6_7 http://dx.doi.org/10.1007/978-3-319-57633-6_8 http://dx.doi.org/10.1007/978-3-319-57633-6_8 http://dx.doi.org/10.1007/978-3-319-57633-6_9 Teaching Agile Methods to Software Engineering Professionals: 10 Years, 1000 Release Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Angela Martin, Craig Anslow, and David Johnson Are Software Startups Applying Agile Practices? The State of the Practice from a Large Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Jevgenija Pantiuchina, Marco Mondini, Dron Khanna, Xiaofeng Wang, and Pekka Abrahamsson Adopting Test Automation on Agile Development Projects: A Grounded Theory Study of Indian Software Organizations . . . . . . . . . . . . 184 Sulabh Tyagi, Ritu Sibal, and Bharti Suri Safety Critical Software How is Security Testing Done in Agile Teams? A Cross-Case Analysis of Four Software Teams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Daniela Soares Cruzes, Michael Felderer, Tosin Daniel Oyetoyan, Matthias Gander, and Irdin Pekaric An Assessment of Avionics Software Development Practice: Justifications for an Agile Development Process . . . . . . . . . . . . . . . . . . . . . 217 Geir K. Hanssen, Gosse Wedzinga, and Martijn Stuip Short Research Papers Inoculating an Agile Company with User-Centred Design: An Empirical Study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Silvia Bordin and Antonella De Angeli On the Usage and Benefits of Agile Methods & Practices: A Case Study at Bosch Chassis Systems Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Philipp Diebold and Udo Mayer Checklists to Support Test Charter Design in Exploratory Testing . . . . . . . . . 251 Ahmad Nauman Ghazi, Ratna Pranathi Garigapati, and Kai Petersen Discovering Software Process Deviations Using Visualizations . . . . . . . . . . . 259 Anna-Liisa Mattila, Kari Systä, Outi Sievi-Korte, Marko Leppänen, and Tommi Mikkonen Exploring Workflow Mechanisms and Task Allocation Strategies in Agile Software Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Zainab Masood, Rashina Hoda, and Kelly Blincoe XII Contents http://dx.doi.org/10.1007/978-3-319-57633-6_10 http://dx.doi.org/10.1007/978-3-319-57633-6_10 http://dx.doi.org/10.1007/978-3-319-57633-6_11 http://dx.doi.org/10.1007/978-3-319-57633-6_11 http://dx.doi.org/10.1007/978-3-319-57633-6_12 http://dx.doi.org/10.1007/978-3-319-57633-6_12 http://dx.doi.org/10.1007/978-3-319-57633-6_13 http://dx.doi.org/10.1007/978-3-319-57633-6_13 http://dx.doi.org/10.1007/978-3-319-57633-6_14 http://dx.doi.org/10.1007/978-3-319-57633-6_14 http://dx.doi.org/10.1007/978-3-319-57633-6_15 http://dx.doi.org/10.1007/978-3-319-57633-6_15 http://dx.doi.org/10.1007/978-3-319-57633-6_16 http://dx.doi.org/10.1007/978-3-319-57633-6_16 http://dx.doi.org/10.1007/978-3-319-57633-6_17 http://dx.doi.org/10.1007/978-3-319-57633-6_18 http://dx.doi.org/10.1007/978-3-319-57633-6_19 http://dx.doi.org/10.1007/978-3-319-57633-6_19 Are Daily Stand-up Meetings Valuable? A Survey of Developers in Software Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 Viktoria Stray, Nils Brede Moe, and Gunnar R. Bergersen Doctoral Symposium Papers Knowledge Management and Reflective Practice in Daily Stand-Up and Retrospective Meetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Yanti Andriyani Self-Assignment: Task Allocation Practice in Agile Software Development. . . . 292 Zainab Masood Software Development Practices Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . 298 Herez Moise Kattan and Alfredo Goldman Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Contents XIII http://dx.doi.org/10.1007/978-3-319-57633-6_20 http://dx.doi.org/10.1007/978-3-319-57633-6_20 http://dx.doi.org/10.1007/978-3-319-57633-6_21 http://dx.doi.org/10.1007/978-3-319-57633-6_21 http://dx.doi.org/10.1007/978-3-319-57633-6_22 http://dx.doi.org/10.1007/978-3-319-57633-6_23 Improving Agile Processes Reflection in Agile Retrospectives Yanti Andriyani1(✉), Rashina Hoda1, and Robert Amor2 1 SEPTA Research, Department of Electrical and Computer Engineering, The University of Auckland, Building 903, 386 Khyber Pass, New Market, 1023 Auckland, New Zealand yand610@aucklanduni.ac.nz, r.hoda@auckland.ac.nz 2 Department of Computer Science, The University of Auckland, Auckland, New Zealand trebor@cs.auckland.ac.nz Abstract. A retrospective is a standard agile meeting practice designed for agile software teams to reflect and tune their process. Despite its integral importance, we know little about what aspects are focused upon during retrospectives and how reflection occurs in this practice. We conducted Case Study research involving data collected from interviews of sixteen software practitioners from four agile teams and observations of their retrospective meetings. We found that the impor‐ tant aspects focused on during the retrospective meeting include identifying and discussing obstacles, discussing feelings, analyzing previous action points, iden‐ tifying background reasons, identifying future action points and generating a plan. Reflection occurs when the agile teams embody these aspects within three levels of reflection: reporting and responding, relating and reasoning, and recon‐ structing. Critically, we show that agile teams may not achieve all levels of reflection simply by performing retrospective meetings. One of the key contri‐ butions of our work is to present a reflection framework for agile retrospective meetings that explains and embeds three levels of reflection within the five steps of a standard agile retrospective. Agile teams can use this framework to achieve better focus and higher levels of reflection in their retrospective meetings. Keywords: Agile retrospective meeting · Reflection · Levels of reflection · Teams · Agile software development · Reflective practice 1 Introduction Retrospective meetings embody the ‘inspect and adapt’ principle of Agile Software Development (ASD) [1, 2]. They are designed to enable agile teams to frequently eval‐ uate and find ways to adjust their process [3]. There are several purposes for retrospective meetings, such as to evaluate the previous work cycle or sprint; to determine the aspects that need to be focused on as areas of improvement; and to develop a team action plan [4]. The purpose and the techniques of the retrospective meeting have been stated and described clearly as a guide for agile teams [2, 5, 6]. Much of the existing research focuses on the techniques of performing retrospective meetings and provides lesser detail about the reflection process involved [5–9]. The Reflective Agile Learning Model (REALM) [7] classified reflection in ASD practices © The Author(s) 2017 H. Baumeister et al. (Eds.): XP 2017, LNBIP 283, pp. 3–19, 2017. DOI: 10.1007/978-3-319-57633-6_1 into reflection-in-action or reflection that occurs during a practice, and reflection-on- action or reflection that occurs post a practice based on definitions of the same by Argyris and Schön [10]. A retrospective meeting was seen to embody reflection-on-action where the agile teams reflect post finishing their sprint [7]. However, what is focused on during retrospectives and how reflection occurs in this practice is not well understood. To address this gap, we conducted Case Study research by observing four agile teams and interviewing 16 of their members guided by the following research questions: RQ 1: What aspects are focused on during the retrospective meeting? RQ 2: How does reflection occur in the retrospective meeting? 2 Related Work 2.1 Agile Retrospective Meeting There is a standard format commonly used to conduct an agile retrospective meeting which involves setting the stage, gathering data, generating insight, deciding what to do and closing the retrospective meeting [2]. Setting the stage involves welcoming and explaining the aim of the retrospective meeting. Gathering data involves agile teams sharing their review and feedback, reporting on what happened during the previous sprint and briefly discussing with other team members. In generating insight, agile teams participate in a further discussion and making agreements about what issues to focus on, and then on how to solve those issues and what areas that need to improve in the deciding what to do stage. Closing the retrospective involves summarizing the retrospective meeting and appreciating all team members’ efforts. There are several recommendations for embedding reflective practice within standard agile practices as it is related to team performance improvement [7–9]. Cockburn [8] intro‐ duced a reflection workshop which involves collecting issues and generating tasks and decisions. This workshop is performed regularly during or after the post-iteration work‐ shop. Babb et al. [7] investigated reflection in agile practices based on Argyris and Schön’s [10] classification and introduced the Reflective Agile Learning Model (REALM). REALM describes how some agile practices embody reflection-in-action and reflection- on-action. Retrospective meetings were seen to embody reflection-on-action where the agile teams reflect post finishing their sprint [7]. Most of the existing research focuses on the techniques of performing a retrospective or identifying a broad classification of the type of reflection that occurs, e.g. reflection- on-action [7]. What actual topics or aspects are discussed during a retrospective and how reflection occurs, however, is not well understood. We build upon these works by inves‐ tigating the retrospective meeting in depth. 2.2 Reflective Practice Reflective practice according to Osterman and Kottkamp [11], is defined as “a means by which practitioners can develop a greater level of self-awareness about the nature 4 Y. Andriyani et al. and impact of their performance, an awareness that creates opportunities for profes‐ sional growth and development”. Bain et al. [12] classified reflection into five levels: reporting, responding, relating, reasoning and reconstructing. Level 1 and 2 are reporting and responding and enable learners to share brief descriptions of their experience, their feelings about events, facts or problems that they encountered. Level 3 is relating and involves connecting experi‐ ence with personal meaning. Understanding at this level occurs when learners try to highlight good points (e.g. their ability, successful work) and negative points (e.g. mistakes, failure) to learn and identify areas of improvement. Level 4 is reasoning where learners explore the information shared as well as background knowledge related to the occurrences. Level 5 is reconstructing which signifies a high level of learning where learners generate the general framework of thinking, which is specified in a plan or action for responding to similar obstacles in the future. Our study refers to levels of reflection proposed by Bain et al. [12] and adjusts the levels into three main levels, i.e. reporting and responding, relating and reasoning and reconstructing, based on our observations of the agile retrospectives in practice. Reporting and responding are grouped together as the first level as these levels closely related to reviews sharing and discussions at the beginning of the retrospective meeting. Relating and responding are grouped as the second level as agile teams participate in a further discussion after they reported and responded to the reviews. The third level, the reconstructing level is embodied when agile teams discuss to formulate a plan as an improvement for the next sprint. 3 Research Method The aim of this study is to investigate how reflection occurs in retrospective meetings. Understanding this is particularly important as agile teams are reported to focus more on their technical progress and tend to pay less attention to how reflection is performed thereby compromising their potential for improvement [7, 13]. This research is conducted by implementing the Case Study research method [14]. First, existing studies related to reflection in retrospective meetings were reviewed, as summarized in Sect. 2. The research gaps identified provided guidance on formulating the interview questions. To gain rich data from interviews, we developed semi-struc‐ tured questions consisting of main questions and follow-up questions. The data collec‐ tion method is described in Sect. 3.1 and participant demographics summarized in Table 1. All interviews and observation data were collected by the first author in person. The raw data and emerging findings from the analysis were discussed in detail with the supervisory team (co-authors) who provided feedback and guidance. 3.1 Data Collection Participants. We wanted to include software practitioners with a minimum of 2 years’ industrial agile experience to participate in our research. During one of the Auckland Agile meetups, we received interest in participation from an agile team lead working at Reflection in Agile Retrospectives 5 the largest online auction company in New Zealand, Trade Me. Trade Me had been practicing agile software development for over three years and provided access to four teams. Its headquarters are located in Wellington and the regional offices are in Auckland and Christchurch. For confidentiality purposes, the teams are named Jupiter, Saturn, Uranus, and Neptune. The team names and team members’ details can be seen in Table 1. Each team consisted of between 3 and 10 members. All members were invited and those willing were interviewed. All teams held retrospective meetings, which lasted for between 15 and 65 min. Sixteen individual practitioners from the four teams participated in the interviews and the observations. All team members had a dedicated role in their team Table 1. Team and team members demographics (RMD: Retrospective meeting duration in minutes; P#: Individual participant number; FAP: first agile project) Team Name Interview ed/total members Agile method RMD and the frequencies P# Role Agile experience (Year) Agile projects (Total) Jupiter 5 out of 10 Scrum 65 min (every two weeks) P1 UI Designer 1 6–8 P2 Developer 0.5 1 P3 Developer 7+ 6–7 P4 Business analyst 7+ 20+ P9 Tester 3+ 10+ Saturn 6 out of 10 Scrum 55 min (every two weeks) P4 Business analyst 7+ 20+ P5 Developer 3 10+ P6 Designer 1 month FP P7 Designer 0.5 FAP P8 Tester 3+ 6 P9 Tester 3+ 10+ Uranus 2 out of 3 Kanban 45 min (every two weeks) P10 Tester, Developer 1 2 P11 Scrum Master, Business Analyst, Product Owner 6 6 Neptune 4 out of 6 Scrum 15 min (when needed) P12 Tester 2 1 P13 Developer 1.5 FAP P14 Developer 1 FAP P15 Tester <1 year FAP Working across all four teams P16 Test chapter lead 7 10+ 6 Y. Andriyani et al. and there were three participants that committed across different teams: P4 was not only fully committed as a Business Analyst in Team Jupiter but also supported Team Saturn. Similarly, P9 was a tester in Team Saturn and a half tester in Team Jupiter. P16 worked as a test lead across all four teams. Interviews and Observations. Face-to-face individual and one group interview (of six team members) were conducted to gain comprehensive explanations, which would help derive the real concerns from both individual and team perspectives. We conducted one- on-one interviews with all participants (P1–P16), where the duration of individual inter‐ views varied from 35 to 50 min. We asked some semi-structured questions about their experiences and perspectives related to reflection in a retrospective meeting. Some sample questions include: “Based on the three main points discussed in a retrospec‐ tive (i.e. what went well, what went wrong, what can you improve), which one(s) do you think are most helpful for your team’s reflection?”, “How does your team use those points to find solutions and ways to improve? Could you give some real examples?” and “What is the outcome of your retrospective meeting? Does your team/scrum master preserve points from the meeting?”. A sample question of the group interview includes: “I noticed that your team exhibited some different ways of sharing knowledge, (e.g. post-it notes, verbal communication, drawing). Did it help your team to perform reflection? How?” The group interview was conducted immediately after the retrospective meeting of Team Jupiter with six of its team members. Given the variable meeting times, work commitments and deadlines for different teams, it was not possible to gain further team availability for group interviews with the remaining three teams over and above the individual interviews and team observations. Observations were conducted during the retrospective meetings of all the teams and of their general workplace. The observations aimed to capture the details of the retro‐ spective meeting (i.e. time spent, attendees, and discussion involved) and to help validate the findings from the interviews. Photographs were taken during the observations in order to document the actual situations in the meetings and the report presented by the agile team members. Notes were taken to highlight the important aspects being shared. The information collected (e.g. photographs and notes) from the observations were used to support individual interviews by including the photographs and describing the activ‐ ities in the retrospective meetings as observed first hand. The duration of each obser‐ vation depended on how long the team conducted the retrospective meeting. Three out of four teams conducted the meeting for around 40–60 min each and one team, Neptune, had a shorter 15 min’ retrospective. Observational data (e.g. photographs and notes) were found to support the findings from the interview data analysis thereby strengthening them. 3.2 Data Analysis This research involved sixteen individual interviews, one group interview (of six team members), and notes taken from retrospective meeting observations which were analyzed by conducting a thematic analysis. Thematic analysis is a method that aims to Reflection in Agile Retrospectives 7 recognize, analyze, evaluate and report patterns in data [15], which enables researchers to search across a data set of interviews. Braun and Clarke [15] classify the analysis into six phases: transcribing data, generating initial codes, searching for themes, reviewing themes, defining and naming themes and making a report. Sixteen interviews were transcribed and imported into NVivo software to facilitate coding and thematic analysis. Generating initial codes involved code identification by analyzing interesting features of a sentence, which were highlighted and added as a node in NVivo representing a new code, such as identifying and discussing obstacles and discussing feelings. Searching for themes involved comparing data with different codes to see whether they have similar meanings or aspects. Parent themes were classified based on five (grouped into three) levels of reflection, where each code was classified based on the definition of each level. Fig. 1. Levels of reflection in retrospective meetings: a thematic map (using levels of reflection from Bain et al. [12]) Reviewing themes involved generating a thematic model to define the links and the relationships between the themes (see Fig. 1). Defining and naming themes involved the generation of several themes that emerged from the analysis, representing the aspects discussed during retrospective meetings, which was formulated and explained in this paper. 4 Findings Following the thematic data analysis process, we identified seven themes that represent important topics or aspects discussed in the retrospective meeting, which were then mapped to the five (grouped into three) levels of reflection [12] (see Sect. 2.2). 8 Y. Andriyani et al. Table 2 summarizes these themes along with their mapping to the reflection levels. These themes and levels are described below along with pertinent quotes and photo‐ graphs from observations. The figures below (see Figs. 2 and 3) were captured during the observation and show a glimpse of Team Jupiter and Team Saturn’s retrospective meeting. Table 2. Themes representing topics discussed during retrospectives, their description, examples, and mapping to levels of reflection based on [12]. Levels of reflection Themes/topics discussed Description of themes Examples Reporting and Responding Identifying and discussing obstacles Problems, issues and concern causing blockages Unfinished tasks and dependencies (e.g. expertise, activity, resource or entity and technical.) Discussing feelings The Subjective response that reflects the situation, fact or events from the previous sprint Negative and positive feelings Relating and Reasoning Analyzing previous action points Evaluate the process improvement based on previous action points Some improvement achieved or persisting obstacles Identifying background reasons Analyzing some causes and aspects related to issues on team improvement Testing environment issue related to external person in different location, who is difficult to contact Identifying future action points Evaluating what areas need to be focused on more to be defined as future action points Evaluate successful stories and failures Reconstructing Generating a plan Define some action points for the next plan Action points Fig. 2. Team Jupiter’s retrospective Fig. 3. Team Saturn’s retrospective Reflection in Agile Retrospectives 9 4.1 Reporting and Responding Reporting and responding can be realized when an agile team shares some aspects (e.g. identifying and discussing obstacles and discussing feelings) while providing reviews and feedback of the previous sprint. Each team had different techniques of performing their reviews. All teams were seen to engage in the reporting level of reflection by actively iden‐ tified and discussed obstacles and feelings. Similarly, all four teams were seen to be actively involved in responding to their retrospective meeting discussions by providing brief comments on the obstacles and feelings being shared. Teams were seen to report on obstacles such as dependencies and unfinished tasks and respond with negative and positive feelings based on the previous sprint, described below. Identifying and Discussing Obstacles. Obstacles reported in the retrospective meet‐ ings related to the aspects that hindered the team from making progress. During the retrospective meeting, agile teams gathered all the problems that occurred in the previous sprint, which would be useful for the teams to highlight areas of improvement. There were two specific obstacles reported: dependencies and unfinished tasks. Dependencies. Most of the participant (11 out of 16) mentioned dependencies as the specific type of obstacle most commonly reported in the retrospective meeting. “If it’s delayed at the first point, if something is wrong at the first point the next person feels it. So, if one brings it up [in the retrospective] and if it’s a true concern you will have support because it does affect people.” P16 – Test Chapter Lead (Across All Teams) By sharing problems about dependencies team members became aware of the other team members’ tasks and how they related to their own tasks. By being aware of this issue the team could think about ways to solve the dependency problems. Unfinished tasks. Unfinished tasks were mentioned by three participants as an obstacle reported in retrospective meetings. An unfinished task was a problem where team members could not accomplish the tasks they had planned or considered the team to be making slow progress. “We were not achieving that daily goal and it is a kind of demotivating… let’s say you plan 10 stories for the sprint and you achieve just two or three. The rest we couldn’t complete for what‐ ever reason. So, we say that is one thing which didn’t go well.” P12 – Tester, Team Neptune Surfacing this obstacle was helpful for teams to understand how much more effort was required to finish the tasks, what tasks were challenging and why the tasks were difficult to finish. For example, when Team Neptune faced a problem with a requirement that delayed finishing tasks, they asked for clarifications from the product manager. It was evident that dependencies led to unfinished tasks in some cases. Discussing Feelings. Besides obstacles, agile teams also shared their feelings which were visualized in several forms, e.g. as drawings or journey lines. The feelings shared by team members represented the sense of facts and occurrences from the previous sprint, such as when they were feeling down or happy. 10 Y. Andriyani et al. There was an example of positive feelings shared, which had a positive impact on the team’s productivity, where their work can be distributed well. Team Neptune recruited an additional tester after they had a problem with tester resource. They felt happy because their team was complete and balanced between developers and testers. “We do put down smiley. When we got a new tester on board, a new person we had a happy smiley saying that our squad is complete.” P12 – Tester, Team Neptune These obstacles and feelings identified and discussed during the retrospective meeting were supported by our observations of the retrospective meetings of Teams Jupiter, Saturn, and Uranus. It was observed that Team Jupiter reported their review by defining some words on sticky notes (see Fig. 4(a)). (a) Team Jupiter (b) Team Saturn (c) Team Uranus Fig. 4. (a) Words to describe obstacles and feelings in the Retrospective meeting; (b) and (c) Journey lines visualizing emotions during a sprint in Retrospective meetings For example, ‘muddy’ was used to describe a difficult situation where team members had difficulty in understanding the detailed description of specific user stories in the project. Upon asking a team member about what was the meaning of ‘muddy’, a partic‐ ipant explained: “So, I think, he and I came up with the term of ‘muddy’; from observation - they were really struggling to get the right data and really had to analyse the data for this project. I observed that and for me, I would pick out a description which would explain what I’ve observed; as a general team.”, P1 – User Interface Designer, Team Jupiter. 4.2 Relating and Reasoning Relating and reasoning can be seen when agile teams compile the obstacles and the feelings shared (from the previous reporting and responding level) and investigate the relationship between those aspects. These levels consisted of activities such as analyzing previous action points, identifying background reasons, identifying future action points. The explanations below present the results from the individual interviews, which supported by group interview and observations. Analyzing Previous Action Points. An ‘action point’ refers to a specific item selected by the team to focus on for improvement. In analyzing previous action points, agile Reflection in Agile Retrospectives 11 teams referred to the action points agreed upon by the team in the previous retrospective and evaluated the actual effort made by the team on that specific point. “..that’s how you define if you made any changes, we measure yourself based on your action points and that you’ve actually made changes for. You could make 200 action points of your 20 weeks, but not a single one of those was followed up on, you really haven’t done anything.” P4 – Business Analyst, Team Jupiter and Saturn From the example above, it was seen that agile teams reflected on the previous action points by measuring the outcomes achieved by the teams (i.e. good or slow progress). This statement was further supported by the observations where during the retrospective meetings, agile teams shared the process improvement or the failures of the previous sprint. Identifying Background Reasons. The background reasons of the existing issues were identified when teams were not actively progressing, they would explore the reasons why and what blockers were related to this problem. By identifying the background reasons, teams would understand what aspects needed to be improved. This point is supported by Team Jupiter’s group interview, which a team member tried to identify the reason of the major problem during the retrospective meeting. “I think we addressed like the major issues are causing the squad stuck at the moment and things like test environment and [..] dealing with an external dependency like platform team in [city name]” P4 – Business Analyst, Team Jupiter and Saturn During the retrospective meeting observation of Team Saturn, it was seen that there was a cause analysis discussion. For example, when team members shared their sad feelings experienced during the first week sprint, team members shared the reasons, such as unclear user stories or the user story was considered as a big task. The scrum master guided the team to identify the causes by asking why they used the sad feeling notation for the first week. Several reasons were shared, such as too many tasks, the previous estimation and the actual effort were different, the unclear scope of work restricted their progress. Discussing those reasons led to the point where the team realized the main background reason was about inaccurate estimation, i.e. the team had created high achievement expectations for the big tasks without considering the actual effort required. Identifying Future Action Points. Identifying future action points happened when the teams analyzed previous action points and identified the background issues, which followed by identifying areas of improvement and asking ideas and agreement from the teams. From the discussion, the teams gained the understanding of the existing issues which lead to the thoughts of what areas need to improve and how to improve. Identi‐ fying future action points, the teams discussed areas of improvement, which were focused on the process improvement. For example, in the retrospective meeting, most agile teams stressed testing environment issues that delayed the team progress. “we list down what didn’t go well or problems or whatever, we usually derive action points on those things, which is a good way to improve maybe something immediately like getting a test environment set up so we can test something.…like a more immediate thing… but there are also action points that are related to the squad as well; determine a team chart or something like that.” P2 – Developer, Team Jupiter 12 Y. Andriyani et al. From the example above, it was seen that by knowing the existing issues the team to will understand several areas of the process that need to be focused on. To determine future action points the teams also discussed by asking each other’s opinions. “when we discussed it [a plan], we asked other people what they think about it, do they agree or don’t they? If everyone says they think they agree with what you are saying, then we say so what the action for that?” P12 – Tester, Team Neptune During the retrospective observations of Team Saturn, an example of how the team identified their future action points was noted. Team Saturn had identified that the main reason for their slow progress was inaccurate estimation. Some ideas for addressing this included elaborating the stories into small tasks, providing the clear ‘definition of done’ for specific tasks, and asking for clarifications from the product manager about the scope of work. The team members were asked their opinions and perspectives about these ideas. Most team members agreed on asking for clarifications from the product manager and elaborating the stories into small tasks. Consequently, the Scrum master of Team Saturn made these ideas as official action points for the next sprint. 4.3 Reconstructing The Reconstructing level of reflection seems to happen when a team constructs an agreement on a specific plan based on the team members’ perspectives. There were three out of the four teams (Jupiter, Saturn, Uranus) that seemed to engage in the recon‐ structing level as they performed further discussions and finalized by generating action points. Generating a Plan. In reconstructing, teams generated plans decided from their discus‐ sion in the retrospective meeting. Action points are an explicit outcome of the retro‐ spective meeting. It is useful to remind all team members about the goal for the next sprint, who will responsible, and what are the associated deadlines. “So, when they go up on their board and they are doing their sprint work, they can see, “Right, let’s not forget what came out of this retro” and it is getting ticked off.” P11 – Scrum Master, Business Analyst and Product Owner, Team Uranus This point was brought up in a group interview (of Team Jupiter) where most of the team members agreed that action points were used as a reference for evaluating improve‐ ment in the next retrospective meeting. “umm we pulled out action points on the board. So, over the next two weeks, we will make sure that everything talked about we follow through on.” P4 – Business Analyst, Team Jupiter and Saturn It was observed that Team Jupiter preserved their concrete action points on their Scrum board (see Fig. 5). Another evidence from the observations was Teams Saturn and Uranus did not have action points but their Scrum master made some notes during the meetings and shared verbally the points that needed to be focused on at the end of the retrospective meeting. Reflection in Agile Retrospectives 13 Fig. 5. Action points generated by team Jupiter posted on their Scrum Board (Photo taken during on-site observations) 5 Discussion We now discuss the findings related to a reflection framework for agile retrospectives including the levels of reflection achieved by the teams studied, implications for practice and limitations of the study. In response to the RQ1: What aspects are focused on during the retrospective meeting? We found that there are six important aspects discussed in the retrospective meetings: identifying and discussing obstacles, discussing feelings, analyzing previous action points, identifying background reasons, identifying future action points and generating a plan. In response to the RQ 2: How does reflection occur in the retrospec‐ tive meeting? We found that the reflection that occurs in retrospective meetings can be classified into three levels of reflection [12], reporting and responding, relating and reasoning, and reconstructing. 5.1 A Framework of Reflection in Agile Retrospective Meeting Based on these findings, we present a reflection framework for agile retrospectives (Fig. 6) that combines the five steps of the standard agile retrospective – set the stage, gather data, generate insight and decide what to do, close the retrospective – and the levels of reflection – reporting and responding, relating and reasoning, and recon‐ structing [12] within those steps. 14 Y. Andriyani et al. Fig. 6. Reflection in agile retrospective meeting (levels of reflection depicted in shaded areas based on [12]) Setting the stage involves welcoming and explaining the aim of the retrospective meeting. Gathering data step embodies the reporting and responding level of reflection as agile teams share their reviews (e.g. identifying and discussing obstacles and discus‐ sing feelings). Identifying and discussing obstacles and feelings in retrospective meet‐ ings was seen to correspond to ‘descriptive reflection’ [16] – a reflection which attempts to answer questions such as: What is happening? What is this working, and for whom? For whom is it not working? How do I know? How am I feeling? What am I pleased and/or concerned about? What do I not understand? The obstacles and feelings shared by all team members answer these questions. From the obstacles and feelings reported, the teams would be able to record and collect important points of the previous sprint. By having reviews (e.g. obstacles and feelings) of the previous sprint, team members can be prepared to deal with other similar experiences. Generating insight step embodies the relating and reasoning level, where agile teams are involved in analyzing previous action points, identifying the background reasons behind identified issues and identifying future action points. Discussing these aspects was seen to be related to ‘descriptive reflection’, which attempts to answer questions: does this relate to any of my stated goals and to what extent are they being met? [16] and why the issues happen in the previous sprint? The answers to these ques‐ tions support the reflection in the form of comparative analysis and looking back to the background issues, which help agile teams to determine what areas needed to be focused on. Agile teams move to deep analysis on ideas or perspective shared to identify future action points for the next sprint. It can be perceived that there is a transformation in the discussion from answering what is happening? in the previous sprint; to what are the alternative views of what is happening? and what are the implications of the matter when viewed from these alternative perspectives? [16]. These questions are answered when all team members provide their accounts about solutions of the obstacles or ways to improve. In the deciding what to do step, agile teams have an explicit formulation which is generated in the form of action points (generating plans). The action points will be used as a reference for agile teams to act upon and improve the process. Close the retrospec‐ tive step involves summarizing the outcomes of the retrospective meeting. Reflection in Agile Retrospectives 15 5.2 Levels of Reflection Build on Each Other We now discuss the findings related to the levels of reflection achieved by the teams studied. A key finding of our study was that not all teams were performing on every level of reflection. So, while all teams performed retrospective meetings, not all achieved the higher levels of reflection, in particular reconstructing. Table 3 summarizes the levels of reflection achieved by each of the teams and the associated aspects or topics discussed in each level. Table 3. Levels of reflection achieved by the teams (J: Jupiter; S: Saturn; U: Uranus; N: Neptune) Levels of reflection Aspects discussed in retrospective meeting J S U N Reporting and responding Identifying and discussing obstacles ✓ ✓ ✓ ✓ Discussing feelings ✓ ✓ ✓ X Relating and reasoning Analyzing previous action points ✓ ✓ ✓ ✓ Identifying background reasons ✓ ✓ ✓ ✓ Identifying future action points ✓ ✓ ✓ X Reconstructing Generating a plan ✓ ✓ ✓ X Three teams were found to be fully engaged in all levels of reflection and one of the teams, Team Neptune, performed partially on the first two levels and did not achieve the final level of reflection, i.e. reconstructing. Based on the observation of their retro‐ spective meeting, it was seen that they did not discuss their feelings explicitly and only discussed briefly the obstacles related to changing of task priorities needing confirmation with the product manager. They did not discuss it further as once they agreed on that obstacle then the product manager directly proceeded to the Scrum Board, discussed the issue and wrapped up the meeting. They did not record any outcomes, such as a plan or action points, from the meeting. There was little evidence of analyzing previous action points, identifying background reasons and identifying future action points. Besides, the duration of the meeting was also short, around 15 min, and they reported performing retrospective meetings only when it was necessary. Another interesting observation was that they had adapted the retrospective practice, which seemed too repetitive for them and people often seemed to have forgotten about what happened during the last two weeks’ sprint. As result, they were used to placing all the individual reviews written up on sticky notes in a “retro box” – a box especially allocated to collect individual reflec‐ tion. If there were no sticky notes during a two weeks’ sprint, they would not perform a retrospective meeting. The case of Team Neptune is likely related to the fact that three out of six members of Team Neptune were new to agile projects. They had in effect introduced a new reflective practice, that of using a retro box, as a way to identify the need for conducting a standard retrospective. However, a lack of reaching the reconstruction level suggests that they were not able to generate a plan for improvement as several aspects of the retrospective meeting were missing. Our findings confirm that the levels of reflection are related and build on each other [12]. Furthermore, we show that the highest level of 16 Y. Andriyani et al. reflection, reconstructing, may not be reached at all or not reached effectively until the prior levels are accomplished effectively. 5.3 Implications for Research and Practice For the researchers in the area of reflective practice and agile teams, our findings present a new perspective for exploring reflective practice in agile teams. Using the framework presented in the previous section, researchers can study agile teams’ reflective practice in terms of levels of reflection both in retrospective meetings and other practices that involve reflection (e.g. daily standup, pair programming [7]). Future studies can explore new aspects or topics covered in each level and further explore how the levels build upon each other in different team contexts. For agile practitioners, our findings show that not all agile teams reach all levels of reflection by simply performing retrospectives. By being aware of the different levels of reflection meant to be achieved in each retrospective step, teams can consciously strive to achieve the most out of their retrospective meetings. In particular, they can see that only reporting and responding and relating and reasoning levels are not enough rather reconstructing to generate action points and following up on those points in future meetings is critical to harnessing retrospective meetings to achieve continuous improve‐ ment. Thus, in order to maximize the benefits of their retrospective meetings, we recom‐ mend agile teams use our reflection framework (Fig. 6) to self-assess their level as a whole based on their personal understanding of their team context and track it in practice to achieve higher levels of reflection. 5.4 Limitations A key limitation of this study lies in the fact that observations of a single retrospective meeting per team is not strong enough to establish and confirm a particular team’s overall level of reflection. For example, it may be that in other retrospective meetings Team Neptune reached higher levels of reflection. However, the findings were arrived at by combining the data from interviews as well as the observations, which provides multiple perspectives that support the findings. Another related limitation is that the findings are limited to the contexts studied in this research, which in turn are dictated by the avail‐ ability of participants. Further studies can confirm, adapt, or extend our framework to include different team contexts and reflective practices. 6 Conclusion Previous studies have focused on specifying the techniques of conducting a retrospective meeting, with little focus on how the reflection in the retrospective meeting actually occurs. One of the key contributions of our work is to present a reflection framework for agile retrospective meetings that explains and embeds five (grouped into three) levels of reflection within the five steps of a standard agile retrospective meeting. Critically, we show that agile teams may not achieve all levels of reflection simply by performing Reflection in Agile Retrospectives 17 retrospective meetings. As the levels of reflection build upon each other, teams need to effectively identify and discuss their obstacles and feelings in the reporting and responding level, followed by analyzing previous action points, identifying background reasons, and identifying future action points in the relating and reasoning level and generating a plan in the reconstructing level. Embedding these levels of reflection into the retrospective meeting will help agile teams achieve better focus and higher levels of reflection from performing retrospective meetings. Another implication is an increase in their awareness of the main aspects that need to be discussed in the retrospective meeting and how to formulate these aspects to generate a plan for improvement. Acknowledgement. This research is supported by The University of Auckland and the Indonesia Endowment Fund for Education (LPDP) S-669/LPDP/2013 as scholarship provider from the Ministry of Finance, Indonesia. References 1. Deemer, P., Benefield, G., Larman, C., Vodde, B.: A Lightweight Guide to the Theory and Practice of Scrum Version 2.0, vol. 2015 (2012) 2. Derby, E., Larsen, D., Schwaber, K.: Agile Retrospectives: Making Good Teams Great. Pragmatic Bookshelf, Raleigh (2006). 0977616649 3. Fowler, M., Highsmith, J.: The agile manifesto. Softw. Dev. 9, 29 (2001) 4. Sutherland, J., Schwaber, K.: The Scrum Guide. The Definitive Guide to Scrum: The Rules of the Game (2011) 5. Salo, O.: Systematical validation of learning in agile software development environment. In: Althoff, K.-D., Dengel, A., Bergmann, R., Nick, M., Roth-Berghofer, T. (eds.) WM 2005. LNCS (LNAI), vol. 3782, pp. 106–110. Springer, Heidelberg (2005). doi: 10.1007/11590019_13 6. Salo, O., Kolehmainen, K., Kyllönen, P., Löthman, J., Salmijärvi, S., Abrahamsson, P.: Self- adaptability of agile software processes: a case study on post-iteration workshops. In: Eckstein, J., Baumeister, H. (eds.) XP 2004. LNCS, vol. 3092, pp. 184–193. Springer, Heidelberg (2004). doi:10.1007/978-3-540-24853-8_21 7. Babb, J., Hoda, R., Nørbjerg, J.: Embedding reflection and learning into agile software development. IEEE Softw. 31, 51–57 (2014). doi:10.1109/MS.2014.54 8. Cockburn, A., Highsmith, J.: Agile software development: the people factor. Computer 34, 131–133 (2001) 9. Dingsøyr, T., Hanssen, G.K.: Extending agile methods: postmortem reviews as extended feedback. In: Henninger, S., Maurer, F. (eds.) LSO 2002. LNCS, vol. 2640, pp. 4–12. Springer, Heidelberg (2003). doi:10.1007/978-3-540-40052-3_2 10. Argyris, C., Schon, D.A.: Organisational Learning II: Theory, Method and Practice. Organisation Development Series. Adisson Wesley, Reading (1996) 11. Osterman, K., Kottkamp, R.: ReflectivePractice for Educators: Improving Schooling through Professional Development. Corwin Press, Newbury Park (1993) 12. Bain, J.D., Ballantyne, R., Packer, J., Mills, C.: Using journal writing to enhance student teachers’ reflectivity during field experience placements. Teachers Teach. Theor. Pract. 5, 51– 73 (1999). doi:10.1080/1354060990050104 13. Hoda, R., Babb, J., Nørbjerg, J.: Toward learning teams. IEEE Softw. 30, 95–98 (2013). doi: 10.1109/MS.2013.90 18 Y. Andriyani et al. http://dx.doi.org/10.1007/11590019_13 http://dx.doi.org/10.1007/978-3-540-24853-8_21 http://dx.doi.org/10.1109/MS.2014.54 http://dx.doi.org/10.1007/978-3-540-40052-3_2 http://dx.doi.org/10.1080/1354060990050104 http://dx.doi.org/10.1109/MS.2013.90 14. Yin, R.K.: Case Study Research: Design and Methods. Sage Publications, Inc. (2003) 15. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77–101 (2006) 16. Jay, J.K., Johnson, K.L.: Capturing complexity: a typology of reflective practice for teacher education. Teach. Teacher Educ. 18, 73–85 (2002) Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. Reflection in Agile Retrospectives 19 http://creativecommons.org/licenses/by/4.0/ What Influences the Speed of Prototyping? An Empirical Investigation of Twenty Software Startups Anh Nguyen-Duc1(✉), Xiaofeng Wang2, and Pekka Abrahamsson1 1 Department of Computer and Information Science (IDI), NTNU, 7491 Trondheim, Norway {anhn,pekkaa}@ntnu.no 2 Free University of Bozen-Bolzano, Piazza Domenicani 3, 39100 Bolzano, Italy xiaofeng.wang@unibz.it Abstract. It is essential for startups to quickly experiment business ideas by building tangible prototypes and collecting user feedback on them. As proto‐ typing is an inevitable part of learning for early stage software startups, how fast startups can learn depends on how fast they can prototype. Despite of the impor‐ tance, there is a lack of research about prototyping in software startups. In this study, we aimed at understanding what are factors influencing different types of prototyping activities. We conducted a multiple case study on twenty European software startups. The results are two folds; firstly we propose a prototype-centric learning model in early stage software startups. Secondly, we identify factors occur as barriers but also facilitators for prototyping in early stage software startups. The factors are grouped into (1) artifacts, (2) team competence, (3) collaboration, (4) customer and (5) process dimensions. To speed up a startup’s progress at the early stage, it is important to incorporate the learning objective into a well-defined collaborative approach of prototyping. Keywords: Prototype · MVP · Prototyping-learning loop · Validated learning · Speed · Software startups 1 Introduction With the startup movement, software industry is witnessing a paradigm shift from serving customer requirements to creating customer value. The challenge for software companies is no longer primarily on implementing customer requirements, but rather on finding customer demands and providing a solution that delivers customer value [2]. Addressing uncertainty in both solution and problem domains has often been ad-hoc and based on guesswork, which becomes one of the main reasons for failing startup companies [3]. A demand on systematic approaches to manage the uncertainty has led to an increased research interest on Lean Startup [4], New Product Development (NPD) [5], software startups [6] and continuous experimentation [7]. In a competitive environment such as software industry, time-to-market is becoming more and more critical as a success factor for startup companies. Business ideas under development once revealed can be easily threatened by high speed copycats [9]. More‐ over, competitors can also follow an on-going journey of validating product-market fit © The Author(s) 2017 H. Baumeister et al. (Eds.): XP 2017, LNBIP 283, pp. 20–36, 2017. DOI: 10.1007/978-3-319-57633-6_2 and arrive faster in the destination. Regardless of company sizes and application domains, the knowledge of influencing factors for a quick learning loop is important for software startups to form best-fit strategy in developing business experimentation [10]. A ‘Build-Measure-Learn’ loop, as a central concept of the Lean Startup method‐ ology, aims at speeding up the new product development cycle [4]. The central part of the loop is to build a representation of the business, a so-called Minimum viable product (MVP), to collect feedback from customers and to learn from that. Steve Blank empha‐ sizes the goal of MVPs is “to maximize learning through incremental and iterative engineering” [2]. In the startup context, developers quickly and iteratively develop a software application to validate business ideas [12]. As such, the study of validated learning can be beneficial from Software Engineering (SE) concepts and practices, such as rapid prototypes and evolutionary prototypes [13–15]. Consequently, the time-to- release of prototypes is essential to determine the total time in the validated learning loop. Software startup research is increasingly recognized by researcher’s community, with many practical aspects, such as User Experience, Software practices, competences and startup ecosystem [6]. Despite of the importance, there is a lack of research about prototyping in software startups. In a multi-influenced context with funding, human resource and market concerns, it is crucial to understand how the speed of learning can be supported by prototyping activities and what are the influencing factors. In a previous study, we investigated how a prototype is built in software startups [12]. We found that prototyping activities as a core value of startup experimentation needed to be seen as a multifaceted phenomenon [12]. In this work, we are particularly interested in the factors that slow down the learning process and those that speed it up. The research question (RQ) is: What factors influence the speed of prototyping in software startups? The paper is organized as follows. Firstly, we present the background about business- driven experimentation in software projects, software prototype and a proposal of an analytical model of startup prototyping (Sect. 2). Then, we describe our research approach and the cases studied (Sect. 3). After that, the qualitative findings are presented (Sect. 4). Finally, we reflect on the findings, the threats to validity (Sect. 5), and draw the conclusion and future work (Sect. 6). 2 Background 2.1 Business Driven Experimentation From SE perspective, validated learning means the focus on integrating business value in defining software development processes and practices. Even though experiment systems are recognized as beneficial to software projects, there are barriers in adopting them, such as integration of customer feedback, synchronizing vendors in short cycles and lack of reasoning about customer requirements [16, 17]. Bosch et al. [18] advocate for adjusting the Lean startup methodology to accommodate the development of What Influences the Speed of Prototyping? 21 multiple ideas and to integrate them when time for their testing and validation is too long. Bosch suggested using 2-to-4-week experimentation iterations followed by exposing the product to customers in order to collect feedbacks. Fagerholm et al. present a model for continuous experimentation for start up companies [7], in which a key element is the ability to release a prototype with suitable instrumentation, to manage experiment plans, link experiment results with a product roadmap, and to manage a flexible business strategy. Olsson et al. present a Hypothesis Experiment Data-Driven Development model that integrates feature experiments with customer feedback in Agile projects [19]. While these work characterize a process-like approach in developing startups’ software products, Paternoster et al. grounded a model from 13 software startups which describes a pattern that software startups often build evolutionary proto‐ types [20]. This study focuses on how startups are prototyping in reality and the influ‐ encing factors of the speed of learning by prototyping. 2.2 Prototype and Prototyping Activities Brook mentioned “In software engineering, at least, the concept of rapid prototyping has a name and a recognized value, whereas it does not always have the same status in computer design and in building architecture” [21]. Prototyping implies a quick and economic approach that serves to achieve understanding of what final products should be [15]. From a technical perspective, prototypes can be differentiated according to its relation to later product development. Throwaway prototypes are used mainly for spec‐ ification purposes; and they are not used as actual building blocks [15]. They are mostly used in exploratory and experimental prototyping. Evolutionary prototypes provide a basis for a real system, which is evolved out of the prototypes; they are used in evolu‐ tionary prototyping but can also be found in experimental prototyping (if it shows that they provide a good basis for a system) [15]. From a business perspective, startups can create a representation of product ideas, a so-called MVP, without actual product implementation. Eric Ries describes a classifi‐ cation of different types of MVPs [4], which are commonly used in the startup commun‐ ities. For instance, a MVP can be a short animation that explains what a product does and why users should buy it. It can also be a user interface that looks like a real working product, but the actual business process is manually carried out (Wizard of Oz MVP). A concierge MVP is a manual service that consists of exactly the same steps users would go through with the product. A few research paid attention on improving prototyping activities, such as the speed and effectiveness [28, 29]. Janssen et al. suggested code reuse to speed up writing code to prototype [28]. Grevet et al. described a 6-stage prototyping approach to speed up throw-away prototyping for new social computing systems using existing online systems [29]. In our work, we address the speed of prototyping from a socio-technical perspec‐ tive, considering prototyping activities under human, market, finance and team factors. 22 A. Nguyen-Duc et al. 2.3 A Prototype-Centric Learning Model in Software Startups The Build-Measure-Learn loop is a key concept in Lean Startup [4]. The loop is used to manage and to operate software startups in finding a sustainable business model. A key idea is to minimize waste and to focus only on the elements, which will be tested. Lynn et al. describe another cycle, Probe and Learn, that is applicable to manage uncer‐ tainties about market, technology and time-to-market [25]. The authors suggest that startups should go to customers with an early version of a product to learn about the market, different applications and segments. Nguyen-Duc et al. propose a hunter-gather double loop to capture the evolution of startup activities from idea to achieving a product market fit [26]. The model visualizes the portion of product development vs. customer development activities across the startup stages. While these studies provide an emphasis on organization and evolution, they are well landed in an abstract space, not straight‐ forward to apply from the SE perspective. In the SE literature, Gordon et al. propose a rapid prototyping system approach to understand the prototype development of a system [27]. In the model, both low-fidelity and high-fidelity prototypes are essential parts of developing a system [27]. Preliminary product design activities create a throwaway prototype from the problem domain. A series of throwaway low-fidelity prototypes can be created to capture the ideas of what to built. Similarly, high-fidelity prototypes can also be evolved several times before reaching the product launch. A literature survey of software development shows that startups often build a proto‐ type in an evolutionary fashion and quickly learn from users’ feedback [20]. We argue that both throwaway prototypes and evolutionary prototypes are important parts of startups’ journey to a launched product. From the Lean startup perspective [4], learning is an input and also an outcome for a prototype. We tailored the double loop model in the previous work [26] by adapting Gordon’s system prototyping elements [27] to capture the prototyping processes in the startup context, as shown in Fig. 1. The model focus on prototyping as the core concept and compose four loops: • Idea-prototype loop: iterations of refining business idea through throwaway proto‐ typing • Throwaway prototype loop: iterations of constructing and learning from throwaway prototypes • Evolutionary prototype loop: iteration of constructing and learning from evolutionary prototypes • Pivot loop: starting a new cycle from the current product to a pivoted idea Considering the model as a state-based system, it is possible to travel from a state to any other one. However, the typical flow would happen within two loops. It can also happen that a startup starts the loop from any state, for example, by doing a throwaway prototype before getting to a stated problem. In the scope of this work, we did not go in-depth about how these loops happen in our cases. The work will explore factors that occur during the startup progress and influence throw-away and evolutionary proto‐ typing. What Influences the Speed of Prototyping? 23 3 Research Approach 3.1 Multiple Case Study Design This study is one part of a larger research activity that investigates the role of engineering activities in software startups. The objective is to explore commonalities, challenges and engineering patterns in software startups, from the business idea to a launched product. This study reports the findings from empirical data regarding prototyping activities. We conducted multiple case studies for a robust result in typical software startup population [11]. The unit of analysis is a startup company. We aimed at collecting as many startups as possible for a variety of the sample. As the aim is to reflect the state-of-practice rather than finding a secret recipe of success, we included startups in different stages and with different revenue statuses. There is often a difficulty in identifying a real startup case among other similar phenomenon, such as freelancers, SMEs or part-time startups. We defined five criteria for our case selection: (1) a startup that operates for at least six months, so their expe‐ rience can be relevant, (2) a startup that has at least a first running prototype, (3) a startup that has at least an initial customer set, i.e. first customer payments or a group of users, (4) a startup that has an intention to scale their business model, (5) a startup that has software as a main part of business core value. The process of identifying and collecting data was done in 11 months, from March 2015 to February 2016. Cases were searched from four channels, (1) startups within the professional networks of the authors, (2) startups in the same town with the authors, (3) startups listed in Startup Norway and (4) Crunchbase database. The contact list includes 219 startups from Norway, Finland, Italy, Germany, Netherlands, Singapore, India, China, Pakistan and Vietnam. After sending out invitation emails, we received 41 feed‐ backs, approximately 18.7% response rate. Excluding startups that are not interested in the research, or startups that do not pass our selection criteria, the final set of cases are 20 startups, aliased as S1 to S20. Fig. 1. A prototype-centric learning model in software startups 24 A. Nguyen-Duc et al. 3.2 Data Collection and Analysis Semi-structured individual interviews were used to collect data, since they enable the focus on pre-defined research topics and flexible structures to discover unforeseen infor‐ mation [28]. Methodological triangulation in data collection is also implemented by using evidence from documents and observations (in S01-S05, S09). Business docu‐ ments, such as business model canvases and business plans were exposed to the research team as a preliminary step prepared for interviews. Observations were useful to under‐ stand how prototypes were implemented and used in the working environment. The interviewees were asked questions about (1) business background (2) idea visu‐ alization and prototyping (3) product development (4) challenges and lessons learnt. The stories about startup ideas, prototypes and product development is organized into the schema as described in Fig. 1. Most of the interviews were conducted by the first author, with the attendance of a second researcher (the third author or sometimes external researchers in our network). This researcher has a long experience conducting interviews in software companies. Each interview lasted from 55 min to 70 min and the interviewees were informed about the audio recording and its importance to the study. We used a thematic analysis – a technique for identifying, analyzing, and reporting standards (or themes) found in qualitative data [22]. We started by reading all interview transcripts and relevant documents, and coded them according to open coding [22]. A set of pre-determined categories were used to guide the coding process, as we have some interests in topics of (1) business original, (2) prototyping practices (3) pivoting (4) testing (5) challenges and (6) key performance indicators (KPIs). We attempted to label all meaningful text segments with appropriate codes. To feed data to this study, we filtered the codes that are related to prototyping, technical implementation, and testing activities prior to product launching. According to Sect. 2.2, throwaway prototypes were low-fidelity artifacts, such as mockup, wireframe, or simple code. Evolutionary proto‐ types were perceived as product building blocks, such as heavy code activities, i.e. feasibility testing of functionality, building new feature, etc. The relationship of the factors to the speed of prototyping or production was identified via text about challenges, or text specifying consequence on time-to-market or time to collect user feedback. We noted and reported evidence on prototyping as follows (1) factors that relate to proto‐ typing activities in generals, (2) factors that slow down the prototyping activities and (3) factors that speed up the prototyping activities. 3.3 Case Description The characteristics of our cases are given in Table 1. It is noticeable that a large number of the studied cases deliver peer-to-peer services as marketplaces or platforms (S01, S02, S03, S07, S08, S11, S13, S16, S20). There are also cases that deliver value in Business-to-Busi‐ ness model (B2B) (S04, S06, S10, S12, S15, S17). The cases are dominantly characterized by web-based and mobile-based software product with client-server architecture. We also identified the product focuses in early and later phases of the software startups [23]. Among them, there are some startups with annual revenue of one million euro or more (S06, S09). What Influences the Speed of Prototyping? 25 Regarding the development strategy, interestingly, there are seven cases (35%) that have (parts of) product developed outside company boundary. Table 1. Startup cases characteristics Code Product type Early focus Later focus Dev. strategy No. of prot. Dev. method. S01 Photo marketplace Feature Insource 2 Agile S02 News generator UX New feature Outsource 4 Agile S03 Homemade food market UX Insource 2 Adhoc S04 Construction management Simple feature New feature Outsource 5 Distributed agile S05 Underwater camera Feasible technology Outsourcing, subcontracting 7 Informal agile S06 Sale visualization tool UX Flexible, scalable Insource 3 Informal scrum S07 Location recommendation Feature, UX Insource 3 Informal agile S08 Ticket platform Intuitiveness, friendliness Scalable and new features Outsource 2 Agile S09 Educational quiz system User friendliness Scalable, Stable Insource 5 From adhoc to distributed agile S10 IoT OS platform Ecosystem Functionality Insource 4 NO INFO. S11 Ticket platform User friendly, simple More features, complexity Insource 2 Adhoc S12 Elearning platform Feature Insource 3 Agile S13 Shipping services NO INFO. NO INFO. Outsource 3 NO INFO. S14 News services Feature provider Platform as a service Insource 2+ Continuous development S15 Smart grid application NO INFO. NO INFO. Insource NO INFO. NO INFO. S16 Secondhand marketplace innovative feature Product line Insource 3 NO INFO. S17 Simulation based training UX, feature Flexibility, Scalability Insource 2 + NO INFO. S18 Open source messenger Community Feature Open source 4 Adhoc S19 Location based alert system UX Feature and enhanced UX Insource 5 Agile S20 Elearning system User friendliness Standardization Insource 2 Agile *Notation: NO INFO. means missing information The major reported development methodology is Agile, with iterative deliveries and frequent customer feedback: “… Scrum based development, sprints of two weeks, standup, wrap-up meeting, we like to work in this way.” (S06). In some cases, the company reports a type of informal Agile process: “… fully informal but truly agile process with working release maintained, … iterative development of functionality and refactoring” (S05) 26 A. Nguyen-Duc et al. One specific question asked to interviewees is how many prototypes have been made before product launching. The answers vary from two to seven prototypes, either throw‐ away or evolutionary ones, before a launch. In many cases, we considered prototypes as a tangible artifact that is experimented with (potential) users, customers and internal/ external stakeholders. 4 Result Figure 2 describes the influencing elements on throw-away prototyping (detail on Sect. 4.1) and evolutionary prototyping (detail on Sect. 4.2). It should be noticed that the direction of impact is not given. Some elements specifically show the positive/nega‐ tive influences while other elements remain as general observations. Fig. 2. Factors influencing the prototype-centric learning loops 4.1 Elements Influencing Throwaway Prototyping 4.1.1 Adoption of Collaborative Mock-up Tools By adopting various tools, i.e. paper sketch, GUI mockups and wireframe tools, startups achieve a fast and an economic prototype without any technical expertise, as described in (S02, S09, S11, S13). In these cases, startups conducted very short iterations, from a few days (S02, S11, S13) to a few weeks (S09), from a product or a service idea to having the first user feedback. In S04, printing GUI layout in papers is reported as a good practice for teamwork, especially improving the customer involvement: “normally we draw in the piece of paper first and then we make mock-ups… and then the customer joins us on that journey, then we click on the paper, we go to another one …” (S04). It is also common that startups build mockups by using cloud-based software services. For such an online tool, the teamwork mode is reported as an important feature that facilitates collaborative design efforts among distributed team members (S02). What Influences the Speed of Prototyping? 27 4.1.2 UX Designer Onboard Business side of a startup (often CEOs) is always in a need of expressing and visualizing their ideas into more tangible artifacts. By doing that, sitting next to a designer is highly desirable for CEOs in early stages. In S02, the CEO expresses the need for a close collab‐ oration with a designer in team: “In this case, I would really like a designer that sits here together with us …” (S2). The role of a design in mobile application is highlighted in another discussion with S2: “You might think of user interface as a make-up for a person. But I think UI is the capacity that an app needs to interact with people.” It happens simi‐ larly in S12, when the CEO mentions about the process of designing the graphical part of their prototypes: “The alternative is to create a specification … and just developing that document and all the process around it is typically very resource intensive. We talk about a future, … we make a prototype at a first phase implementation and then we adjust from there based on dialogues in between us.” (S12). For frontend-rich applications, a designer is a champion of the user experience, considering the viewpoint of users and keeping consistency among graphical elements across different platforms. 4.1.3 Choices of Faking or Building There are often many uncertainties about customers and their expectations in the early stages of startups. Starting with a single-feature prototypes, or other approaches with implementation come always with a risk of wasting effort. It is considered time-saving to start with a clear mind about the throw-away strategy, by focusing on demonstrating business value rather than reusing the technical components (S02). Uncertainty about what to build and how to build often come with quick and dirty experiments without proper architectural designs, appropriate coding practices and documents. In this manner, frequent change of requirements or feature requests could lead to the increase of technical debts in later phase. Experimenting by the development of a runnable prototype was a costly and time-consuming experience in S09. In this way, the value of a prototype should exceed its cost. In S03, the development team has a clear plan for experimenting without “making the product” until they get the right product design. S11 applies the concept of “fake it until you make it”, to simulate a final product without primary quality, both with functionalities and user experience. However, the focus on the speed has also led to the minimum part of viability. In S11, customer demonstration was done in a wizard of oz manner [4], customer interacting with an actual user interface, but business logics and backend functionality were done by manual work. Even though it is inefficient, the approach is easy and fast to build. 4.1.4 Collaboration Across Diverged Mindsets We observed that in most of the cases, the ideas came from the CEOs, who are often business people or serial entrepreneurs. While the decisions about what the products should do come from a business mindset, they are implemented by developers with a technical mindset. In some cases (S01, S04, S05), there are challenges in communicating the product ideas and convincing the developers about the product value. In S04, it took as much time to discuss on the value proposition as to sketch a mockup. Vice versa, the communication of technical difficulties is also a time-consuming task, as mentioned by 28 A. Nguyen-Duc et al. a developer in S05: “She [the CEO] is very sharp about business and finance stuffs, but it takes a long discussion to explain her about the importance of having flexible product design …” (S05). The communication challenge might also happen between startups and customers, when no concrete prototypes are provided: “We work with a customer organization, learn how they have worked with the current solutions and describe our proposal via the prototype. It is hard for them to realize the benefit without concrete examples…” (S04). It also appears that a prototype is late released due to the wrong estimation of the CEO, who has no technical background. For example, in S1, the CEO insisted on a customer feedback having a new field in a frontend form, which caused the change of both business logic layer and data table structure. 4.1.5 Identification of a Right Set of Feedbacks Steve Blank emphasizes the importance of early involvement of end users in product development [2]. Particularly, in startups developing products for mass market (or B2C business model), the feedback from the representative users of a market segment is essential. Nevertheless, not all users’ input is equally valuable to product development. It was difficult to find the customer feedback that is useful for validating hypotheses in S02: “I have attended a various types of events like that. To be honest, there are not so many interesting things there …” (S02). The CEO wandered in town and talked to different people about the product idea. However, the approach is quickly found ineffi‐ cient, as the users’ feedbacks are often shallow. After that, the CEO targeted a group of innovative users from startups and research community and documented many inter‐ esting ideas for the product features. The integration of such lead users, “whose strong needs will become general in a market-place months or years in the future” [24], appears to be an important factor to accelerate the speed of startup learning. Lead users are also able to contribute via suggestions, testing and feedback, or even participate in the devel‐ opment and co-creation of new products or services, as observed in S14: “We always do that in a close relation to our actual client stakeholders. Once we decide to narrow it on a new product area, the first thing we do is to get a partnership with a customer so that we can work together on a daily basis as stakeholders and product devel‐ opers…” (S14). 4.1.6 Fostering Customer Knowledge and Embedding into Prototypes Prototypes can be seen from three different perspectives, function, look-and-feel and role, in which role is the representation of usability of the prototype [2]. In order to maximize lessons learned from a prototype, the vision on how end-users adopt a final product need to be visualized and captured in the prototype. As the actual end users are often not well known in the early phases, the integration of the user’s role into the prototype design is a fuzzy task. The time pressure on prototyping makes startups skip a detailed analysis of users’ behaviors. It seems that the adoption of customer/ market analysis tools are not so common in our startup sample. In S02, the CEO emphasized the role of mapping tools, such as a customer journey map to describe the customer’s experience: “I have been told by my friends about the tool [a customer What Influences the Speed of Prototyping? 29 journey map]. We used it to describe how customer interact with the system and where could be the gap” (S02). 4.2 Elements Influencing Evolutionary Prototyping 4.2.1 Utilizing Plug-and-Play Components in Prototype Utilizing ready-made components, such as Open source software (OSS) libraries and frameworks unlocks the capacity of experimenting functional as well as non-functional features. The adoption of OSS components was mentioned in all of the cases, from using tools (S19), integration of OSS code (S02, S03, S05, S20), to participation in OSS community (S18). The main benefits include reduced development cost and faster time- to-release, which were mentioned by the CTOs of (S19) and (S20): “…we might not even come to the idea of making it happen if we do not have OSS as an experiment. Without OSS it would take a lot of time and very costly” (S19). It is an even more obvious choice in open source type of platforms: “It is very hard nowadays not to use OSS artifacts, especially when with Android development …” (S20). It also appears that many advanced technologies were adopted via using OSS: “A core part of our product includes a machine learning algorithm. We are lucky enough to find ml library in C++, entirely OSS, super cool” (S02). By taking ready-made components, startups also reduce proto‐ typing time by simplifying architectural aspects to some existing patterns. 4.2.2 Synchronizing Customer Feedback in Loops Communication among team members or between a startup company and its external stakeholders is found as a significant factor delaying an iteration release. Insufficient communication due to misunderstanding, cultural difference, language barrier, lack of supporting tools happens often in outsourcing and remote partnership scenarios (S01, S09): “Basically, we found some limitations that made it difficult to be efficient in the way to communicate. And since we’re teams in different places it’s really important that information flow works and also to make sure that all people—don’t have to be involved in everything, and be able to group efficiently and create like projects, and store docu‐ ments, and all these things, and have video-share links, and articles, and all these things.” (S09). The misunderstanding and reworking also happens when customers are distant to developers and the customer feedbacks are not fully perceived. In S13, the CEO and sales people interacted with customers and collected insightful feedback from them. However, the feedback is not communicated efficiently to the development team in other locations. This leads to unnecessary re-work with communication and imple‐ mentation effort and hence slows down the time to release. 4.2.3 Conflicting Feature Requests It is a typical situation that evolutionary prototypes are built based on feature requests from the first customers. Gradually, when having more customers, new feature requests might vary from the business direction or even conflict with the previous functionalities. S14 describes how they handled such situation: “either we solve them by providing them different products or we do ignore parts of the market… We make a very clear statement 30 A. Nguyen-Duc et al. to what we think the future of journalism is, then we pursue that and the cost of that is neglecting parts of our market” (S14). Similarly, S15 expresses how their product evolved through different iterations: “There will always be requirements arriving… Sometimes the new requirements disrupt the old requirements. At the moment, we are working to disrupt the old products” (S15). Considering what to develop and which features to include adds complexity to future releases. Additionally, requests coming in the middle of the development sprint from large customers might influence the feature priority and delay the release further: “We’re in that situation all the time, it’s very difficult to say no because giant customers telling you we need that functionality. If you’re going to have us as customers you’re going to have to make it, we need it in the contract that you have to make it. We also build it, we built it bigger and bigger” (S11). 4.2.4 Feature Creeps Many startups add new features to fit the prototype to a changing group of early customers. This leads to two possible challenges of satisfying customer demands, so- called (1) feature creep and (2) product portfolio. Feature creep refers to the addition of features to a product in a continuous manner: “We are adding features all the time. This is not a product that will ever stop evolving. We will always have a strong engineering team to develop the product forward. We are not talking about maintenance here. We are talking about this being the core of the company’s competence” (S13). Startups rarely have a requirement management process to manage product complexity. Conse‐ quently, feature creeps are considered harmful to the production and enhancement of core features. Moreover, this can be an unwanted expansion that requires changes also in the product architecture and even in the strategic direction. In S04, after the first two releases addressing a construction manager’s requirements, the third release was developed for a construction operator’s demands. Consequently, S04’s product scope has grown from a single feature MVP to a supply-chain management system: “So then we had a small one just for easy communication between users of the building and the maintenance guys… So the second feature was to manage document flow. And the third was to have a 3D model of the building. And all these things here we spent a lot of time and we were building in parallel with different prospects” (S04). In a larger scale, the expansion could lead to deriving a product portfolio. Startups face with challenges of keeping both the focus to increase the quality of core delivered values and satisfaction of important customers. While not all good ideas can be turned into features, some ideas are selected to develop further and might become the core value providers for startups. 4.2.5 Solid Technical Competence Onboard In several cases (S09, S01, S03, S06) the technical competence determines the speed of feature releasing. Startups’ technical members are required to possess good technical skills and they also need to be productive in an ambiguous development environment: “We don’t hire people basically for them being cheap because we don’t have time. Our challenge is time and to be more productive other kind of competing companies … it’s What Influences the Speed of Prototyping? 31 much better to have people that can—within a short time, could produce good code” (S09). It is also important to write code in a clean and structured manner, to be quality- aware in the early phases: “The back end was pretty good because he had hired my boss at my current company … there was some friction there in how to develop systems between the professional programmer, my boss, and the copy paste programmers. I think that also contributed to it not working.” (S11). The combination of technical competence and customer understanding is emphasized in another case: “… It is very hard to find people both good at technology and have a good sense of commercial edge…” (S08). 4.2.6 Dependence on Fast Changing Technologies Startups often struggle with thriving in a technical uncertainty, whether under market pull or technology push impacts [20]. Due to different reasons, e.g., specific devices, platforms or protocols becoming popular in market, or new technology gaining momentum, there are needs for changing the current product’s features to accommodate new technology (S01, S09, S11). In a small scale, for instance, the adoption of new animation effects, a different type of map, etc. leads to an extension of the current or coming iterations. In S02, the development of an IOS application is delayed after the codebase and all dependent libraries were forced to be upgraded to a newer version of Swift. The team took time to resolve all the changes so the next release can be done in Swift 3.0. The technology uncertainty is expected with mobile applications, as stated by the CEO of S11: “…at the moment we are changing the technology platform. This perhaps has been the biggest challenge we have decided where to stand and make a new platform on development technology… So next generation which will be out in the market place around summer next year will be quite heavily rearranged.” (S11). In a large scale, the technical change can lead to a change of business directions. 5 Discussion 5.1 Reflections on the Results We captured what happened during the early phases of the studied twenty software startups. We identified the factors that are found to influence the speed of prototyping across different types of prototypes. They can be grouped into (1) Artifacts, (2) Team competence, (3) Collaboration, (4) Customer and (5) Process dimensions. Artifacts include c