The influence of a Test-driven Development on the

Matjaž Pančur (2005) The influence of a Test-driven Development on the. PhD thesis.

Preview

PDF (Disertacija) Download (2193Kb)

Abstract

This dissertation is about the empirical assessment of quantitative and qualitative parameters of Test-Driven Development (TDD). The introduction presents the problem of unsuccessfulness of software development projects and emergence of agile methodologies. The common characteristics of all agile methods including most popular amongst them - Extreme Programming (XP) - are presented. Detailed knowledge of methodologies' components enables focused tailoring of chosen methodology by introduction of proper techniques and development processes that minimize or eliminate its weaknesses. The second chapter describes basic characteristics of the XP and the role of TDD within it. TDD represents central practice of XP and it is attributed with the biggest share of XP's successfulness among developers. In TDD, developer writes tests before corresponding implementation code. A process called ITL (Iterative Test-Last Development) was derived from TDD with all of its characteristics except one: developers in ITL write tests after the code. TDD and ITL are described and compared in detail. Currently there are no known proofs about the effects of TDD usage. Some case studies exist, but unfortunately only a few controlled experiments. At the end of the second chapter all related research is presented, reviewed and compared to our approach. The third chapter focuses on detailed description of experiment design and its execution. The goal of our research is to identify if and how writing tests first influences productivity, quality (external and internal) and tests (quantity and quality).We also want to estimate various other parameters that describe development process: time spent in phases of coding, testing and refactoring; frequency of test evaluations, percentage of successful tests and test to code ratio. All these quantitative parameters are evaluated through participating developers' qualitative observations, with focus on productivity, efficiency, quality and difficulties in adoption of TDD and ITL. Based on our theories, hypotheses are formalized. All experimental subjects are described, along with experimental tasks, data collection and validation mechanisms. We explore and dismiss all threats to experimental validity, and then we present data analysis procedures with special emphasis on statistical power. Experiments are conducted in two sets: the first set from February 2003 to June 2003, the second set from February 2004 to June 2004. Our subjects were undergraduate Computer Science students in their eight semester of study. Two equivalent groups were formed and the only independent variable in the experiment was the development process (TDD or ITL). All subjects listened to the same classes, worked with same tools under same conditions, etc. Each set consisted of two separate experiments in two different contexts. The first context was programming in pairs (emulation of TDD in XP), while in the second context subjects worked alone (emulation of using TDD in some other agile or traditional methodology). The fourth chapter presents results and analysis of experimental data. We found no statistically significant differences between TDD and ITL regarding productivity and quality. Because of sufficient statistical power we can conclude with high probability that there are no big differences in productivity and quality. We also didn't find any statistically significant differences in the number of tests, but the test quality was 7% higher in TDD group. TDD steers its developers towards increased test quality whereas the number of tests stays roughly the same as in ITL. ITL has 20% more successful test runs, which also confirms our theory and TDD group's process conformance. We found no differences in test-to-code ratio and test evaluation frequency. Because of contradictory results we can neither confirm nor reject hypotheses about differences in time spent in coding, testing and refactoring. The effect size is essential in designing an experiment with sufficient power. Our effect size estimations will help other researchers in replicating our experiments or designing new experiments with focus on TDD. On average, students were less enthusiastic about TDD's productivity, efficiency and quality compared to responses from professional developers. Students perceive TDD as much harder practice to adopt then professionals do. ITL, on the other hand, is much easier to adopt and is also perceived as much more efficient than TDD. Majority of students also tend to use ITL in the future, which is not the case with TDD. In the conclusions, we evaluate our findings and contributions and offer some guidance about introduction of TDD and ITL in a professional development environment and in a University environment. Some suggestions for further research are presented at the end. Our main contributions to science are: . Design and implementation of the experimental environment. . Estimation of effect sizes for quantitative parameters of TDD and ITL. The size of the effect can be small at most in productivity and external quality, medium in test quantity, frequency of test evaluations and test to code ratio. We proved a big effect in the difference of mean values between TDD and ITL in test quality and percentage of successful test runs. . Method for empirical assessment of quantitative parameters (productivity, quality, tests, time spent in development phases, frequency of test evaluations, percentage of successful tests and test to code ratio) and qualitative parameters (productivity, efficiency and quality, difficulties in adoption) for TDD and ITL. . Validity analysis of research hypotheses of the differences between TDD and ITL in productivity, quality, tests, time spent in development phases, frequency of test evaluations, percentage of successful tests and test to code ratio. . Recommendations for introduction of TDD and ITL in a professional development environment and validation of suitability and its acceptance time in a University environment.

Actions (login required)