Beg, Borrow, or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research

An ICSE 2000 Workshop Report 
Limerick, Ireland,  5 June 2000



Susan Elliott Sim
Department of Computer Science
University of Toronto
10 Kings College Rd.
Toronto, Ontario Canada M5S 3G4
Tel. +1 (416) 978-4158
Fax. +1 (416) 978-4765
Dr. Janice Singer
National Research Council Canada
Bldg M-50, Rm. 257
Montreal Rd.
Ottawa, ON Canada K1A 0R6
Tel: +1 (613) 991-6346
Fax: +1 (613) 952-7151
Dr. Margaret-Anne Storey
Assistant Professor
Department of Computer Science
University of Victoria
PO Box 3055 STN CSC
Victoria, BC Canada V8W 3P6
Tel: +1 (250) 721 8796
Fax: +1 (250) 721 7292


                                                                                                                                                                                                                              Susan Elliott Sim                                                                                                                                                                                                                               Janice Singer                                                                                                                                                                                                                              Margaret-Anne Storey                                                                                                                                                                                                                             

                                                                                                                                                                                                                              Dept. of Computer Science                                                                                                                                                                                                                               National Research Council                                                                                                                                                                                                                              Dept. of Computer Science                                                                                                                                                                                                                             

                                                                                                                                                                                                                              University of Toronto                                                                                                                                                                                                                               Bldg. M-50, Rm 257 Montreal Rd                                                                                                                                                                                                                              University of Victoria                                                                                                                                                                                                                             

                                                                                                                                                                                                                              10 Kings College Rd., Toronto                                                                                                                                                                                                                               Montreal Rd.Ottawa, ON                                                                                                                                                                                                                              PO Box 3055 STN CSC                                                                                                                                                                                                                             

                                                                                                                                                                                                                              Toronto, ON,                                                                                                                                                                                                                               Ottawa, ON                                                                                                                                                                                                                              Victoria, BC

                                                                                                                                                                                                                              Canada M5S 3G4                                                                                                                                                                                                                              Canada   K1A 0R6                                                                                                                                                                                                                              Victoria, BC Canada V8W 3P6                                                                                                                                                                                                                             

                                                                                                                                                                                                                              +1 (416) 978 4158                                                                                                                                                                                                                              +1 (613) 991 6346                                                                                                                                                                                                                              +1 (250) 721 8796                                                                                                                                                                                                                             




Executive summary


The goal of this workshop was to provide an interactive forum for software engineers and empirical researchers to investigate the feasibility of applying proven methods from other research disciplines to software engineering research. In the call for papers, Pparticipants were invited to submitted position papers describing problems that might benefit from a multidisciplinary approach. Expert guest speakers from software engineering and other disciplines addressed the issues highlighted in the papers with the goal of encouraging more multidisciplinary research.  Papers accepted for the workshop were grouped into four sessions based on similarity.  These sessions were: 1) Foundations for Empirical Research, 2) Survey of Research Designs, 3) Ethnography, and 4) Working with Industry & Communicating Results.  The invited speakers for the sessions were Marian Petre, Bill Curtis, Liam Bannon, and Dieter Rombach, respectively.  Three issues were raised repeatedly in different forms during the discussions through the course ofduring the workshop sessions. the day.  First, participants found the apparent lack paucity re was a dearth of education and training  iof software engineers in n empirical methods  in software engineering curriculum an impediment to research and communication of results.  Second, there was general agreement that multidisciplinary techniques must be understood on their own terms before they can be appropriately applied.  Finally, there were extensive discussions on the nature of evidence necessary to construct convincing arguments and the methods required to produce this evidence. 


 Over 50 people participated in the workshop.  The need for more extensive curriculum development, training and effective communication of results in empirical software engineering was identified.



Software engineering, empirical methods, multidisciplinary research.


1.     Background


In recent years there has been an increase in interest in applying empirical methods in studying software engineering empirically.  There are now several workshops such as WESS (Workshop on Empirical Studies of Software Maintenance)[1] and journals such as this one.  There are now several workshops (for example, the Workshop on Empirical Software Engineering (WESS)) and journals (such as EMSE) focusing on this topic. Increasingly, Consequently, researchers in software engineering are looking to other disciplines for techniques that can be applied within software engineering.  We organized this workshop, “Beg, Borrow or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research”, as we believe that other fields have much to contribute to the advancement of empirical software engineering.

 Theories and methods for empirical research have been developed by other disciplines over a long period of time. Just as it makes sense to re-use code within software engineering, it also makes sense to harness theories and methods from other fields to advance software engineering research.

For example, an empirical researcher interested in the use of patterns could learn from an anthropologist about how to conduct ethnographies and field studies.  Such knowledge, for example, could be used to observe how patterns affect the culture of a software engineering development environment. A psychologist may be able to further advise how to set up controls, analyze data, and write up any results.

As with all borrowing, no approach from another field can be applied wholesale without being adapted in some way. Researchers need to understand the benefits and drawbacks of different approaches and to identify techniques best suited to their theoretical leanings and the constraints on their work. If this understanding is not gainedachieved, the application of methods and theories from other disciplines may result in critical errors that would invalidate the research. Therefore, it is critical that we understand different approaches before applying them.   

We organized this workshop, “Beg, Borrow or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research,” to help researchers learn about methods from other disciplines through interaction with each other and guest speakers [2]. 

1.workshop goals


TThe goal of this workshop was to provide an interactive forum for software engineers and empirical researchers to investigate the feasibility of applying proven methods from other research disciplines to software engineering research. Expert guest speakers were invited to address the issues brought up with the hope of encouraging more multidisciplinary research.  The workshop was held as a single track to promote a stronger sense of community and increase knowledge exchange.

We were pleasantly surprised by the level of methodological sophistication in the position papers.  All the papers demonstrated a firm grasp basic research design issues.The papers submitted broached a broad range of research design issues.  The methods discussed in the papers went beyond simple controlled experiments, and included qualitative and quantitative techniques such as ethnography and numerical ecology.  The authorsy also drew on a wide range of theories to base their studies and interpret their data, including process modeling (Becker-Kornstaedt), use case models (Anda and Jørgensen), and discourse analysis (Sharp, Woodman, and Robinson).  The papers also posed subtle questions such as: How can distributed software development best be studied? (Rönkkö) How do you generalize requirements elicited using ethnography to other users?  (Underwood et al.) How do we study the design of secure software when security is a trade secret?  (Butler)

Section 2 describes the four sessions of presentation and discussion that were held during the workshop.  The common themes that arose over the course of the day are summarized in Section 3 and suggestions for future workshops are presented in Section 4. 



2.     workshop themesessionss


In the call for papers, Iinterested participants were invited to submit position papers describing problems in their current research that could or already were benefiting from multidisciplinary research. We received 33 submissions, from which 23 papers were selected to be included in accepted for the workshop.  The problems described related to some aspect of empirical software engineering, from research design to data analysis, from theory for understanding a phenomenon, to difficulty obtaining access to a target population.

The We used the position papers were used to guide athe  selection of guest speakers, who discussed how empirical methods from other disciplines could be applied within a software engineering context.  Invited speakers were selected for their expertise in dealing with problems similar to those discussed in the position papers.


 The selected papers were categorized according to four themes:

·         Foundations for Empirical Research

·         Survey of Research Designs

·         Ethnography

·         Working with Industry & Communicating Results

In the four sessions, a short presentation was given for each of the relevant position papers.  These presentations were followed by an invited guest presentation.  The invited speaker also led an open discussion on the issues identified in the papers.   The workshop concluded with a general discussion on the common themes identified throughout the day. 


The topics discussed in the four sessions are summarized below.  Further details on the schedule, the format of the workshop, links to the position papers and presentations given by the invited speakers are available on the workshop website at: Over 50 people participated in the workshop.




2.1     Foundations for Empirical Research


The papers presented in this session all addressed fundamental themes foundational issues for empirical software engineering, such as problem definition, a framework for empirical research programs, and theory formulationappropriate for defining the science of empirical research in software engineering. The papers in this track session were: 

·        Beg, Borrow, and Steal - But What, and What For? by Y. Dittrich.

·        Theory, Models and Methods in Software Engineering Research by R. Jeffery.

·        Beg, Borrow or Steal: OK but its not all One-Way Traffic! by G. McGrath & L. Uden.

·        Multidisciplinary Solutions for Multidisciplinary Problems by M. Wells & R. Harrison.

·        Borrow Fundamentals from Other Science and Engineering Disciplines by F. Xia.

This track of papers was addressed by invited guest speaker Marian Petre of the Open University, UK, and was followed by a general discussion opened to all participants.

In her presentation, Petre addressed a number of basic questions for empirical research, such as the purpose of empirical data, characteristics of a good research question, and the necessity of theory as a shared basis for investigation.  Empirical researchers in software engineering need to ask “why use a particular technique” not just “how can a technique be applied.”  The traditional dichotomy between qualitative and quantitative techniques is misleading because the distinction should be made in terms of the evidence required to answer a research question, e.g. how and why as opposed tovs. who and what.  Petre recommended these three steps to empirical study design:

1.          What is the question you need to answer?

2.          What evidence will satisfy you? Only then ask....

3.          What technique will provide appropriate evidence?

The discussion also extended to the types of evidence collected and differences in the analysis of this data, such as qualitative and quantitative methods.  As Petre commented: "Qualitative knowing is absolutely essential as a prerequisite foundation for quantification in science.  Without competence at the qualitative level, one's computer printout is misleading or meaningless."

In the discussion, one issue that arose in many forms was greater openness in empirical research.  In order for our results to be better understood and more widely accepted, we needed to have procedural and reporting standards, to increase In the led discussion, it was agreed that we should strive for more rigourrigor and integrity in our research practices, and to invite criticism of our data, interpretation, and theories.  This The increased rigor includesd more attention to ethics - making sure that we treat our research subjects justly and humanely.  In terms of borrowing, q

Questions were raised about different philosophical approaches and making sure that we understand what the underlying theory is before we blindly adopt an approach.  It was agreed that Ccare mustch be taken not to apply methods out of context.  Empirical researchers in software engineering need to ask “why use a particular technique” not just “how can a technique be applied”.   Marian Petre recommended these 3 steps to empirical study design:

1.What’s the question?

1.What evidence will satisfy you? Only then ask....

1.What technique will provide appropriate evidence?


The discussion also extended to the types of evidence collected and differences in the analysis of this data, such as qualitative and quantitative methods.    As Dr. Petre commented:  "Qualitative knowing is absolutely essential as a prerequisite foundation for quantification in science.  Without competence at the qualitative level, one's computer printout is misleading or meaningless."


2.2     3.2  Survey of Research Designs


This session was motivated by a set of accepted papers that gave an intriguing set of alternative research designs, including single-subject designs, numerical ecology, and process modeling. 

The papers presented in this session provided an overview of various research designs.  The papers in this track session includedwere: 

·        Knowledge Elicitation for Descriptive Software Process Modeling by U. Becker-Kornstaedt.

·        N = 1: An Alternative for Software Engineering Research? by W. Harrison.

·        Human Judgment in Effort Estimation of Software Projects by M. Jørgenson, D.I.K. Sjøberg, G. Kirkebøen, B. Anda & L. Bratthall.

·        How to Sstudy Individual Programmers? by M. Morisio.

·         Adapting and Extending Empirical Studies to the Global Software Process  by M. Lehman, J. F. Ramil & G. Kahen.

This set of papers was addressed by invited guest speaker Bill Curtis from TeraQuest Metrics Inc., U.S.A.  In his talk, Curtis reviewed some of the methods that he had used over the years in empirical studies.  He hasd studied how individual developers designed solutions using timed experiments.  But and videotaped meetings to study  when he studied design teams within an organization.  , he used performed a case study using videotapes of meetings. In other words,Curtis emphasized that different problems require different methods for investigation.  Furthermore, data can be very difficult to interpret because it is difficult to find reasonable units to quantify real-world phenomena .  Iand incidental differences, such as individual variability, can often overwhelm meaningful trends.

The subsequent discussion centered on issues relating to difficulties of measurement.  No discipline has developed an effective means for measuring intellectual artifacts.  This omission is particularly problematic in software engineering because its primary products are intellectual artifacts.  Furthermore, there needs to be a robust mapping between how a phenomenon is modeled using data and reality.  Another issue in measurement is the characterization of individual differences in software developers; we still don’t know what characteristics are important and how to measure them.  dealing with individual differences in software engineering, the importance of expert knowledge, Consequently, there are limits to both what we can control in empirical research and what we can learn from controlled research.  Finally, how to map models to reality, difficulties in measurement, the importance of continuous measurement, and the variations and limits of control in empirical research.   tThe importance of including empirical methods in software engineering education was also emphasized several times during the discussion led by Bill Curtis.  Few computer science graduates have learned even basic empirical methods, which has negative consequences for them both as practitioners interpreting studies and as researchers performing them  Need more about Bill Curtis’ presentation. . 



2.3     3.3   Ethnography


The papers presented in this session were all directly or indirectly related to ethnography.  Some of them reported on projects that used ethnographic methods and others reflected on the utility of these methods.  The papers presented were: 

·        Methodological Constraints in Researching Systems Development Work by E. Guy.

·        What are Software Practice Studies Good For? by O. Lindeberg., O.

·        A Real-world Approach to Real-world Research:  Appropriateness in Situation, Appropriateness in Method  by G. Mallalieu & H. Edwards.

·        Ethnography and Distributed Software Development by K. Rönkkö.

·        Using Ethnography and Discourse Analysis to study Software Engineering Practices by
H. Sharp, M. Woodman & H. Robinson.

·        Focussing User Studies: Requirements Capture for a Decision Support Tool by J. Underwood, R. Luckin, R. Cox, D. Watson & R. Tate.

These papers were addressed by invited guest speaker Liam Bannon from the University of Limerick, Ireland.    Liam Bannon gave an overview of ethnography and the importance of pluralism in methods applied.  In particular, he discussed findings from CSCW (cComputer sSupported cCo-operative wWork) research that are very relevant to many aspects of empirical software engineering research.  Social issues relating to work practices, group dynamics, discourse between group members, and political issues among group members are all very important and should not be ignored.  Bannon also criticized the emphasis on measurement in empirical research and warned us to be aware of the “McNamara Fallacy,” which proceeds according the following four stages:

·        The first step is to measure whatever can be easily measured. This is OK as far as it goes.

·        The second step is to disregard that which can’t be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading.

·        The third step is to presume that what can’t be measured easily really isn’t important. This is blindness.

·        The fourth step is to say that what can’t be easily measured really doesn’t exist. This is suicide.

The open discussion dealt with some of the unique challenges that ethnographic techniques pose for the researcher.  When using these techniques, the researcher needs to be highly sensitive to his or her role in the study because the researcher is the instrument for data collection.  It can also be more challenging to gain access to a site or a phenomenon with an ethnography because there are additional logistical and political issues.  Ethnography also has a set

Other issues identified during the led discussion included the importance of the researcher’s role, how to identify a good ethnography, what are the of intellectual underpinnings of a selected method, that need to be understood to use the technique effectively.  This knowledge is necessary for the researcher and the reader to be able tohow to get around problems of access (technical/logistical and political difficulties) identify a good ethnography. Once again, this information is missing , the apparent lack of education and training in research methods, and difficulties in reconstructing software engineering environments to be studied. from software engineering education. 


2.4     3.4  Working with Industry and Communicating Results


The papers presented in this session either presented empirical research being done with industry or they discussed difficulties encountered trying to proposed means for effectively communicatinge empirical results to industry.  The papers in this final session included: 

·        Understanding Use Case Models  by B. Anda & M. Jörgensen.

·        Aspects of Large-Scale Process Studies in Industrial Settings by A. Birk.

·        Software Design: Why it is Hard to Do Empirical Research by S. Butler.

·        Success Factors for Software Experience Bases:  What We Need to Learn from Other Disciplines by R. Conradi, M. Lindvall & C. Seaman.

·        Improving Information System Success using Cooperative and Multidisciplinary Development Techniques by M. Maritz & R. Harrison.

·        Learning and Instructional Issues in Software Engineering by M. Murphy & R. Halstead-Nussoch.

·        Improving Software Technology Transfer: Making Decisions based on Evidence by W.  Menezes & S. Lawrence-L. Pfleeger.

This set of papers was addressed by invited guest speaker Dieter Rombach from the Fraunhofer IESE and, University of Kaiserslautern, in Germany.  Rombach addressed each of the papers in this session along six dimensions: its view of the nature of software engineering; use of empirical studies; inspirations taken from other fields; the topic being studied; level of industry interest    Dieter Rombach started his presentation by discussing some general issues as follows:  What is the nature of software engineering? What is the role of empirical studies in software engineering?  What range of empirical studies are possible (consider inspirations from other fields)?  What are the topics of study?  How much interest is there from industry?  And what difficulties are encountered in the communication of results?  Professor Rombach summarized the key points in each of the session papers using these themes.    on the topic; and approach for communicating results.  These six dimensions were a powerful tool for analysis and provided a new level of insights into the studies presented.  Rombach felt that empirical studies were critical to software engineering because

More…… finish with this quote perhaps: 

As Professor Rombach stated Eexperimentation is prototyping for process engineering.. 

During the discussion, we identified a number of factors that were critical to working with industry.  First, building trust and good working relationships is essential to gaining access to a field site.  These sites allow the researcher to tackle a problem in a meaningful context that includes constraints.  Second, the researcher must build a business case to justify the study.  Finally, the arguments must be constructed based on evidence that is relevant to the target audience.  Executives and scientists have different backgrounds and work on different reward systems.  Consequently, they have different ideas of which arguments have merit. 



3.     common themes



Throughout the day, several three common themes became obviousissues were raised repeatedly in different forms.  They were education, appropriate application of multidisciplinary techniques, and the nature of evidence.


In general there is a lack of education in empirical methods in software engineering and yet there is clearly a need to apply these methods in many facets of software engineering.  Software engineers and researchers have different training requirements with respect to empirical methods.  Software engineers need to be able read studies critically to determine whether a new technique or tool should be applied to their organizations.  Researchers need to able to design studies that produce clear results and to assimilate new methods.  This training should be included early in the curriculum because it entails an entire philosophy of software engineering as an empirical discipline, rather than a discipline that proceeds by trial-and-error or rules of thumb.  One notable exception to this trend is the work of Rich Halstead-Nussloch and Mike Murphy at Southern Polytechnic State University in Georgia, USA, where they are in developing curriculum and a text book for teaching empirical software engineering at the undergraduate level. 

  (I think Dieter mentioned an exception to this in Germany?  Someone else also mentioned it was done at their school, was it Warren Harrison?).

From Janice’s slides (need some prose here):

·Curriculum, education, training, communicate results


oSoftware engineers

·Philosophy of empiricism


Appropriate Application of Multidisciplinary techniques

Bannon introduced the term “pluridisciplinary” in his talk and he uses it to denote inter- and multi-disciplinary techniques.  Collaboration can occur between and across disciplines and this cross-fertilization can take place among more than two disciplines.  When borrowing techniques and theories from other disciplines, we must do so responsibly and respectfully.  We need to understand this intellectual collateral in its own right, before subverting them to our own purposes.  Moreover, the results from software engineering ought to be of sufficiently high quality that they make a contribution to the field from which they borrowed techniques and theories. 


Nature of Evidence

The nature of sufficient and convincing evidence was discussed extensively throughout the day.  Both quantitative and qualitative evidence are necessary for driving research.  Petre recommended using the nature of evidence required as a basis for designing a study.  Curtis discussed the difficulty of producing meaningful measurements.  Bannon pointed out the fallacy of measuring inappropriately.  Rombach stressed the importance of evidence as the basis for arguments recommending software practices and for building a business case to justify research.  Good evidence needs to be replicable and testing, thus supporting assessment of its credibility.  This evidence must be packaged with a technical innovation to encourage adoption by industry.  In summary, evidence is the raison d’être for empirical research.


4.     Future work


One day was simply not enough time to broach all of the issues that were identified in the position papers and that were brought up during the course of the day.  There were many suggestions for future workshops, for example:

·        empirical studies across communities,,

·        theory of software development,,

·        philosophy of empiricism, and

·        good practices.

In addition, the software engineering community would benefit from more training and tutorials that focus on particular methods that are being borrowed from other disciplines.  Typically software engineering education does not include empirical research as part of its curriculum. 

We plan to hold a similar workshop to We feel that it would be beneficial to hold another “Beg, Borrow or Steal” in 2002.  However, we will recommend schedulinge two days for this workshop and includinge some tutorial-like”  sessions, as well as some free form working group discussions.






We would like to thank the other members of the program committee for reviewing the papers submitted to this workshop:

·        Steve Easterbrook, University of Toronto

·        Rachel Harrison, University of Reading UK

·        Juan Ramil, Imperial College UK

·        Jarrett Rosenberg, Sun Microsystems

·        Carolyn Seaman, University of Maryland,

·        Baltimore County USA

·        Harvey Siy, Lucent Technologies

·        Norman Vinson, NRC, Canada


In addition, the ICSE workshop chairs, Gail Murphy and Antonia Bertolino, were very supportive of our workshop. 


The Beg, Borrow and Steal workshop resulted from a previous workshop on Empirical Studies of Software Maintenance (WESS 99).   We (Susan Sim, Margaret-Anne Storey, and Janice Singer) were lucky enough to participate in a session on methods for studying software engineering. Also participating were Juan Ramil, Carolyn Seaman, Nicholas Zvegintzov, Martin Sheppaerd, with Jarrett Rosenberg acting as our leader. During this session, we discussed at some length the methodological and theoretical knowledge that other fields could contribute to empirical studies of software engineering. The results of this session can be seen at:

Finally, we would like to thank Anke Weber from the University of Victoria for support with the web page and scheduling arrangements.




[1]     Martin Shepperd. (ed) “Fifth IEEE Workshop on Empirical Studies of Software Maintenance (WESS99),” 3-4 September , 1999, Oxford, UK.  See also .

[2]     Janice Singer, Margaret-Anne Storey, and Susan Elliott Sim.(eds) “Beg, Borrow, or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research,5 June, 2000, Twenty-Second International Conference on Software Engineering, Limerick, Ireland.  See also .

Are references needed?  I don’t have our submission for ICSE (which I think did include references).

If we need references, perhaps we can pull them from there.  Could also have the WESS website and our website as references.