Beg, Borrow, or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research
’2000 Workshop Report
Elliott Sim Janice Singer Margaret-Anne
of Computer Science National
Research Council Dept.
of Computer Science University
of Toronto Bldg M-50, Rm
257 Montreal Rd University
of Victoria 10
Kings College Rd , Toronto Ottawa, ON PO
Box 3055 STN CSC ON, Canada M5S 3G4 Canada
K1A 0R6 Victoria, BC Canada V8W
(416) 978 4158 +1
(613) 991 6346 +1
(250) 721 8796 firstname.lastname@example.org
goal of this workshop was to provide an interactive forum for software
engineers and empirical researchers to investigate the feasibility of applying
proven methods from other research disciplines to software engineering
submit ted position papers describing
problems that might benefit from a multidisciplinary approach. Expert guest
speakers from software engineering and other disciplines addressed the issues
highlighted in the papers with the goal of encouraging more multidisciplinary
Over 50 people participated in the
workshop. The need for more extensive curriculum development,
training and effective communication of results in empirical software
engineering was identified.
Software engineering, empirical methods, multidisciplinary research.
recent years there has been an increase in interest in
empirical methods in software engineering. There are now several workshops ( for example,
the Workshop on Empirical Software
Engineering (WESS) ) and journals ( such as EMSE ) focusing on this topic. Increasingly, researchers
in software engineering are looking to other disciplines for techniques that
can be applied within software engineering. We organized this workshop, “Beg, Borrow or
Steal: Using Multidisciplinary Approaches in Empirical Software Engineering
Research”, as we believe that other fields have much to contribute to the
advancement of empirical software engineering.
Theories and methods for empirical research have been developed by other disciplines over a long period of time. Just as it makes sense to re-use code within software engineering, it also makes sense to harness theories and methods from other fields to advance software engineering research.
For example, an empirical researcher interested in the use of patterns could learn from an anthropologist about how to conduct ethnographies and field studies. Such knowledge, for example, could be used to observe how patterns affect the culture of a software engineering development environment. A psychologist may be able to further advise how to set up controls, analyze data, and write up any results.
with all borrowing, no approach from another field can be applied wholesale
without being adapted in some way. Researchers need to understand the benefits
and drawbacks of different approaches and to identify techniques best suited to
their theoretical leanings and the constraints on their work. If this
understanding is not
the application of methods and theories from other disciplines may result in
critical errors that would invalidate the research. Therefore, it is critical
that we understand different approaches before applying them. The goal of this workshop was to provide an
interactive forum for software engineers and empirical researchers to
investigate the feasibility of applying proven methods from other research
disciplines to software engineering research. Expert guest speakers were
invited to address the issues brought up with the hope of encouraging more
multidisciplinary research. The
workshop was held as a single track to promote a stronger sense of community
and increase knowledge exchange.
Interested participants were invited to
submit position papers describing problems in their current research that could
or already were benefiting from multidisciplinary research. We received 33
submissions, from which 23 papers were selected to be
included in the workshop. The problems described related to some
aspect of empirical software engineering, from research design to data
analysis, from theory for understanding a phenomenon, to difficulty obtaining
access to a target population. The position
papers were used to guide a
The selected papers were categorized according to four themes:
· Foundations for Empirical Research
· Survey of Research Designs
· Working with Industry & Communicating Results
In the four sessions, a short presentation was given for each of the relevant position papers. These presentations were followed by an invited guest presentation. The invited speaker also led an open discussion on the issues identified in the papers. The workshop concluded with a general discussion on the common themes identified throughout the day.
topics discussed in the four sessions are summarized below. Further details on the schedule, the format
of the workshop, links to the position papers and presentations given by the
invited speakers are available on the workshop website at: http://www.csr.
papers presented in this session
fundamental themes appropriate for
defining the science of empirical research in software engineering.
The papers in this track were:
· Beg, Borrow, and Steal - But What, and What For? by Y. Dittrich.
· Theory, Models and Methods in Software Engineering Research by R. Jeffery.
· Beg, Borrow or Steal: OK but its not all One-Way Traffic! by G. McGrath & L. Uden.
· Multidisciplinary Solutions for Multidisciplinary Problems by M. Wells & R. Harrison.
· Borrow Fundamentals from Other Science and Engineering Disciplines by F. Xia.
This track of papers was addressed by invited guest speaker Marian Petre of the Open University, UK, and was followed by a general discussion opened to all participants.
In the led discussion, it
was agreed that we should strive for more rigour and
integrity in our research practices. This include d
more attention to ethics - making
sure that we treat our research subjects justly and humanely. Questions were raised about different philosophical
approaches and making sure that we understand what the underlying theory is
before we blindly adopt an approach. Care mu ch
be taken not to apply methods out of context.
Empirical researchers in software engineering need
to ask “why use a particular technique” not just “how can a technique be
applied” . Marian Petre recommended these 3 steps to
empirical study design: 1. What’s the
question? 1. What evidence
will satisfy you? Only then ask.... 1. What technique
will provide appropriate evidence? The discussion also extended to the types of
evidence collected and differences in the analysis of this data, such as
qualitative and quantitative methods. As Dr. Petre commented: "Qualitative
knowing is absolutely essential as a prerequisite foundation for quantification
in science. Without competence at the
qualitative level, one's computer printout is misleading or meaningless."
The papers presented in this session provided an
overview of various research designs. The
papers in this track included:
· Knowledge Elicitation for Descriptive Software Process Modeling by U. Becker-Kornstaedt.
· N = 1: An Alternative for Software Engineering Research? by W. Harrison.
· Human Judgment in Effort Estimation of Software Projects by M. Jørgenson, D.I.K. Sjøberg, G. Kirkebøen, B. Anda & L. Bratthall.
study Individual Programmers? by M. Morisio.
· Adapting and Extending Empirical Studies to the Global Software Process by M. Lehman, J. F. Ramil & G. Kahen.
This set of papers was addressed by invited guest speaker Bill Curtis from TeraQuest Metrics Inc., U.S.A.
The subsequent discussion
centered on issues relating to difficulties of
dealing with individual differences in software
engineering, the importance
of expert knowledge, how to map models to reality, difficulties in measurement, the importance of continuous measurement, and the variations and limits of control in empirical
research. The importance of including empirical
methods in software engineering education was also emphasized several times during the
discussion led by Bill Curtis. Need more about Bill Curtis’ presentation.
papers presented in this session were all directly
or indirectly related
to ethnography. The
papers presented were:
· Methodological Constraints in Researching Systems Development Work by E. Guy.
What are Software Practice
Studies Good For? by Lindeberg
A Real-world Approach to
Real-world Research: Appropriateness in
Situation, Appropriateness in Method
· Ethnography and Distributed Software Development by K. Rönkkö.
Using Ethnography and
Discourse Analysis to study Software Engineering Practices by
H. Sharp, M. Woodman & H. Robinson.
· Focussing User Studies: Requirements Capture for a Decision Support Tool by J. Underwood, R. Luckin, R. Cox, D. Watson & R. Tate.
papers were addressed by invited guest speaker Liam Bannon from the University
of Limerick, Ireland.
Liam Bannon gave an overview of
ethnography and the importance of pluralism in methods applied. In particular he
discussed findings from CSCW ( computer supported co-operative
work) research that are very relevant to many
aspects of empirical software engineering research. Social issues relating to work practices, group dynamics,
discourse between group members, and political issues among group members are
all very important and should not be ignored.
Other issues identified
during the led discussion included the importance
of the researcher’s role , how to identify a good ethnography, what are the intellectual
underpinnings of a selected method, how to get around problems of access
(technical/logistical and political difficulties) , the apparent lack of education and training in
research methods, and difficulties in reconstructing software engineering
environments to be studied.
papers presented in this session either presented empirical research being done
with industry or
they discussed difficulties encountered trying to communicat e
empirical results to industry. The papers in this final session included:
· Understanding Use Case Models by B. Anda & M. Jörgensen.
· Aspects of Large-Scale Process Studies in Industrial Settings by A. Birk.
· Software Design: Why it is Hard to Do Empirical Research by S. Butler.
· Success Factors for Software Experience Bases: What We Need to Learn from Other Disciplines by R. Conradi, M. Lindvall & C. Seaman.
· Improving Information System Success using Cooperative and Multidisciplinary Development Techniques by M. Maritz & R. Harrison.
· Learning and Instructional Issues in Software Engineering by M. Murphy & R. Halstead-Nussoch.
Technology Transfer: Making Decisions based on Evidence by
set of papers was addressed by invited guest speaker Dieter Rombach from the Fraunhofer IESE
, University of Kaiserslautern, in Germany. Dieter Rombach started
his presentation by discussing some general issues as follows: What is the nature of software engineering?
What is the role of empirical studies in software engineering? What range of empirical studies are possible
(consider inspirations from other fields)?
What are the topics of study?
How much interest is there from industry? And what difficulties are encountered in the communication of
results? Professor Rombach summarized the key points in each of the
session papers using these themes. More…… finish with this quote perhaps: As Professor
Rombach stated “ Experimentation
is prototyping for process engineering” .
Throughout the day,
several common themes became
In general there is a lack of education in empirical methods in software engineering and yet there is clearly a need to apply these methods in many facets of software engineering.
(I think Dieter mentioned an exception to this in Germany? Someone else also mentioned it was done at
their school, was it Warren Harrison?). From Janice’s slides (need some prose here): · Curriculum,
education, training, communicate results o Researchers o Software
engineers · Philosophy of
One day was simply not enough time to broach all of the issues that were identified in the position papers and that were brought up during the course of the day. There were many suggestions for future workshops, for example:
empirical studies across communities
theory of software development
· philosophy of empiricism, and
· good practices.
In addition, the software engineering community would benefit from more training and tutorials that focus on particular methods that are being borrowed from other disciplines. Typically software engineering education does not include empirical research as part of its curriculum.
We plan to hold
a similar workshop to “Beg, Borrow or Steal” in 2002. However, we will schedul e two days for this workshop and
includ e some “tutorial -like” sessions, as well as some free form working group discussions.
We would like to thank the other members of the program committee for reviewing the papers submitted to this workshop:
· Steve Easterbrook, University of Toronto
· Rachel Harrison, University of Reading UK
· Juan Ramil, Imperial College UK
· Jarrett Rosenberg, Sun Microsystems
· Carolyn Seaman, University of Maryland,
· Baltimore County USA
· Harvey Siy, Lucent Technologies
· Norman Vinson, NRC, Canada
In addition, the ICSE workshop chairs, Gail Murphy and Antonia Bertolino, were very supportive of our workshop.
The Beg, Borrow and Steal workshop resulted from a
previous workshop on Empirical Studies of Software Maintenance (WESS
ard, with Jarret
Rosenberg acting as our leader. During this session, we discussed at some
length the methodological and theoretical knowledge that other fields could
contribute to empirical studies of software engineering. The results of this
session can be seen at:
Finally, we would like to thank Anke Weber from the University of Victoria for support with the web page and scheduling arrangements.
Are references needed? I don’t have our submission for ICSE (which I think did include
references). If we need references, perhaps we can pull them
from there. Could also have the WESS
website and our website as references.