Beg, Borrow, or Steal: Using Multidisciplinary Approaches in
Empirical Software Engineering Research
An ICSE ’2000 Workshop Report
Limerick, Ireland, 5 June
2000
|
Susan Elliott Sim Department of Computer Science University of Toronto 10 Kings College Rd. Toronto, Ontario Canada M5S 3G4 Tel. +1 (416) 978-4158 Fax. +1 (416) 978-4765 simsuz@cs.utoronto.ca |
Dr. Janice Singer National Research Council Canada Bldg M-50, Rm. 257 Montreal Rd. Ottawa, ON Canada K1A 0R6 Tel: +1 (613) 991-6346 Fax: +1 (613) 952-7151 janice.singer@iit.nrc.ca |
Dr. Margaret-Anne Storey Assistant Professor Department of Computer Science University of Victoria PO Box 3055 STN CSC Victoria, BC Canada V8W 3P6 Tel: +1 (250) 721 8796 Fax: +1 (250) 721 7292 mstorey@uvic.ca |
Susan
Elliott Sim Janice Singer Margaret-Anne
Storey
Dept.
of Computer Science National
Research Council Dept.
of Computer Science
University
of Toronto Bldg. M-50, Rm
257 Montreal Rd University
of Victoria
10
Kings College Rd., Toronto Montreal Rd.Ottawa, ON PO
Box 3055 STN CSC
Toronto, ON, Ottawa,
ON Victoria,
BC
Canada M5S 3G4 Canada
K1A 0R6 Victoria, BC Canada V8W
3P6
+1
(416) 978 4158 +1
(613) 991 6346 +1
(250) 721 8796
simsuz@cs.utoronto.ca
janice.singer@iit.nrc.ca mstorey@uvic.ca
Executive
summary
The
goal of this workshop was to provide an interactive forum for software
engineers and empirical researchers to investigate the feasibility of applying
proven methods from other research disciplines to software engineering
research. In
the call for papers, Pparticipants
were
invited to submitted position papers describing
problems that might benefit from a multidisciplinary approach. Expert guest
speakers from software engineering and other disciplines addressed the issues
highlighted in the papers with the goal of encouraging more multidisciplinary
research. Papers accepted for the workshop were grouped into
four sessions based on similarity.
These sessions were: 1) Foundations for Empirical Research, 2) Survey of
Research Designs, 3) Ethnography, and 4) Working with Industry &
Communicating Results. The invited speakers for the sessions were Marian
Petre, Bill Curtis, Liam Bannon, and Dieter Rombach, respectively. Three
issues were raised repeatedly in different forms during the
discussions through the course ofduring the
workshop sessions. the day. First, participants
found the apparent lack paucity re was a dearth of education
and training iof software engineers in n empirical
methods in
software engineering curriculum an impediment to
research and communication of results. Second, there was general agreement that multidisciplinary
techniques must be
understood on their own terms before they can be appropriately applied. Finally, there were
extensive discussions on the nature of evidence necessary to construct
convincing arguments and the methods required to produce this evidence.
Over 50 people participated in the
workshop. The need for more extensive curriculum development,
training and effective communication of results in empirical software
engineering was identified.
Software
engineering, empirical methods, multidisciplinary research.
In
recent years there has been an increase in interest in applying
empirical methods in studying software engineering empirically. There are now several workshops such as WESS
(Workshop on Empirical Studies of Software Maintenance)[1] and journals
such as this one. There are now several workshops (for example,
the Workshop on Empirical Software
Engineering (WESS)) and journals (such as EMSE) focusing on this topic. Increasingly, Consequently, researchers
in software engineering are looking to other disciplines for techniques that
can be applied within software engineering. We organized this workshop, “Beg, Borrow or
Steal: Using Multidisciplinary Approaches in Empirical Software Engineering
Research”, as we believe that other fields have much to contribute to the
advancement of empirical software engineering.
Theories and methods for empirical
research have been developed by other disciplines over a long period of time.
Just as it makes sense to re-use code within software engineering, it also
makes sense to harness theories and methods from other fields to advance
software engineering research.
For
example, an empirical researcher interested in the use of patterns could learn
from an anthropologist about how to conduct ethnographies and field
studies. Such knowledge, for example,
could be used to observe how patterns affect the culture of a software
engineering development environment. A psychologist may be able to further
advise how to set up controls, analyze data, and write up any results.
As
with all borrowing, no approach from another field can be applied wholesale
without being adapted in some way. Researchers need to understand the benefits
and drawbacks of different approaches and to identify techniques best suited to
their theoretical leanings and the constraints on their work. If this
understanding is not gainedachieved,
the application of methods and theories from other disciplines may result in
critical errors that would invalidate the research. Therefore, it is critical
that we understand different approaches before applying them.
We organized
this workshop, “Beg, Borrow or Steal: Using Multidisciplinary Approaches in
Empirical Software Engineering Research,” to help
researchers learn about methods from other disciplines through interaction with
each other and guest speakers [2].
TThe goal of this workshop was to provide an
interactive forum for software engineers and empirical researchers to
investigate the feasibility of applying proven methods from other research
disciplines to software engineering research. Expert guest speakers were
invited to address the issues brought up with the hope of encouraging more
multidisciplinary research. The
workshop was held as a single track to promote a stronger sense of community
and increase knowledge exchange.
We were pleasantly surprised by the level of
methodological sophistication in the position papers. All the papers demonstrated a firm grasp basic research design
issues.The papers submitted broached a broad range of
research design issues. The
methods discussed in the papers went beyond
simple controlled experiments, and included qualitative and
quantitative techniques such as
ethnography and numerical ecology. The authorsy also drew on a wide range of theories to base their studies and interpret their data, including process modeling (Becker-Kornstaedt), use case
models (Anda and Jørgensen), and discourse
analysis (Sharp, Woodman, and Robinson). The papers
also posed subtle questions such as: How can distributed software development
best be studied? (Rönkkö) How do you generalize
requirements elicited using ethnography to
other users? (Underwood et al.) How do we study the design of secure software when security is a trade secret? (Butler)
Section 2
describes the four sessions of presentation and discussion that were held
during the workshop. The common themes
that arose over the course of the day are summarized in Section 3 and
suggestions for future workshops are presented in Section 4.
In the call for
papers, Iinterested participants were invited to
submit position papers describing problems in their current research that could
or already were benefiting from multidisciplinary research. We received 33
submissions, from which 23 papers were selected to be
included in accepted for the workshop. The problems described related to some
aspect of empirical software engineering, from research design to data
analysis, from theory for understanding a phenomenon, to difficulty obtaining
access to a target population.
The We used the position
papers were used to guide athe selection
of guest speakers,
who discussed how empirical methods from other disciplines could be applied
within a software engineering context. Invited
speakers were selected for their expertise in dealing with problems similar to
those discussed in the position papers.
The selected papers were categorized
according to four themes:
·
Foundations
for Empirical Research
·
Survey
of Research Designs
·
Ethnography
·
Working
with Industry & Communicating Results
In the four sessions, a short presentation was given for each of the relevant position papers. These presentations were followed by an invited guest presentation. The invited speaker also led an open discussion on the issues identified in the papers. The workshop concluded with a general discussion on the common themes identified throughout the day.
The
topics discussed in the four sessions are summarized below. Further details on the schedule, the format
of the workshop, links to the position papers and presentations given by the
invited speakers are available on the workshop website at: http://www.csr.UVic.CAuvic.ca/icse2000/. Over 50 people
participated in the workshop.
The
papers presented in this session all addressed
fundamental themes foundational
issues for empirical software
engineering, such as problem definition, a framework for
empirical research programs, and theory formulationappropriate for
defining the science of empirical research in software engineering.
The papers in this track session were:
·
Beg, Borrow, and Steal - But
What, and What For? by Y. Dittrich.
·
Theory, Models and Methods
in Software Engineering Research by R. Jeffery.
·
Beg, Borrow or Steal: OK but
its not all One-Way Traffic! by G. McGrath & L.
Uden.
·
Multidisciplinary Solutions
for Multidisciplinary Problems by M. Wells
& R. Harrison.
·
Borrow Fundamentals from
Other Science and Engineering Disciplines by F. Xia.
This track of papers was addressed by invited guest
speaker Marian Petre of the Open
University, UK, and was followed by a general discussion opened to all
participants.
In her presentation, Petre addressed a
number of basic questions for empirical research, such as the purpose of
empirical data, characteristics of a good research
question, and the necessity of theory as a shared
basis for investigation. Empirical
researchers in software engineering need to ask “why use a particular
technique” not just “how can a technique be applied.” The traditional dichotomy between qualitative and
quantitative techniques is misleading
because the distinction should be made in terms of the evidence required to
answer a research
question, e.g. “how and why” as opposed
tovs. “who and what”. Petre recommended these three steps to empirical
study design:
1.
What i’s
the question you need to answer?
2.
What evidence
will satisfy you? Only then ask....
3.
What technique
will provide appropriate evidence?
The discussion also extended to the types of evidence collected and differences in the analysis of this data, such as qualitative and quantitative methods. As Petre commented: "Qualitative knowing is absolutely essential as a prerequisite foundation for quantification in science. Without competence at the qualitative level, one's computer printout is misleading or meaningless."
In the discussion, one issue that
arose in many forms was greater openness in empirical research. In order for our results to be better understood
and more widely accepted, we needed to have procedural and
reporting standards, to increase In the led discussion, it
was agreed that we should strive for more rigourrigor and
integrity in our research practices, and to invite criticism of
our data, interpretation, and theories. This The increased
rigor includesd
more attention to ethics— - making
sure that we treat our research subjects justly and humanely. In terms of borrowing, q
Questions were raised about different philosophical
approaches and making sure that we understand what the underlying theory is
before we blindly adopt an approach. It was agreed
that Ccare mustch
be taken not to apply methods out of context.
Empirical researchers in software engineering need
to ask “why use a particular technique” not just “how can a technique be
applied”. Marian Petre recommended these 3 steps to
empirical study design:
1.What’s the
question?
1.What evidence
will satisfy you? Only then ask....
1.What technique
will provide appropriate evidence?
The discussion also extended to the types of
evidence collected and differences in the analysis of this data, such as
qualitative and quantitative methods. As Dr. Petre commented: "Qualitative
knowing is absolutely essential as a prerequisite foundation for quantification
in science. Without competence at the
qualitative level, one's computer printout is misleading or meaningless."
This session
was motivated
by a
set of accepted papers that gave an intriguing set of alternative research
designs, including single-subject designs, numerical ecology, and process
modeling.
The papers presented in this session provided an
overview of various research designs. The
papers in this track session includedwere:
·
Knowledge Elicitation for
Descriptive Software Process Modeling by U. Becker-Kornstaedt.
·
N = 1: An Alternative for
Software Engineering Research? by W. Harrison.
·
Human Judgment in Effort Estimation
of Software Projects by M. Jørgenson, D.I.K.
Sjøberg, G. Kirkebøen, B. Anda & L. Bratthall.
·
How to Sstudy Individual Programmers? by M. Morisio.
·
Adapting and Extending
Empirical Studies to the Global Software Process by M. Lehman, J. F. Ramil & G. Kahen.
This
set of papers was addressed by invited guest speaker Bill Curtis from TeraQuest Metrics Inc., U.S.A. In his talk, Curtis reviewed some of the methods
that he had used over the years in empirical studies. He hasd studied how individual developers designed solutions
using timed experiments. But and
videotaped meetings to study when he studied design
teams within an organization. , he used performed a case
study using videotapes of meetings. In other words,Curtis emphasized that different problems require
different methods for investigation. Furthermore,
data can be very difficult to interpret because it is difficult to find
reasonable units to quantify real-world phenomena .
Iand incidental
differences, such as individual variability, can often overwhelm meaningful
trends.
The subsequent discussion
centered on issues relating to difficulties of measurement. No discipline has developed an effective
means for measuring intellectual artifacts.
This omission is particularly problematic in software engineering
because its primary products are intellectual artifacts. Furthermore, there needs to be a robust mapping
between how a phenomenon is modeled using data and reality. Another issue in measurement is the
characterization of individual differences in software developers; we still don’t know what characteristics are important and how to measure
them.
dealing with individual differences in software
engineering, the importance
of expert knowledge, Consequently, there are limits to both what we can
control in empirical research and what we can learn from controlled
research. Finally, how to map models to reality, difficulties in measurement, the importance of continuous measurement, and the variations and limits of control in empirical
research. tThe importance of including empirical
methods in software engineering education was also emphasized several times during the
discussion led by Bill Curtis. Few
computer science graduates have learned even basic empirical methods, which has negative
consequences for them both as practitioners interpreting studies and as researchers
performing them Need more about Bill Curtis’ presentation. .
The
papers presented in this session were all directly or indirectly related
to ethnography. Some of them
reported on projects that used ethnographic methods and others reflected on the
utility of these methods. The
papers presented were:
·
Methodological Constraints
in Researching Systems Development Work by E. Guy.
·
What are Software Practice
Studies Good For? by O.
Lindeberg., O.
·
A Real-world Approach to
Real-world Research: Appropriateness in
Situation, Appropriateness in Method by G. Mallalieu & H.
Edwards.
·
Ethnography and Distributed
Software Development by K. Rönkkö.
·
Using Ethnography and
Discourse Analysis to study Software Engineering Practices by
H. Sharp, M. Woodman & H.
Robinson.
·
Focussing User Studies:
Requirements Capture for a Decision Support Tool by J. Underwood, R. Luckin, R. Cox, D. Watson & R. Tate.
These
papers were addressed by invited guest speaker Liam Bannon from the University
of Limerick, Ireland. Liam Bannon gave an overview of
ethnography and the importance of pluralism in methods applied. In particular, he
discussed findings from CSCW (cComputer sSupported cCo-operative
wWork) research that are very relevant to many
aspects of empirical software engineering research. Social issues relating to work practices, group dynamics,
discourse between group members, and political issues among group members are
all very important and should not be ignored.
Bannon
also criticized the emphasis on measurement in empirical research and warned us
to be aware of the “McNamara Fallacy,” which proceeds according the following four
stages:
·
“The first step is
to measure whatever can be easily measured. This is OK as far as it goes.
·
The
second step is to disregard
that which can’t be easily measured or to give it an arbitrary
quantitative value. This is artificial and misleading.
·
The
third step is to presume that what can’t be measured easily really isn’t important.
This is blindness.
·
The
fourth step is to say that what can’t be easily measured really doesn’t exist.
This is suicide.”
The open discussion
dealt with some of the unique challenges that ethnographic
techniques pose for the researcher.
When using these techniques, the researcher needs to be highly sensitive to
his or her role in the study because the researcher is the instrument
for data collection. It can also be
more challenging to gain access to a site or a phenomenon with an ethnography
because there are additional logistical and political issues. Ethnography also has a set
Other issues identified
during the led discussion included the importance
of the researcher’s role, how to identify a good ethnography, what are the of intellectual
underpinnings of a selected method, that need to be
understood to use the technique effectively.
This knowledge is necessary for the researcher and the reader to be able
tohow to get around problems of access
(technical/logistical and political difficulties) identify a
good ethnography. Once again, this information is missing , the apparent lack of education and training in
research methods, and difficulties in reconstructing software engineering
environments to be studied. from software engineering education.
The
papers presented in this session either presented empirical research being done
with industry or they discussed difficulties encountered trying to proposed means
for effectively
communicatinge
empirical results to industry. The papers in this final session included:
·
Understanding Use Case
Models by B. Anda & M. Jörgensen.
·
Aspects of Large-Scale
Process Studies in Industrial Settings by A. Birk.
·
Software Design: Why it is
Hard to Do Empirical Research by S. Butler.
·
Success Factors for Software
Experience Bases: What We Need to Learn
from Other Disciplines by R. Conradi, M. Lindvall & C. Seaman.
·
Improving Information System
Success using Cooperative and Multidisciplinary Development Techniques by M. Maritz & R. Harrison.
·
Learning and Instructional
Issues in Software Engineering by M. Murphy & R. Halstead-Nussoch.
·
Improving Software
Technology Transfer: Making Decisions based on Evidence by
W.
Menezes & S. Lawrence-L. Pfleeger.
This
set of papers was addressed by invited guest speaker Dieter Rombach from the Fraunhofer IESE and, University of Kaiserslautern, in Germany. Rombach addressed each of the
papers in this session along six dimensions: its view of the nature of software
engineering; use of empirical studies; inspirations taken
from other fields; the topic being studied; level of
industry interest Dieter Rombach started
his presentation by discussing some general issues as follows: What is the nature of software engineering?
What is the role of empirical studies in software engineering? What range of empirical studies are possible
(consider inspirations from other fields)?
What are the topics of study?
How much interest is there from industry? And what difficulties are encountered in the communication of
results? Professor Rombach summarized the key points in each of the
session papers using these themes. on the topic; and approach for
communicating results. These six dimensions were
a powerful tool for analysis and provided a new level of insights into the
studies presented. Rombach felt
that empirical studies were critical to software engineering because
More…… finish with this quote perhaps:
As Professor
Rombach stated “Eexperimentation
is prototyping for process engineering.”.
During the
discussion, we identified a number of factors that were critical to working with
industry. First, building trust
and good working relationships is essential to gaining access to a field
site. These sites allow the researcher
to tackle a problem in a meaningful context that includes constraints. Second, the researcher must build a business
case to
justify the study. Finally, the
arguments must be constructed based on evidence that is
relevant to the target audience. Executives and
scientists have different backgrounds and work on different
reward systems. Consequently, they have
different ideas of which arguments have merit.
Throughout the day, several three common themes became
obviousissues were raised repeatedly in different forms. They were education, appropriate application of multidisciplinary
techniques,
and the nature of evidence.
Education
In
general there is a lack of education in empirical methods in software engineering
and yet there is clearly a need to apply these methods in many facets of
software engineering. Software
engineers and researchers have different training requirements with respect to
empirical methods. Software
engineers need to be able read studies critically to determine whether a new
technique or tool should be applied to their organizations. Researchers need to able to
design studies that produce clear results and to assimilate new methods. This training should be included early in the
curriculum because it entails an entire philosophy of software engineering as
an empirical discipline, rather than a discipline that proceeds by
trial-and-error or rules of thumb. One notable
exception to this trend is the work of Rich Halstead-Nussloch and Mike Murphy at Southern Polytechnic State University in
Georgia, USA, where they are in developing
curriculum and a text book for teaching empirical software engineering at the
undergraduate level.
(I think Dieter mentioned an exception to this in Germany? Someone else also mentioned it was done at
their school, was it Warren Harrison?).
From Janice’s slides (need some prose here):
·Curriculum,
education, training, communicate results
oResearchers
oSoftware
engineers
·Philosophy of
empiricism
Appropriate
Application of Multidisciplinary
techniques
Bannon introduced the term “pluridisciplinary” in his talk and he uses it to denote inter- and multi-disciplinary techniques. Collaboration can occur between and across disciplines and this cross-fertilization can take place among more than two disciplines. When borrowing techniques and theories from other disciplines, we must do so responsibly and respectfully. We need to understand this intellectual collateral in its own right, before subverting them to our own purposes. Moreover, the results from software engineering ought to be of sufficiently high quality that they make a contribution to the field from which they borrowed techniques and theories.
Nature of Evidence
The nature of
sufficient and convincing evidence was discussed extensively throughout the
day. Both
quantitative and qualitative evidence are necessary for driving research. Petre recommended using the nature of evidence
required as a basis for designing a study. Curtis
discussed the difficulty of producing meaningful measurements. Bannon pointed out the
fallacy of measuring inappropriately.
Rombach stressed the importance of evidence as the basis
for arguments recommending software practices and for building a
business case to justify research. Good evidence
needs to be replicable and testing, thus supporting assessment of its credibility. This evidence must be
packaged with a technical innovation to encourage adoption by industry. In summary, evidence is the raison d’être for empirical
research.
One day was simply not enough time to broach all of the issues that were identified in the position papers and that were brought up during the course of the day. There were many suggestions for future workshops, for example:
·
empirical studies across communities,,
·
theory of software development,,
· philosophy of empiricism, and
· good practices.
In addition, the software engineering community would benefit from more training and tutorials that focus on particular methods that are being borrowed from other disciplines. Typically software engineering education does not include empirical research as part of its curriculum.
We plan to hold
a similar workshop to We feel that it would be beneficial
to hold another “Beg, Borrow or Steal” in 2002. However, we will recommend schedulinge two days for this workshop and
includinge some “tutorial-like” sessions, as well as some free form working group discussions.
AcknowledgEments
We
would like to thank the other members of the program committee for reviewing
the papers submitted to this workshop:
·
Steve
Easterbrook, University of Toronto
·
Rachel
Harrison, University of Reading UK
·
Juan
Ramil, Imperial College UK
·
Jarrett
Rosenberg, Sun Microsystems
·
Carolyn
Seaman, University of Maryland,
·
Baltimore
County USA
·
Harvey
Siy, Lucent Technologies
·
Norman
Vinson, NRC, Canada
In
addition, the ICSE workshop chairs, Gail Murphy and Antonia Bertolino, were very
supportive of our workshop.
The Beg, Borrow and Steal workshop resulted from a
previous workshop on Empirical Studies of Software Maintenance (WESS 99). We (Susan Sim, Margaret-Anne Storey, and Janice Singer) were lucky
enough to participate in a session on methods for studying software
engineering. Also participating were Juan Ramil, Carolyn Seaman, Nicholas
Zvegintzov, Martin Sheppaerd, with Jarrett
Rosenberg acting as our leader. During this session, we discussed at some
length the methodological and theoretical knowledge that other fields could
contribute to empirical studies of software engineering. The results of this
session can be seen at:
http://dec.bmth.ac.uk/ESERG/WESS99WGC.html.
Finally,
we would like to thank Anke Weber from the University of Victoria for support
with the web page and scheduling arrangements.
REFERENCES
[1]
Martin
Shepperd. (ed) “Fifth IEEE Workshop on Empirical Studies of Software
Maintenance (WESS99),” 3-4 September , 1999, Oxford, UK. See also http://dec.bmth.ac.uk/ESERG/WESS99WGC.html
.
[2]
Janice Singer,
Margaret-Anne Storey, and Susan Elliott Sim.(eds) “Beg, Borrow,
or Steal: Using Multidisciplinary Approaches in Empirical Software
Engineering Research,” 5 June, 2000, Twenty-Second International Conference
on Software Engineering, Limerick, Ireland. See also http://www.csr.uvic.ca/icse2000/
.
Are references needed? I don’t have our submission for ICSE (which I think did include
references).
If we need references, perhaps we can pull them
from there. Could also have the WESS
website and our website as references.