Course syllabus

Jump to today

Welcome to the course DAT246/DIT246 Empirical Software Engineering. This home page contains all the important information about the course, the learning objectives, the schedule, and more. Carefully review this information, and do not hesitate to make use of the "Discussions"-tab here on Canvas to exchange both with us and among each other. 

Contact details

In case of questions, contact the examiner and teaching assistant via email using your official student email addresses. We will remain responsive during working hours (ca. 8:00-17:00 during working days) and try to get back to you as soon as possible.

Student Representatives

Student representatives will be announced once they have been decided.

Structure

The course consists of two parallel tracks:

  • Empirical Research Methods teaches you how to design and evaluate empirical studies for specific research goals.
  • Bayesian Data Analysis teaches you how to analyze quantitative data and arrive at valid conclusions.

Schedule

Each week will consist of 2 lectures and 1 lab session. The course is designed as follows:

  • Read the Chapters of the course literature (see below) and the Recommended reading before the lectures.
  • Attend the Lectures. If anything is unclear, review the Lecture Notes.
  • Attend the Lab session and work on the Exercises of the respective week.

You find all lecture slides, lecture notes, and exercises in the modules tab, indexed with the course week.

The following table is a projection of the real times that might get outdated. The single source of truth for the lecture and exercise dates can be found via TimeEdit.

Empirical Research Methods Bayesian Data Analysis
Week Content Recommended reading Lecture Content Chapters Lecture / Lab Ex
1 Building a Case Stol: The ABC of SE Research1  Sep 01, 13:15-15:00 Introduction 1—3 Sep 02, 10:15-12:00 /
Sep 02, 13:15-15:00		 E1
2 Research Ethics Sep 07, 10:15-12:00 Mathematical notation for model specification 4—6 Sep 09, 10:15-12:00 /
Sep 09, 13:15-15:00		 E2
3 Threats to Validity Feldt: Validity Threats in Empirical SE Research Sep 14, 10:15-12:00 Information theory and maximum entropy 7 Sep 16, 10:15-12:00 /
Sep 16, 13:15-15:00		 E3
4 Controlled Experiments Sep 21, 10:15-12:00 Interaction effects and Markov chain Monte Carlo sampling 8—9 Sep 23, 10:15-12:00 /
Sep 23, 13:15-15:00		 E4
5 Case  Studies Runeson: Guidelines for Conducting and Reporting Case Study Research in SE1  Sep 28, 10:15-12:00 Generalized Linear Models 10—11 Sep 30, 10:15-12:00 /
Sep 30, 13:15-15:00		 E5
6 Survey Research Ghazi: Survey Research in SE1  Oct 05, 10:15-12:00 Over-dispersion, zero-inflation, and ordered categorical variables 12 Oct 07, 10:15-12:00 /
Oct 07, 13:15-15:00		 E6
7 Guest Lecture: Design Science Oct 12, 10:15-12:00 Multilevel Modeling  13 Oct 14, 10:15-12:00 /
Oct 14, 13:15-15:00		 E7
8 Guest Lecture: Action Research Oct 19, 10:15-12:00 Modeling covariance and continuous varying intercepts 14 Oct 21, 10:15-12:00 /
Oct 21, 13:15-15:00		 E8

The course is at 50% speed, and we, thus, expect you to spend 20h/week on it. You will need those hours, and the first two weeks will have a high workload. All lectures and labs will be on campus. We strongly recommend you to work out Exercise 1 as soon as possible since it's about setting up the environment you will need in this course! Furthermore, we will use the programming language R. If you are unfamiliar with it, review the setup guidelines.

The author of the book "Statistical Rethinking", Richard McElreath, also teaches in Leipzig and his lectures are available on YouTube. You can refer to them for additional insights.

Purpose and Learning Objectives

Course purpose

This course aims to learn scientific approaches, i.e., research methods, and statistics to analyze data we collect. The analysis can then become the basis for decision support in initiatives to improve performances in software development organizations. The course prepares students for the master thesis project.

Learning objectives and syllabus

Knowledge and understanding

  • Describe, understand, and apply empiricism in software engineering
  • Describe, understand, and partly apply the principles of case study research/experiments/surveys.
  • Describe and understand the underlying principles of meta-analytical studies.
  • Explain the importance of research ethics. 
  • Recognize and define codes of ethics when conducting research in software engineering.  
  • State and explain the importance of threats to validity and how to control said threats.
  • Describe and explain the concepts of probability space (incl. conditional probability), random variable, expected value, and random processes, and know a number of concrete examples of the concepts.
  • Describe Markov chain Monte Carlo methods such as Metropolis.
  • Describe and explain Hamiltonian Monte Carlo. 
  • Explain and describe multicollinearity, post-treatment bias, collider bias, and confounding
  • Describe and explain ways to avoid overfitting

Skills and abilities

  • Assess the suitability of and apply methods of analysis on data
  • Analyze descriptive statistics and decide on appropriate analysis methods.
  • Use and interpret code of ethics for software engineering research.
  • Design statistical models mathematically and implement said models in a programming language.
  • Make use of random processes, i.e., Bernoulli, Binomial, Gaussian, and Poisson distributions, with over-dispersed outcomes. 
  • Make use of ordered categorical outcomes (ordered-logit) and predictors
  • Assess the suitability of, from an ontological (natural process) and epistemological (maxent) perspective, various statistical distributions
  • Make use of and assess directed acyclic graphs to argue causality

Judgment and approach

  • State and discuss the tools used for data analysis and, in particular, judge their output.
  • Judge the appropriateness of particular empirical methods and their applicability to attack various and disparate software engineering problems.
  • Question and assess common ethical issues in software engineering research. 
  • Assess diagnostics from Hamiltonian Monte Carlo and quadratic approximation using information theoretical concepts, i.e., information entropy, WAIC, and PSIS-LOO.
  • Judge posterior probability distributions for out-of-sample predictions and conduct posterior predictive checks.

Course literature

This course will follow one book: R. McElreath. Statistical Rethinking: A Bayesian Course with Examples in R and STAN. 2nd edition. ISBN: 9780367139919. We will study Chapters 1-7, 9-14. Chapter 8 we will not emphasize much, while chapters 15-16 we will not cover in this course.
Course boo

The book uses R and Stan through the rethinking package to specify and sample statistical models. The first exercise (E1) provides instructions on how to set things up in Windows, OS X, and Linux.

Additionally, a number of papers will be used which you'll be able to find in the File area. We will clearly indicate if a paper is not compulsory reading, i.e., if we don't say anything, assume that it is compulsory reading.

Course design

Each week we will cover 1-3 chapters in the book. Before we meet we expect you to have read the chapters for that week and, in addition, gone through the videos connected to the chapters. The lectures that we will have will focus on three things:

  1. Covering the most important things from each chapter.
  2. Going through practical hands-on examples.
  3. Presenting research methods commonly used in empirical software engineering (i.e., things you will use for your master thesis).

I cannot stress enough how important it is that you a) read the chapter(s), and b) go through the videos connected to each chapter before each lecture. I also expect you to go to Canvas (i.e., this course home page) every day and check if things have been added or changed. If there are any changes to the lectures (e.g., if one would be canceled for some reason) then this will be notified above in the "Announcements" tab.

Examination

If you copy any text at all you must reference it appropriately - if in doubt ask. Plagiarism is something we do not look positively upon and each year students are suspended.

This course is examined in two components. First, a written exam at the end of the course. Second, an individual assignment during the course. If you pass the written exam you are given 5 credits. If you pass the individual assignment you are given 2.5 credits.

Grading and Preparation

Students are given the grades fail, 3, 4, or 5 in the course. In order to pass the course, you will need to pass both the assignment and the exam, but we set the final course grade only according to the grade you got on the written exam.

For the written exam all learning outcomes, as listed above, can be tested. For the individual assignment, there is mainly a focus on Bayesian data analysis (i.e., skills and abilities, and judgment and approach).

Additionally, we recommend students go through exercises as found in the modules area. Even though conducting these exercises will not give any extra credits or bonus points that can be added to the written exam or assignment, you will be in a much better place once you've done the exercises.

Deadlines

Dates for the written Exam (note that (1) these are centrally scheduled and potentially subject to change, and (2) you are responsible for signing up in time): 

  • First attempt: 2026-10-26
  • Second attempt: 2027-01-94
  • Third attempt: 2027-08-24

Assignment deadlines: Found for each assignment in the respective "Assignments" tab.

Course summary:

Course Summary
Date Details Due