Evaluation 2: Formative Evaluation Methods for Adaptive Systems

Tutorial proceedings

Adaptive systems are by definition interactive systems. As such they stand to benefit considerably from a development lifecycle that ensures user involvement from the early design stages, and embraces evaluation, in both formative and summative forms.

The guiding perspective of this tutorial is that adequate evaluation methods, reliable assessment criteria and metrics are prerequisites for improving the quality and usability of the next generations of adaptive systems.

Upon successful completion of the tutorial, participants will

• Be aware of the specific problems involved in the evaluation of adaptive systems that differentiate them from their non-adaptive counterparts, and able to solve or circumvent these problems
• Understand and be able to apply the principles of layered evaluation of adaptive systems
• Be able to design a targeted formative evaluation study for an adaptive system (e.g., addressing a given layer and set of criteria) by selecting appropriate methods and criteria

Prerequisites

We recommend that participants who have no prior knowledge in designing empirical studies and statistical analysis visit the morning session "Evaluation 1: Fundamental Empirical Techniques and Caveats" before they attend this tutorial.

Alternatively, participants will be offered the opportunity to read a small set of preparatory materials available on-line before attending the tutorial.

Format

We aim to make the learning experience of this tutorial as interactive and learner-centred as possible. The sessions will use a combination of interactive lectures and practical application of methods to case studies. Participants will be split into groups (of 5-7 participants) to work on a case study of their choice.

In order to tailor the tutorial to the specific needs of the attendees a questionnaire will be administered to all registered participants in advance of the event. This will cover the type of systems they are working on (e.g., adaptive learning, adaptive hypermedia, e-commerce, etc) and their prior experience with evaluation.

Lecture notes will be made available during the tutorial and afterwards on the website.

Date and time

• Tuesday, June 26th 2007, Afternoon session

Website

http://www.easy-hub.org/hub/tutorials/um2007/

Organizers and Contact Persons

	Dr. Stephan Weibelzahl National College of Ireland Dublin Mayor Street IFSC Dublin 1 Ireland +353 1 4498 579 sweibelzahl at ncirl.ie http://www.weibelzahl.de/
	Alexandros Paramythis Institute for Information Processing and Microprocessor Technology (FIM) Johannes Kepler University Linz Altenbergerstr. 69 A-4040 Linz, AUSTRIA +43 732 2468 8442 alpar at fim.uni-linz.ac.at http://www.fim.uni-linz.ac.at/staff/paramythis/
	Dr. Judith Masthoff Department of Computing Science University of Aberdeen Aberdeen AB24 3UE Scotland, UK +44 1224 272299 jmasthoff at csd.abdn.ac.uk http://www.csd.abdn.ac.uk/~jmasthof/

-top-