COMPSCI 296.2, Fall 2004
Experimental Methods in Computer Systems
Monday and Wednesday 2:50 - 4:05 pm
Professor Carla Ellis
If Computer Science is to live
up to its name, then it is important for researchers to develop skills
in the scientific method. Many studies suffer because the results are
based on seriously flawed experimental technique and/or fail to provide
compelling support for the claims being made. Don't let this be you!
Topics will include:
- Measurement and simulation
techniques -- how to choose.
- Workload issues -- characterization,
workload selection (standard benchmark suites, micro benchmarks, synthetic
benchmarks, workload generators).
- Appropriate metrics for
the questions being asked.
- Effective experimental design
-- how not to waste your time.
- Dealing with data -- data
collection, data analysis, and data presentation.
This has been designated a
"regular" course for graduate students and a "research" course for Curriculum
- Term project -- to be developed
in a systematic way, exercising the research skills taught in the class.
It is acceptable to combine this project with systems projects in other
courses, assuming all instructors agree. Teams of two students are encouraged.
- Readings -- examples of
good and bad experimental studies from the literature. The textbook
will be Jain, The Art of Computer Systems Performance Analysis.
- Class presentations --
students will be assigned
particular areas of expertise to develop and be able to contribute to
classroom discussions, the term project will be presented in a mini-conference
at the end of the term, and there may be a requirement for a guest lecture
in the class.
- Exam -- there will probably
be two in-class exams.
Last updated August 22, 2004