Publication

Current multicore OS benchmarks do not provide workloads that sufficiently reflect real-world use: they typically run a single application, whereas real workloads consist of multiple concurrent programs. In this paper we show that this lack of mixed workloads leads to benchmarks that do not fully exercise the OS and are therefore inadequate at predicting real-world behavior. This implies that effective multicore OS benchmarks must include mixed workloads, but the main design challenge is choosing an appropriate mix. We present a principled approach which treats benchmark design as an optimization problem. Our solution leads to a workload mix that uses as much of a system’s resources as possible, while also selecting applications whose performance is most sensitive to the availability of those resources.

@inproceedings{abc,
	abstract = {Current multicore OS benchmarks do not provide
workloads that sufficiently reflect real-world use: they
typically run a single application, whereas real workloads
consist of multiple concurrent programs. In this
paper we show that this lack of mixed workloads leads
to benchmarks that do not fully exercise the OS and
are therefore inadequate at predicting real-world behavior.
This implies that effective multicore OS benchmarks
must include mixed workloads, but the main design challenge
is choosing an appropriate mix. We present a principled
approach which treats benchmark design as an optimization
problem. Our solution leads to a workload
mix that uses as much of a system\&$\#$146;s resources as possible,
while also selecting applications whose performance
is most sensitive to the availability of those resources.},
	author = {Ihor Kuz and Zachary R. Anderson and Pravin Shinde and Timothy Roscoe},
	booktitle = {Proceedings of the 13th Workshop on Hot Topics in Operating Systems (HotOS 2011), Napa, CA, USA},
	title = {Multicore OS benchmarks: we can do better},
	year = {2011}
}