Towards a More Predictable Database System

While much of the research on transaction processing has focused on improving overall performance in terms of throughput and mean latency, surprisingly less attention has been given to performance predictability: how often individual transactions exhibit execution latency far from the mean. Performance predictability is increasingly important when transactions lie on the critical path of latency-sensitive applications, enterprise software, or interactive web services.
This dissertation proposes a systematic approach to solving performance unpredictability issues in traditional database systems. We propose the first profiler we know of to diagnose this issue and identify sources of unpredictability, and, based on the findings we discover from applying this tool on MySQL, introduce techniques to mitigate these source. Specifically, this dissertation proposes three tool/enhancements as follows:
· Variance Profiler (VProfiler): In contrast to most software profiling tools that quantify average performance, we propose a profiler called VProfiler that, given the source code of a software system and programmer annotations indicating the start and end of semantic intervals of interest, is able to identify the dominant sources of latency variance in a semantic context.
· Studying Causes of Unpredictability in Existing Database Systems: We conduct the first quantitative study of major sources of variance in MySQL, Postgres (two of the largest and most popular open-source products on the market), and VoltDB (a non-conventional database). Based on our findings, we investigate alternative algorithms, implementations, and tuning strategies to reduce latency variance without compromising mean latency or throughput.
viii
· Contention-Aware Transaction Scheduling: Nearly all existing database systems use a First-In-First-Out strategy for deciding which transaction should be granted the lock on a row when it becomes free. Inspired by the VATS algorithm, we further study the effect of lock scheduling in database systems, and propose lock-aware scheduling algorithms to reduce average latency, therefore improving the overall predictability of database systems.
VProfiler enable us to identify the root causes of performance variance in existing database systems, and by introducing algorithms targeting those root causes, not only do we improve performance predictability by up to 5.6x, but also improve average performance by up to 6x.