[REF] Efficient Tracing and Performance Analysis for Large Distributed Systems

Distributed systems are notoriously difficult to implement and debug. One important tool for understanding the behavior of distributed systems is tracing. Unfortunately, effective tracing for moderndistributed systems faces several challenges. First, many interesting behaviors in distributed systemsonly occur rarely, or at full production scale. Hence we need tracing mechanisms which impose minimal overhead, in order to allow always-on tracing of production instances. Second, for high-speed systems, messages can be delivered in significantly less time than the error of traditional time synchronization techniques such as network time protocol (NTP), necessitating time adjustment techniques with much higher precision. Third, distributed systems today may generate millions of events per second systemwide, resulting in traces consisting of billions of events. Such large traces can overwhelm existing trace analysis tools. These challenges make effective tracing difficult. We present techniques that address these three challenges. Our contributions include (1) a low-overhead tracing mechanism, which allows tracing of large systems without impacting their behavior or performance (0.14 ¿s/event), (2) a post hoc technique for producing highly accurate time synchronization across hosts (within 10/ts, compared to between 100 ¿s to 2 ms for NTP), and (3) incremental data processing techniques which facilitate analyzing traces containing billions of trace points on desktop systems. We have successfully applied these techniques to two distributed systems, a cooperative caching systemand a distributed storage system, and from our experience, we believe our techniques are applicable to other distributed systems.

Published in:

Modeling, Analysis & Simulation of Computer and Telecommunication Systems, 2009. MASCOTS ’09. IEEE International Symposium on

AUTHORS: Eric Anderson ; Christopher Hoover ; Xiaozhou Li ; Joseph Tucek

PDF VERSION: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=536615


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s