With tracing, we can get to understand what is really happening while our software is running at a great level of detail. Depending on the adopted tracing solution, that applies to user-space components, to OS kernels or even to both. The collected data can be very helpful when investigating bugs and misbehaviors, but can also be used for understanding the internals of the system and for chasing performance degradations or improvements.

On Linux, tracing can be done in a variety of ways. This talk will focus on two: ftrace and LTTng. The aim is describing their characteristics and show how each one works. After that we will show what are the main differences between them in terms of what they are capable of, how the are used and last but not least the overhead they introduce, measured with several benchmarks.

We will also cover the tools that one can use to visualize the often huge amount of data that is usually collected during a tracing session. Both textual ones, such as trace-cmd and babeltrace2, and graphical ones, such as KernelShark and Trace-Compass.

Last but not least, we will offer some insights on how to use these two mechanisms for tracing virtualized environments. I.e., how can you use them for tracing what is going on both on the host and inside one or more guests, at the same time, and how good are they for such purpose?