fosdem.org

Mark Wielaard will give a talk about SystemTap at FOSDEM 2010.

Could you briefly introduce yourself? And how are you involved with SystemTap?

My background is in theoretical computer science (functional languages, logic programming, etc.). Which was all very fascinating and interesting, and probably made me a much better programmer, but also not directly practical. When I started doing real programming jobs at first I didn't know if I would like to continue in the software industry. It all seemed so short-sighted. You never were sure your solutions would really help people in the long term. There was just too much interdependent pieces that had different "owners". Every piece that you build on could be "discontinued" or had uncertain legal terms attached to them that restricted your usage in various ways. So you never felt in control of the future.

Luckily at the same time (now more than twelve years ago) I also came in contact with the Free Software community. The GNU goals of giving users the freedom to run, copy, distribute, study, change and improve all their software were an eye-opener. If only we had a whole free system, that anybody was allowed to use and change for anything they wanted. That would solve all my problems! So, I started to contribute where I could. At first helping out the GNU java efforts (GNU Classpath and GCJ), just because I knew a little Java and found it important to have a fully free implementation of it (in 2003 I became the GNU maintainer of classpath). And these days all the software I use and hack on is Free Software. Who would have thought just a decade ago!

In 2006 I landed a job at Red Hat's Engineering Tools Group. That is the group that hacks on developer tools projects like gcc, Eclipse, binutils, elfutils, glibc, oprofile, strace/ltrace, gdb archer (formerly Frysk) and SystemTap. In 2008 I first started to help out the SystemTap project directly. Since a lot of my background really is around java, I hacked on adding better java event support and getting java backtraces when probing with SystemTap. In general I hack on the user space side of SystemTap, trying to improve the use of dwarf debuginfo to get better introspection (context variables, backtracing) when probing programs and libraries.

What will your talk be about, exactly?

The talk will explain where normal debugging, tracing or profiling of your system or application stops and when full system observability with SystemTap is what you want to do. And how SystemTap helps your investigation into what is happening to then better debug, trace and profile your system and applications.

I am hoping I can show my enthusiasm for making things observable by showing lots of examples. To encourage people to try out SystemTap not just for inspecting what happens very low level in the kernel, but also high level events like a database transaction, when a Java program triggers the garbage collector or following a Python script while it is running. And how making it easy to correlate such events brings deeper understanding how and why your system is doing things.

It will also explain to developers how to make their own programs easier to inspect by adding zero-overhead events/markers that can be observed through SystemTap (and other tools) by their users who want to better understand what and why their system is doing things.

What do you hope to accomplish by giving this talk? What do you expect?

I hope to show that debugging, tracing and profiling are essential when working on larger systems, and that doing it through SystemTap (as a start) is the easiest way to quickly and painlessly introspect your programs. I hope that people will start using SystemTap on their own projects as light-weight introspection tool. And that they will be encouraged to think about (and add) events in their own programs that are interesting enough to expose to their users or fellow hackers to help them better understand what is really going on.

What's the history of the SystemTap project? And how did it evolve?

SystemTap is an evolution of various debugging, tracing and profiling techniques. Making them very light-weight, non-obtrusive and scriptable. One could say that it all started with the dprobes project back in 2000. Dprobes was a dynamic debugging facility for the Linux kernel. It was way ahead of its time, it was monolitic, came with its own language and interpreter in the kernel. Sadly this monolitic design wasn't much liked, and people back then probably didn't realize the potential of the approach of having light-weight (kernel) debugging. Especially having an interpreter inside the kernel and how dprobes was too interwoven with the memory manager was a deal-breaker for getting it included into the mainline kernel. It did bring the important kprobes framework into the kernel though.

SystemTap was started in 2005 and builds on the ideas of dprobes. But has a user space translator, no kernel-side interpreter and tries to go to great lengths of reusing existing event sources and facilities already vailable. We also try to push as many independents parts as possible, so they can be reused by other tracing, debugging and profiling tools.

In the last couple of years a lot of effort has gone into extending beyond the kernel and support user space applications and libraries. Again reusing things already used by others, like the dwarf debuginfo for getting the variables in the context of a user space event probe and for getting user space backtraces (both also work kernel side). Also gcc was extended to produce much more accurate variable tracking in dwarf. Which again not only improves systemtap, but also traditional debuggers like gdb. Recently we have added support for static user space probes, which provided the base for Java and Python support, that I will also demo in my talk.

How does SystemTap compare with DTrace? For example with respect to functionality, architecture, ...

Dtrace has the same spirit as SystemTap. It was also inspired by the dprobes project. And both want to change to way people think about debugging, tracing and profiling, by providing an very light-weight, unobtrusive, secure, scripted probing mechanism. The nice thing about DTrace was that it provided a proof of concept of what would be possible if you deeply integrated observability into all layers of a software system you control. This really validated that the ideas around whole system observability were important.

Sun did release DTrace as Free Software in 2005 after the SystemTap project started. But they used a license (the CDDL) that made sure it couldn't be integrated with any GPLed project. Sadly that means it was a giant show-stopper from the start for using it on GNU/Linux systems. Otherwise I am sure people would have enthusiastically ported it. On the other hand it might have just ended like dprobes since it wasn't really designed with the GNU/Linux ecosystem in mind. SystemTap is designed to reuse as much common infrastructure as possible on GNU/Linux.

Feature-wise they are comparable. The SystemTap stap language is somewhat richer than the DTrace D language, supporting real control structures, which makes it nicer to write scripts (tapsets) in it. The D language on the other hand has script-level thread-local variables, which are really nice to have (in stap you simulate them using associated arrays). Both support global variables, associative arrays, aggregates, etc.

We made sure that at the (C/C++) source level static user space markers are fully compatible. Which means that if you add a probe point event to your application or library it can be observed from either SystemTap or DTrace. So, while the implementations differ a lot, the knowledge exposed by the events you add can be reused by both systems.

Implementation wise DTrace can take some short-cuts since it can depend on being in control of the whole system. For example everything is just recompiled with the same compiler to get frame pointer support. With SystemTap we have to work a little harder to anticipate users running different distros, kernels, with different gcc versions, sometimes, but not always, having all debuginfo available, etc.

SystemTap (and the whole GNU/Linux system) is ported to more different architectures than dtrace, which only supports a handful of processors. For detailed architectural differences take a look at the SystemTap wiki.

Are there usable SystemTap GUI applications?

There is stapgraph, a Cairo based graphing utility, distributed with SystemTap. It can be used to have dynamic graphs of predefined scripts. It is useful if you want a quick overview of a SystemTap script that generates regular output in response to certain events in real time.

This is not the only way to visualize SystemTap scripts though. Since SystemTap is so flexible in its output you can basically use any visualization tool as long as it has a well described input format. One of the examples that come with SystemTap creates output so the result of the script can be piped into gnuplot to generate a graph of disk and cpu use.

SystemTap has been closely integrated with the popular Eclipse IDE. What can developers do thanks to this integration?

There is the Eclipse Linux Tools project which provides various plugins to integrate SystemTap (and other tradition GNU/Linux tracing and profiling tools like oprofile, valgrind and lttng) into the IDE. There is the SystemTap Editor Plugin which adds stap script edition support to Eclipse, including source completion, syntax highlighting, partition support and code folding. It also supports the Callgraph Plugin which uses SystemTap to monitor the status of a program -- function calls, returns, times and even user-space variables. It then provides various ways to visualize the callgraphs and timing information of your program.

And there is the StapGUI project which provides much of the above, but in a stand alone (rpc) version, based on the Eclipse gui framework.

How many developers are working on SystemTap?

According to the source repository there have been a total of 72 different code contributors since the SystemTap project started. In the last year there have been 37 different contributors. It is a very diverse group. We are actively motivating people to contribute systemtap scripts they wrote and that they think will be interesting to other people. We then make sure they have documentation and add them to the testsuite to make sure they will not break by any code changes. That is a nice low barrier entry into contributing to the project.

What new features will we see in SystemTap in 2010?

With Free Software there are always surprises when someone smart suddenly joins the project and pushes you forward in ways never foreseen. But personally I hope to see much more applications adding static tracepoints. For Fedora 13 we have started to collect all the programs that already have them and try to help maintainers to enable them by default for the next release. Hopefully other distributions will follow.

The current support for runtime based languages like Java and Python is not very deep yet. Hopefully the information you can query will be much richer in the future. The basic support is there, but we need more people using it in anger and providing reusable tapsets for others to use.

And hopefully both utrace (a layer to observe processes and threads) and uprobes (built upon utrace to provide a mechanism to have dynamic tracepoints in user space programs, which can non-disruptively collect information about them) will soon be integrated into the mainline kernel. So they will be available to other tracers, debuggers and profilers, enriching the whole observability landscape on GNU/Linux.

Have you enjoyed previous FOSDEM editions?

I have attended FOSDEM for more than 5 years in a row now. And every time was breathtaking. So many talented hackers together. So much enthusiasm. And Brussels is such a nice city. At night there are just too many places to get "one last beer" :)

I am also very grateful to FOSDEM for having played an essential role in liberating Java. It has been a great venue for the GNU Classpath hackers to come together and plan their strategies. FOSDEM hosted the "Escape the Java Trap!" events since 2005.

And when Sun decided to provide their reference implementation as Free Software under the GPL, FOSDEM hosted a developer room where Sun and GNU hackers could meet to discuss how to cooperate together in 2007. I believe these meetings have been pivotal in forming a really great community around Libre Java.

And one of the talks at FOSDEM 2007 by Jim Blandy on gdb kernel tracepoints [PDF] was an eye-opener for me. Because he showed how debugging and tracing really could be used to simulate each other. And that doing debugging through tracing allows you to do things completely unobtrusive. That talk was probably the trigger for me to start look deeper into SystemTap. SystemTap uses the same kprobes technique for kernel tracing that he used to get gdb to debug a live running kernel.

So, Yes! I enjoyed previous FOSDEM events enormously. And I think FOSDEM is the catalyst for a lot of progress made in the Free Software community by bringing so many friendly hackers together.

This interview is licensed under a Creative Commons Attribution 2.0 Belgium License.