FOSDEM

Steven Knight will present at FOSDEM 2800 about his project SCons.

• 1 What do you expect from your presentation at FOSDEM?
• 2 SCons is cross-platform. Is that a major feature for many users?
• 3 Is it hard to remain cross-platform or does Python does most of the heavy lifting?
• 4 Which advanced features of Python does SCons rely on?
• 5 Would those be possible in other languages too?
• 6 How is SCons tested?
• 7 A flexible system is a system that can be 'abused' for many purposes. Have you seen crazy things being done with SCons?
• 8 How is the progress towards version 1.0 going?
• 9 What was your personal reason to start working on SCons?
• 10 Where does the name SCons come from? Especially the 'S' :-)

What do you expect from your presentation at FOSDEM?

I mainly want to get the word out about how to use SCons effectively for Open Source development, and to get direct input on how to help improve SCons so it can be made even more effective for it. The biggest expectation I have is to get a lot of good, in-depth feedback from people with software build issues (both potential or actual SCons users and others). Finding out what problems people have or what doesn't work is more valuable than hearing if they like something I work on.

SCons is cross-platform. Is that a major feature for many users?

It seems to be. For example, I've heard of a fair number of gaming projects (both commercial and Open Source) that use SCons for development, reportedly because it makes it easier to manage development across the multiple platforms they want a game engine to support (multiple gaming consoles and a Windows version and...). The cross-platform support's also a big attractor for many of the larger enterprise software vendors that use SCons to maintain software on multiple platforms.

Is it hard to remain cross-platform or does Python does most of the heavy lifting?

Leveraging Python does give our cross-platform support a significant headstart, in that we don't have to wrestle directly with a lot of the lower-level implementation portability issues that projects that use C/C++ directly often have to worry about. That having been said, remaining truly cross-platform is a constant struggle.

The most consistent problem area is Windows portability, especially path name manipulation. The underlying Python os.path module only takes you so far because it only abstracts out path name syntax, and you still have semantic issues with how the path names interact with the rest of your code and the outside environment.

We also have a lot of the usual issues with trying to keep up to speed with different toolchains, and behavioral differences on various platforms.

Which advanced features of Python does SCons rely on?

Actually, SCons relies on *no* advanced features of Python, if by "advanced features" we mean things only available in newer versions of the language. SCons is written so that it will work if all you have installed is the ancient Python 1.5.2 (which was still the default Python in Red Hat 7.3, for example). We've started to make use of more modern Python modules (like subprocess) and features (like sets) but only if we have some mechanism, like an emulation layer, to keep the functional code compatible with 1.5.2.

That said, we do use Python's introspective capabilities for certain features, mostly things like inserting layers dynamically to display tracing or debugging information without slowing down the normal code path. Also, our parallel build (-j) support is based entirely on Python's threads, with a simple but effective worker-pool architecture.

Would those be possible in other languages too?

The underlying ideas in SCons -- that is, managing a full-tree build by using a scripting language API to program the dependency graph -- can certainly be implemented in other languages. But since SCons configuration files actually *are* Python scripts, the result wouldn't be SCons, of course. (Backwards compatibility might be a bit of a problem...  :-))

How is SCons tested?

We use a very strict testing methodology, adopted from Peter Miller's Aegis change management system, which requires that every change has at least one new or modified test (by default), and that those tests must not only pass when run against the new, modified code, but must also *fail* when run against the currently checked-in, unmodified code.

We started with this methodology from day one of development, and have over time built up a really strong regression test suite. The tests are every bit as much part of the "product" in that we treat any reported bugs not just as a problem in the SCons code itself, but also as a hole in our test coverage that must be fixed.

We use Buildbot to make sure every change is tested on multiple platforms and against every major version of Python from 1.5.2 to 2.5.

We just had a user report being pretty impressed by experiencing minimal hassles upgrading a complicated code base from using a three-and-a-half year old version of SCons. That speaks well for our development methodology and our emphasis on backwards compatibility.

A flexible system is a system that can be 'abused' for many purposes. Have you seen crazy things being done with SCons?

Sure, but usually within the realm of transforming software source files into target files. People have written some pretty comprehensive code to generate lists of target files dynamically from various arbitrary input. Our Wiki has a pretty extensive set of Builder modules that people have contributed for everything from CORBA to C# to Haskell. We've even had someone write a wrapper in Lua around the underlying SCons build engine.

How is the progress towards version 1.0 going?

Given how much attention we pay to backwards compatibility, a lot of people have suggested that we should just call what we have 1.0 and be done with it. There's a good case to be made for that.

Nevertheless, we've always wanted 1.0 to be really ready in as many details as possibile, mainly so that people who look at SCons for the first time when 1.0 is announced have as positive an experience as we can reasonably make it. The main outstanding issues here are:

• We're still shaking out issues from a big refactoring of our signature mechanism. That's been in our checkpoint releases since September, and it's very functionally stable at this point, but I'm still looking at some performance ramifications.

• Our Autotools-like functionality still leaves too much to the individual SCons user. It's kind of like having underlying Autoconf without the higher Automake layer that made things really useful by giving every package the same targets and build behavior, so that everyone can just "./configure; make; make install". Maciej Pasternacki's Google Summer of Code project last year was targeted at this, and we're trying to finish that so we can roll it out before FOSDEM.

• We'd like a way for people to configure our implicit dependency scanner (used to derive dependencies automatically from #include lines) to more accurately reflect the symbols defined by the C preprocessor.

• The User's Guide has been lagging all of the features that are present in the man page.

What was your personal reason to start working on SCons?

I started working with Cons, the Perl-based predecessor to SCons, in 1998 when I found that it basically solved the problem of building multiple side-by-side variants in a single dependency graph with minimal work. I got Cons to do what I wanted in one hour of work from a standing start, instead of weeks of monkeying with Makefiles and still not getting them to build variants in any extensible way. I ended up doing the majority of ongoing work on the Cons code base (including contributing an extensive test suite) before that project basically withered on the vine.

Despite the fact that Cons made me a fairly knowledgable Perl hacker, I had always been leery of Perl's esoteric syntax and TMTOWTDI-derived readability issues. The Software Carpentry competition provided a concrete reason to see if I couldn't take the things I knew needed to be improved in the Cons architecture and make them work better in a friendlier language. So I made the switch to Python and have never looked back.

Since then, I've simply found that I really like working on this problem of trying to give people a framework to make it easier to manage really hard software build problems. One of the things I like most about it is that it's so unglamorous -- most programmers would rather be working on "real software" than the internal build infrastructure. So management typically assigns the Makefiles to be maintained by junior people or summer interns, and then we scratch our heads at why the builds are unreliable and we have to "make clean; make" all the time to make sure the dependencies are correct...

Where does the name SCons come from? Especially the 'S' :-)

The real evolution of the name is pretty mundane. Most of the underlying inspiration does come from the old Perl-based Cons tool. The Cons-inspired design that won the Software Carpentry competition in 2000 was actually named SCCons, for "Software Carpentry Cons". In practice, repeating the two 'c' characters when typing "sccons" looked and felt too much like a typographic error, so I dropped one, and the shortened name was retconned to stand for "Software CONStruction."

The notion that "SCons" is short for "Steven's Cons" is purely an ugly rumor...  :-)