FOSDEM

Scott James Remnant will give a talk about Upstart at FOSDEM 2009.

Could you briefly introduce yourself?

Well, my name is Scott James Remnant. I work for Canonical Ltd, the commercial sponsors of Ubuntu; and my job is a lead developer for the Ubuntu Foundations team.

My particular area is the boot process and the plumbing layer. I care for those pieces of infrastructure between the kernel and the X server. Notably this includes the init system and udev.

I've been involved with Ubuntu since the very beginning, and was one of the Debian developers that Mark hired when originally forming Canonical. I've done a number of different jobs at the company, including finding my way into, and back out of, management.

Previously to working on Ubuntu, I was a member of the Debian project where I maintained dpkg and Libtool - the latter, I did enough work on to become one of the upstream maintainers as well. I've also written a couple of small projects such as the popular Planet aggregator and dircproxy IRC bouncer.

My first real contribution to the Linux community was running a humour site called Segfault.org, but I think that's been forgotten in the mists of time ;-)

What will your talk be about, exactly?

I'll be talking a little about the history of Upstart and the problems with the current releases, and then will introduce the planned Upstart 1.0 and talk a little about the changes that are going to happen and the roadmap to get there.

What do you hope to accomplish by giving this talk ? What do you expect?

I'm hoping to answer many people's questions about what's next for Upstart (of which I spy a few below :p), and what I'm planning to do with it.

Hopefully I'll get a lot of feedback as a result, and will know what people think will work and what people think won't. Maybe I'll even get ideas for new features or better ways of doing things.

Also I'm hoping simply to educate.

What did you do as Ubuntu Development Manager?

Canonical originally had only a single team working on Ubuntu, which grew to fifteen people reporting to one person (Matt Zimmerman, the CTO). We realised that it just wasn't scaling, and had to split the team into two.

I successfully interviewed to manage one of the resulting two teams, and thus roughly half of the Canonical Ubuntu developers reported to me. My job was pretty much the same as any other team lead or manager job at any other company.

I managed the day-to-day work of the team, planned work for each six-monthly cycle, handled performance reviews and so on.

Canonical didn't stop growing, and we quickly added further teams. Along the way we rebranded my team to be the Ubuntu Desktop team, and thus my job title was actually Ubuntu Desktop Team Manager for most of the time.

However I stepped down last year, wanting to return to full time development work.

What were your reasons to start developing the Upstart system? Which problems is it trying to solve?

A number of different reasons really.

Firstly the simple realisation that one of the most core pieces of software in Linux was the least understood and maintained. Nobody actually uses sysvinit's features, and instead works around them with "init scripts" and other similar things.

Secondly work on hotplug and then udev, and increasing support in the kernel for userspace being able to react to changes, made me realise just how adaptive the boot sequence really needed to be.

That work also led to a lot of race conditions, and a lot of busy loops; being able to eliminate those was a big reason.

Why did you decide that Upstart should be backwards compatible with SysV init?

That was easy ;)

There's a long running joke that the reason Ubuntu works as well as it does is the side-effect of the battle between Colin Watson and I.

Colin firmly believes that all changes should be made in as small a steps as possible, always preserving compatibility both backwards and sideways, and that patching and improving an existing system is better than writing a new one.

I firmly believe that sometimes you've just got to ditch the past and start over from scratch. (The standard library inside Upstart is called libnih for a reason :p) To steal a phrase from a favourite author of mine, I am to backwards compatibility what King Herod was to the Bethlehem Playgroup Association.

Since the right course of action is always somewhere inbetween these two extreme points of view, Ubuntu steers well down it.

The only way I was ever going to get Colin to agree that writing a new init system was a good idea was by promising to make it backwards compatible with the old one.

And as ever, that compromise between us works out absolutely for the best.

It's easy for any distribution to adopt Upstart, they only need to throw away the old /etc/inittab file, which hardly anybody uses anyway. All the old init scripts still work just as before.

It's also meant that we don't need any kind of Upstart flag day, since init scripts can be supported forever, we don't need to convert things over to Upstart jobs in a rush.

Instead we've been able to get the system right first!

When will all services in Ubuntu have their SysV init scripts converted to Upstart? What are the difficulties in converting them?

The difficulty has been that Upstart's method of describing when they should be run is not exactly the kindest thing, and gets really complicated for trivial use cases.

It exposes too many internals, basically.

One of the big changes I'll be presenting is the solution to this problem, which will hopefully begin the avalanche of jobs moving over.

Another side reason has been the desire to have at least one other distribution using Upstart; now that Fedora ships with it by default, we can actually try to standardise on job definitions. My hope is that an upstream should feel confident shipping an Upstart job in their own releases, and expect it to just work on all distributions.

Which new features can we expect in Upstart and when?

You'll have to come see my talk for that one ;-)

How will Upstart replace cron and atd?

A goal is to try and centralise all service management facilities in one place. Now, cron might seem a little unrelated to init scripts, but if you actually look at what tends to be in /etc/cron.* they're not as distant as you might think.

For example, on my system here, sysklogd has a cron.daily script; this calls savelog and then actually calls the sysklogd init script to restart the logging daemon.

I believe that such things should just be a part of the sysklogd service definition. You should be able to see that sysklogd has a "rotate-logs" action defined, and run that manually if you want. Since that also requires automatic activation, the init daemon would have to support the notion of such things as "daily".

Once you do that, it's not a big push just to merge the whole thing into init anyway. After all, did you know that cron supports @reboot for user jobs?

Obviously we'd extend it as well, you wouldn't just be limited to standard time definitions, but times related to other events as well. Who has never wanted to define a service as being run from "45s after startup" or similar?

Merging cron's other features has benefits too. cron mails you the output of its jobs, why doesn't init? If apache fails, having the errors mailed to you would be rather handy, don't you think?

So I think they're natural fits.

atd is just a specialised version of crond, from this point of view.

inetd is another question people ask, and that's a much more difficult answer ;) Apple's launchd integrates these two _very_ well.

You were a Debian developer for years. What made you decide to become a Ubuntu developer?

Ubuntu is more fun.

I could never have done something like Upstart in Debian, and certainly couldn't look after something like the boot sequence!

For example, this week in Ubuntu we switched to using /lib/udev/rules.d for the install location of udev rules (matching upstream). Such an action in Debian would require explaining the reasoning, and probably arguing, with dozens of different maintainers - and maybe even not getting agreement.

In Ubuntu, I just did the ~80 uploads myself.

Likewise for simple changes to boot reordering, in Debian, that could be a complete nightmare. In Ubuntu, I have the freedom to do it all myself (you can still persuade others too, of course ;p)

And I really didn't like the way that Debian reacted to Ubuntu. I had thought that it would embrace its offspring, and see that Ubuntu was attempting to reach a completely different individual to what Debian had.

I even thought that Debian might not mind that Ubuntu patched software to make it work differently, especially together.

After all, how many patches does Debian carry against its upstreams where they've failed to get agreement with the maintainer - or are just dealing with a different policy?

The whole ethos of Open Source is that anyone can take your work, and build something different or better out of it. And that you can then see what they've done, and if you like, integrate it back in.

Debian's reaction baffles me to this day.

Compare it to Ubuntu's embracing of all of the different derivatives of it, even those that change fundamental things.

It was the Debian event when somebody turned up in a "Fuck Ubuntu" t-shirt that made me decide to leave.

I realised there were too many people in Debian who believed in Debian more than they believed in free software; and too many people who were so religious about "their package" they they didn't want to see the bigger picture.

This interview is licensed under a Creative Commons Attribution 2.0 Belgium License.