|
|
2003/02/04 - Thomas Vander Stichele
GStreamer
An interview conducted by Alain Buret
Alain Buret - Please present yourself !
Thomas Vander Stichele - I'm Thomas, 26, live in Gent, Belgium, in a big house with three friends and too many computers. I started using Linux for my engineering thesis and haven't looked back since. I helped start a university radio station (URGent, http://urgent.rug.ac.be/) which was a streaming radio pioneer in Belgium and Europe back in the days ('98 ?), and my thesis was directly useful there.
Since graduating I started working for a commercial radio station doing all sorts of technical stuff, ranging from administrating servers and wiring to broadcast equipment handling and setting up new technology. I am lucky enough to be able to even make use of GStreamer in some simple applications here.
Alain Buret - How/when/why did you start working on GStreamer ?
Thomas Vander Stichele - Luckily I save lots of mails, so I can tell you it was sometime early 2001. I think I looked at it first because of a Slashdot article (http://slashdot.org/article.pl?sid=01/02/26/0124235&mode=thread&tid=131) and decided to take a look at it.
I remember that the thing that impressed me first was the great documentation. I started reading through the manual trying to understand what it was about. The second thing that impressed me were the underlying ideas of connecting elements with functionality and make complex pipelines. It probably wasn't a new idea at all, but for someone like me who was still very new to decent multimedia coding (my thesis is a mess now that I think back) it looked amazing.
Of course, then I started actually trying to use it a month or two later, thinking I could use it for a number of projects at the radio stations I work at. I specifically wanted to write a mixer to replace my old one written for my thesis.
Also, I have another spare time project to build a multimedia box in my living room connected to the TV. It's called The Dave/Dina Project (http://davedina.apestaart.org/) I hope to make it use GStreamer for every multimedia function on it, because it's a natural fit. Right now it's more of a proof of concept and a test bed of features. The
actual implementation is a hodgepodge of scripting and external applications. That means that it works, even very well, for daily usage, but you tend to notice how it's put together with different applications.
And then when trying to use GStreamer, I started finding problems here and there, and trying to get my foot in, so to speak. So GStreamer "trapped" me by providing excellent documentation with lots of promise, and then tricking me into helping out to fix stuff ;) I haven't regretted getting involved yet, however.
I started by doing some audio stuff I needed for work, then going on to help out with reworking the build system, separating core from plug-ins, to doing actual releases. Nowadays I start to actually understand the really low-level issues that GStreamer deals with (for example, low-level scheduler and threading issues) and manage to now and then commit improvements in those pieces of code ;) Let's just say that these days I'm not scared anymore to commit what ever piece of code I touched to CVS.
Alain Buret - What was the main goal when starting GStreamer ?
Thomas Vander Stichele - Erik Walthinsen is the project's originator. If I recall correctly, GStreamer evolved from his work at OGI (http://www.cse.ogi.edu/) and is based on concepts out of OGI's Pipeline program (whatever that is) and Microsoft's DirectShow.
The main goals, I think, were pretty simple :
be open-sourced
have a plug-in architecture so a lot of libraries could be used without forcing a dependency on them. It also has the advantage that if tomorrow a new format comes out, and we implement the codec for it in a plug-in, all GStreamer-based apps can then make use of that format and codec.
have a license that would also make it attractive to commercial companies, allowing for binary-only plug-ins. We'd love for Apple to make a binary Quicktime plug-in using GStreamer, for example.
be the open-source library of choice for all sorts of multimedia applications, from simple players and recorders to complex NLE's.
Alain Buret - What advantages do you think GStreamer has over similar frameworks ?
Thomas Vander Stichele - I'm not overly familiar with all of them since most of them are very closed-source or just not usable on Linux. Off the top of my head, they would include SGI's old framework, DirectShow, the BeOs multimedia API, and MacOSX's framework.
I know a few of the ones with less features in use on Linux, like xine, mplayer and xmms.
For the first group, advantages over the proprietary ones is first of all that we're open. But you knew that already. We also have an active community around GStreamer, with quite a few developers coming from different parts of the world and working on various projects. Most of these people have worked on various multimedia projects in the past, but got the same "click" with GStreamer as I did at some point. We have a wide range of people working on GStreamer. Some of them are incredibly good hackers like Erik Walthinsen, Wim Taymans (also from Belgium), David Schleef, Andy Wingo (who sadly went to Namibia for two years), and David Lehn. They manage to actually understand kernel, assembler, scheduling, and whatever other issues. We also have people joining in from other projects. We also have an excellent PR kind of person, Christian Schaller, which is a kind of person every project really needs. He points us in various directions, helps out on infrastructure stuff, talks to other groups, and keeps us connected.
GStreamer borrows the basic simplicity of concepts from DirectShow, with a few changes. Basically, it can be explained simply by an easy example. Take three elements : a disk reader, an Ogg/Vorbis decoder, and a sound playback element. Just connect them in this order, set it to play, and that's it ! This simplicity is an advantage over most frameworks out there.
Another advantage that follows from this simple architecture is that everything is in place to do easy prototyping. The example mentioned above can actually be run from the command-line like this : gst-launch filesrc location="/tmp/music.ogg" ! vorbisfile ! osssink
If you don't use OSS, but esd or alsa or jack, then just replace osssink with esdsink or alsasink or jacksink. If it's a wav file instead of an Ogg/Vorbis file, just replace vorbisfile with wavparse. And if the file is not on your disk but on a website, you can use a network-location plug-in. We also have a graphical editor, that allows you to prototype, design, save, load, and test pipelines. It's what Glade is to GTK+ developers. Eyes go wide open when I show people how to draw an Ogg player in fifteen seconds and have it play ;)
A third advantage (over older frameworks like xine and xmms) is that GStreamer was designed from the ground up to handle more than basic playback and recording. This isn't meant to bring down projects like xine and xmms, because they fill an obvious need, and were there first. They just started with different goals from us, which got them where they are today. For GStreamer, this means that great care has been taken to not make compromises for the simpler cases, but keep the architecture generic enough to be powerful for *all* sorts of multimedia work.
A fourth advantage is that the core itself is both generic and relatively small. This means that the core doesn't really know much about the data it handles - it just passes buffers between elements in pipelines. So GStreamer is far from bloated if you don't want it to be. You can easily make it very small if you need that. Erik used to work for an embedded company with the specific target of using GStreamer in commercial embedded applications. The company went under, but GStreamer can run embedded ;) Also, GStreamer by itself has very few dependencies : libxml 2.0, glib 2.0 and (since recently) popt. libxml 2.0 can be compiled out if you really don't want that either. I don't remember specific details, but I believe Erik managed to strip down stuff to a 100k binary at one point that actually managed to play audio.
A fifth advantage is how the architecture allows you to do really cool things that come naturally. As a simple example, elements describe the format of data they can work with. An Ogg/Vorbis decoder for example accepts application/x-ogg data and could spit out audio/raw. Because of this, you can write an element that automatically figures out how to connect two elements. In practice, this means that you don't need to know what format a file is in. You just let GStreamer figure it out by itself and if you have the right plug-in installed, it will play the file.
I could go on, but I'll leave some more things for you to find out.
Alain Buret - What do you think is still missing to have a full multimedia framework on GNU/Linux ?
Thomas Vander Stichele - If you mean "what is missing in GStreamer to be this", there are a few things I can think of and probably some more I haven't thought of yet. Here's what we still need that I know of :
more specialized input and output elements. As an example, mplayer has a very hackish kernel module for Matrox cards that somehow allows it to get TV output refreshes right so you don't get the "jagged" line in the middle of the screen because a video output refresh happens in the middle of a TV refresh. I would love for GStreamer to be able to use this module in a plug-in to make use of that feature. We have a few hardware-specific plug-ins, like v4l input, dxr3 output, but we can always use more.
better underlying libraries, or better handling of them. GStreamer by nature uses a lot of external libraries for plug-ins (currently about 45). A lot of these are still in development, or hard to track, or not very well managed, and so on. For example, avifile is a very hard project to keep track of and doesn't really do release management well. But we need to work with it anyway because people want their DivX ;-) files to play. (We'll probably move to use FFmpeg's libavcodec soon though) I think the community as a whole is just starting out to actually work well together as a whole. This is an important point, and most multimedia coders are so far out there with their own stuff that they don't always have time to look at the bigger picture. This is actually one of Erik's strong points, in a way - he plans ahead.
more open specification of codecs ! This is beyond our control, but it would help all of us open-source hackers a lot.
better metadata handling. The architecture is there in GStreamer, but it needs fleshing out.
Time caches. These are used to cache time offsets into files. This is mostly useful for video to quickly seek to the right location. Wim, our most prolific coder at this point, is currently working on this.
some form of API stability. As a group we are currently working on this. We're at the point where large parts of the core haven't been touched for months, or only slightly. Now that GStreamer is going into Gnome 2.2, we are ready to officially mark parts as stable for an indefinite amount of time.
some company to actually use GStreamer extensively and thus allow for one or more core hackers to work on it part-time or full-time. GStreamer, in my eyes, is an amazing project, and it is fantastic that it manages to work so well without any full-time hackers on it. On the other hand, GStreamer would greatly benefit from one or more people being able to live and breathe it almost full-time ;) There are still very interesting topics to be explored for GStreamer. Send us a mail, make us an offer ...
someone to work on the documentation for writing plug-ins.
better media tests and error handling. It's amazing what can go wrong with multimedia files. For example, some people out there tag their Vorbis audio files with ID3 comments, which in theory is specifically forbidden by Ogg/Vorbis. But people do it anyway, so you have to work around it, because otherwise people complain "that xmms handles it fine, why doesn't your software ?"
language bindings. We have preliminary bindings for Java, Perl, Python, Mono, C++, and Qt/KDE. But all of them could use work to be very useful.
KDE integration. We would love for KDE to adopt GStreamer as well. Unfortunately, lots of issues could prevent this. GStreamer is mostly pure C, while KDE developers prefer C++. One of KDE's multimedia programmers specifically told me he would never allow a C multimedia library to be used in KDE *on the grounds that it is not C++ alone*. On the other hand, we have an excellent KDE developer around who wrote KDE/Qt bindings to make a simple KDE-based player using GStreamer. We will continue to make it easy for KDE developers to step in, help them out wherever possible, and generally encourage them to work with us. I think this is one of those things which only time will tell - if there is a very motivated and very crazy KDE developer that wants to write a framework from scratch, and he gets there before we do, then that will be it ;) I don't really see that happening any time soon though. And, I hope it doesn't happen. The more libraries Gnome and KDE can share, the better.
Alain Buret - People often say that GNU/Linux is not ready yet for the Desktop, multimedia support is one of the reasons. What do you personally think about that assumption?
Thomas Vander Stichele - First of all, let me say that I myself do not care very much at this point about the whole "is Linux ready for the desktop" question. On the other and, I feel very good about the fact that I can do all of my daily work on Linux, at home and at work. I feel at least as good about my dad having made the choice to use Linux at home, and seeing that Linux is at the point where he is actually happy with that decision as well ! And maybe, just maybe, I'll even get my mother to use it too. At some point in the future.
That said, multimedia support IS a good reason not to switch to Linux. There are various reasons for this. First of all, people nowadays expect their email client to play video. Can we do this ? Actually, yes - we have a bonobo component to play video and it could probably easily be called from Evolution, in-line, just as you can view jpeg's. Is it a good idea to do so at this point ? No, because it's amazing how many broken multimedia files there are out there and how much code it takes to make your framework robust against all of those.
Second, the biggest problem is probably that it's by nature a hard battle to win if you cannot get access to specifications of codecs you try to support. Windows will probably always have better support for whatever .avi file you play. Macintosh will always have better support for Quicktime. By definition, we will always lag behind in those areas, unless something changes about the whole situation.
Third, it is a very complex field behind a very simple idea. Users mainly have a bunch of files they want to play, and the files contain audio and video. How much simpler can a concept be ? In practice, it's a mess. I don't know how many codecs you can wrap inside an .avi file, but there are too many to handle. Same for .mov files.
So while users only see five types of files on their system, to play each format you need lots of back-end and codecs and libraries to make it work. If the bug report says "GStreamer can't play my avi file", it takes a lot of work to figure out what is wrong. So it will still take a lot of work to make things Just Work.
Fourth, we need applications that use GStreamer to actually test if things work, and work well. We're at a position where a few applications make good use of it, and more start popping up every day. See the next point.
Alain Buret - How many applications are using it ?
Thomas Vander Stichele - Hard to tell. Not all of them are out there yet. Here's a quick list of applications that you can easily get and work well:
GStreamer Player, a basic audio and video player
RhythmBox (http://www.rhythmbox.org/), which started out as a clone of iTunes but is evolving into a very nice music management system.
gnome-sound-recorder, Gnome 2.2's sound recorder, uses GStreamer to record audio in a lot of formats, and actually work
nautilus-media, which is a Gnome module I'm managing. It currently implements property pages for audio files (allowing you to see relevant stuff like metadata and audio properties), an audio view to scan through and play audio in a directory, and a video thumbnailer which generates stills from the video files in your directory.
marlin, a sound wave editor in it's early stages
There are some things I have done that are not specifically "out there" or good enough for general use. I suspect other people have similar things. For example, I've written an audio archive application for work purposes that stores broadcasts in pieces of one hour in both high and low quality. We keep the high quality for two weeks so we can record it to disc to send to people who ask for it, and we keep the low quality for half a year because we are legally forced to ;) That application has been running for about a year now, so that's pretty good. I also wrote a tool to detect high frequencies, which turns on the traffic announcement signal on RDS.
GStreamer allows you to easily script simple stuff (file conversion, recording, and so on) allowing you to make simple applications that suit your needs.
Beside that, a lot of stuff is in the pipeline. Some examples :
gnonlin is a non-linear editing library based on GStreamer. It extends the concepts to a higher conceptual level, and should form the basis for audio and video composers.
gstreamix is the rewrite of my thesis project ;-) I hope to use it as the back-end for a radio broadcast system which we will use at the college radio.
vdv will be a python-based dvd player.
getcontrol is a GUI system for media boxes.
gst-record is a library and application to record any form of video to any supported format.
See http://gstreamer.net/status/?category=2 for a list of projects using or intending to use GStreamer.
Further, we actively try to stimulate other projects to work with GStreamer. We have made our way into Gnome, we have good hopes of getting Damien to use GStreamer in GnomeMeeting, we lobbied successfully for some projects to relicense under LGPL, and so on.
Alain Buret - What are your expectations from your Fosdem talk ?
Thomas Vander Stichele - I hope to give a fairly decent overview of what GStreamer can do, and entice people to try it out for themselves. It's all about getting people excited and thinking about the applications they could use GStreamer for. I hope I can manage to attract some people to joining us, and tell all the others about the techology they will be using in the near future. Also, I hope to get impressions of people about what we should concentrate on in the next year, and how we can improve on things, and how we can lower the threshold for them getting involved.
|
|
 |
Special announcement |
|
FOSDEM 2003
will take place
on February 8 - 9 2003
in Brussels... |
|
|
FOSDEM search |
|
|
Contest |
|
Best background:
|
|
| Create the coolest Fosdem background design and win cool stuffs... More info |
|
|
Sponsors corner |
|
|
