What is emerge, why is it important and what was Koen’s proposal?
On Unix platforms -that includes Linux and Mac OS X-, software is usually installed to /usr: applications in /usr/bin and /usr/sbin, libraries in /usr/lib, headers in /usr/include, common resources in /usr/share, etc. Also, dependency management is usually something you can count on: when you install kdelibs5-dev in Ubuntu, it will automatically install libqt4-dev, kdelibs5-data, libfreetype (runtime), etc That makes setting up a development environment a very easy task: look for shared libraries, header files, etc in the common places and you will probably find them.
On Windows there is nothing like that. When you want to compile an application, you need to provide (build and install) all its dependencies, and you need to tell Visual Studio where to find everything. Even CMake usually needs some help in the form of a hint for CMAKE_PREFIX_PATH. As you may imagine, building KDE, which has more than 200 third party dependencies and tens of modules (and with the move + split to git, many more) becomes an almost insurmountable task.
‘Emerge’ to the rescue: inspired by Gentoo‘s emerge, Ralf Habacker, Christian Ehrlicher, Patrick Spendrin and others (yours faithfully included) developed a tool which downloads the source, configures, builds, installs and packages KDE and its dependencies. It makes a world of difference when building KDE. Actually, it makes building KDE on Windows possible. Once more: thank you very much guys, impressive tool.
There are two well-differentiated parts in emerge, the ‘engine’ and the ‘recipes’.
The ‘engine‘ takes care of downloading the sources (tarballs, checkouts from Subversion and git clones, etc) , configuring and building for the usual build systems (QMake, CMake, NMake makefiles, etc), installing, bundling packages together and much more. All that for four compilers (MSVC2008, MSVC2010, MinGW32 and MinGW-W64). It’s the equivalent to Gentoo‘s emerge, Debian‘s dpkg + debhelper/cdbs or Homebrew and MacPorts for Macintosh.
The ‘recipes’ are the package-specific “instructions” a particular piece of software needs to build: what’s the name of the tarball? URL to download from? What versions are available? What build system to use to configure and build? Does it need any Windows-specific patch? They are written in Python using some helper modules ’emerge’ provides (the debhelper-like part I mentioned above).
Currently, there are about 70 recipes in emerge, organized in our ‘portage tree’ (bear in mind the names are taken from Gentoo but the internals of the tool are completely different). With those 70 recipes, we are able to build most KDE modules. Problem is to provide all the optional features, we are missing probably 200 to 250 more recipes. Given that KDE on Windows is quite short on developers, we have to decide: either we fix bugs and improve the integration of KDE on Windows, or we keep track of the dependencies. I won’t lie: for KDE on Windows, I’d rather focus on development than on packaging.
Given that writing and maintaining recipes does not strictly require programming skills (although they help ðŸ™‚ ), during my talk at FOSDEM I kept asking people to join the KDE on Windows team as packagers, even if you know little to nothing about software development. If you think about it, that’s what we have in Linux distributions: there are hundreds (thousands?) of packagers in Debian, Fedora, openSuse, etc which take care of the ‘recipes’ to build software developed by someone else (‘upstream’, in the parlance). What we have in KDE on Windows, where we are the KDE packagers, the KDE developers (upstream) and the emerge developers, is actually an anomaly.
Then Koen took the microphone and said: why don’t you open emerge to other projects?
And I there I realized he was damn right
We have developed an awesome tool to download, configure, build, package and install third party software on Windows, to manage dependencies, to update, etc. For four compilers, no less. The ‘engine’ of emerge is not really tied to KDE. We already have a lot of recipes for software which is not KDE specific: openssl, Qt, bzip2, libpng, mysql, etc.
I see no actual reason to disallow Gnome, GStreamer, OpenSceneGraph, LibreOffice and many others in (assuming someone wants to take care of them, of course – no, do not look at me! I don’t have time!). Currently, the only ‘barrier’ preventing those recipes in is ’emerge’ is developed in the KDE repository and you need a KDE developer account to commit your recipes. That’s about it.
There is also a ‘perception’ problem: “if emerge is developed in KDE, it’s because it’s a KDE-thingy, right?”. Well, no. One way to avoid that would be to graduate emerge from KDE and make it an independent project, one in which the current developers have accounts, and new developers get an “emerge project” account instead of a KDE account to commit recipes. Please note “graduating” does not mean “expelling”, “firing” or anything peyorative. It does not mean that ’emerge developers’ (engine and recipes) are worse than KDE developers, or less KDE developers than our god David Faure: in KDE, we consider Debian, Fedora, openSuse, etc KDE packagers as equals, they are also KDE developers and many of them are in the KDE eV (funny thing is I am not).
My wish for today: think of one application or library you’d like to see in Windows and become its maintainer in emerge for Windows.
For now, let’s do this in the kde-windows mailing list and if this idea succeeds, then we’ll talk about graduating. You could also try to add support for a new compiler (OpenWatcom, Intel C++, etc) but that’s a lot more more and it’s not a priority now.
PS: Yes, I know about CoApp, Microsoft‘s similar effort, with some magic involved. I’ve been watching and trying it since day 1. I’m really interested. It’s just not there yet and I do not have spare time to help Garrett (hey Microsoft, hire me and I’ll have all the time in the world! ðŸ™‚ )