Fedora packaging in Gentoo

17.2.2014 01:23 | Přečteno: 1502× | linux/unix | | poslední úprava: 19.5.2016 14:54

This will be a post about the possibilities of Fedora packaging on Gentoo. I've been doing quite some Fedora packaging when actively using that distribution and later added more because of the importance of Fedora for my current employer.

Why Gentoo

Eating your own dogfood. The meaning of that sentence reaches new dimensions when you're saying it as an open source software developer. Even more so, when you're the sort of a developer I am, digging into lots of projects. I basically need to be able to incorporate any patches, releases or Git snapshots into my own system and see how well it plays with all those services I'm using on a daily basis as well as tools that I want to just test.

There are numerous possibilities with any distribution. But you typically end up doing a compromise between a stable binary distribution for every day use and pieces of Linux From Scratch. It's not hard to achieve the results you want to but it is somewhat inconvenient and you tend to spend a huge lot of time with that. And there's no way to easily repeat it. So you would finally end up automating parts of the process. It's also difficult to renew a working environment at least resembling the distribution you started with. With many packages, it's difficult to ensure that your test builds use the test build of the package and not the installed binary package that is part of your normal installation. You can take that as a very lazily assembled problem statement.

Instead of building up all of the necessary automation tools from scratch, I chose the Gentoo Portage as the toolset. As I wanted to achieve full integration with my system, I decided the system will itself be Gentoo. Therefore while I consider myself a user of the Gentoo Linux distribution, it is more of a side effect of choosing the Gentoo Portage as a build platform for my testing. This was the main reason I started using years ago, and it was also the main reason that I returned back to Gentoo after roughly five years of only using binary distributions. And it helps to answer why I'm not generally recommending my distribution of choice to anyone else.

When using and discussing Gentoo, I discovered a couple of other advantages that are less important to me but that improve my user and community experience a lot. Gentoo seems to be company-neutral. When using Gentoo, there's no natural inclination to a specific company that you would experience using Fedora, CentOS, OpenSUSE or Ubuntu. Debian would fit as well, while being a bit more tricky than Gentoo in this respect. And indeed there seems to be a large community of Gentoo developers whose members are affiliated with all sorts of companies and organizations. Arch Linux would be on the same level as Gentoo here, but I never liked the toolset. That's for the politics.

Going back to technical features that improve my experience with Gentoo. The stable package set proved to be a rather conservative one, with a unique QA system for a community distribution. I haven't seen anything comparable in the release-based distributions, not to say rolling release and I don't dare to say I understand the process, yet. It's not comparable to Debian, which has a stabilization period for the whole distribution. It's not comparable to Fedora, where package stabilization is more or less handled by the package maintainer himself. In my opinion, the results are incredibly good, especially when Gentoo lives with QA handicaps like rolling release and a relatively small developer base.

Why Fedora

The importance of the Fedora distribution for my employer, Red Hat Czech, doesn't need to be explained. I will avoid the overall quality of Fedora as much as possible, as it substantially changes over time for better or worse. Of binary package distributions, Fedora is currently the most complete distribution for my daily tasks. That includes playing with software that's not packaged in other distributions including Fedora build tools. And I'm pretty much familiar with the process of adding new packages for whatever is missing. The RPM spec files remind me of Gentoo ebuilds and I've been using the Fedora build system for years.

Also, Fedora seems to be slowly moving to a direction I believe in for common usage, example of which is the announced future distinction of the server and workstation use cases. It is a distribution of choice for many developers and it seems to have a rather active developer community outside Red Hat. For good or bad, what you see in Fedora is very often what you will see in other distributions soon. That pushes Fedora into a position of an important development decision point as well as a preview of the future Linux ecosystem.

Fedora development workflow

Fedora package definitions are stored in separate Git repositories. The repository contains a master branch,one branch for each supported Fedora release and optionally one brach for each supported EPEL release. The EPEL branches are somewhat hybrid between Fedora and RHEL (and derivatives), as they are part of the Fedora Project but they're being built for the enterprise linux releases. The version control workflow is mostly in the hands of individual package maintainers. The package maintainer explicitly requests the Fedora build service to assemble a build using the top of a branch. For released branches, updates have to be created explicitly as well and then a semi-automatic procedure is used for stabilizing the package update. So there's a couple of services that need to be interfaced and a couple of convenience tools to help with the packaging tasks.

So far I had to use a second physical or virtual system with Fedora for the pacakge work. I cheated a little bit and actually used a chroot with the filesystem tree from my previous Fedora 18 installation and it worked pretty well. Fedora 18 is now rather outdated. When Fedora 19 was out, I tried to upgrade the installation but the upgrade failed with no error message. The installation was still usable except kernel and related updates were broken since then as any new initramfs images were trying to perform the upgrade.

Fedora tools packaged for Gentoo

So far I packaged the tools I needed for everyday Fedora development and their dependencies, so I can abandon my obsolete Fedora installation. Most of the tools and libraries have their upstream projects hosted at http://fedorahosted.org/ and source code available via Git and release tarballs. Some of the projects offer sourcd code via Bazaar and Subversion. I decided to ignore the released versions for now and I created a number of live ebuilds pulling the code directly from upstream repositories. That is in line with my intention to try to resolve any problems in the upstream first. All the packages are available in David Heidelberger's ixit overlay.

# layman --fetch
# layman --add ixit
# emerge fedora-packager --autounmask-write
# dispatch-conf
# emerge fedora-packager

$ fedora-packager-setup

Now you can use the fedpkg and koji commands with some limitations.

Differences from Fedora

There are a couple of things that won't work for you on Gentoo. You won't be able to do local builds easily as they rely on the local RPM metadata. I didn't package the tools necessary to do a local mockbuild as I didn't need it. I haven't tested many of the features, so you may need to either fix it or use a Fedora installation in case of problems.

As mentioned earlier, my intention was to try to fix any integration issues with upstream first. So far the only major difference from Fedora is that Gentoo already defaults to Python 3, which also constitutes an exception from the above upstream first rule. Where it's necessary, the ebuilds change the code using sed to force using Python 2. I don't see a reason to disturb the upstream developers until they themselves to make the code compatible with Python 3. It might be useful to submit patches to select Python 2 in an environment where Python 3 is default.

Apart from the Python and RPM differences, I don't see anything that would make Gentoo an inferior Fedora packaging platform comparing to any released Fedora or Rawhide. The lack RPM metadata is limiting, though. Looking forward to any suggestions and ideas as I'm myself curious how all of this will work out.

Final notes

Gentoo packaging is fun. I packaged some other software and put it to ixit as well and more live ebuilds are yet to be added. But my ultimate goal is to get those packaging tools as well as a number of networking related packages included in the official Gentoo repository.

The choice of English language in the article should make it readable for a broader audience if needed. It has been written at once, without any careful editing, so please take it as it is.

Edit 2016-05-19

Fedora packaging tools for Gentoo are still maintained and have now their own overlay repository on github. Also now with Gentoo adding an overlay is as easy as putting the following lines into /etc/portage/repos.conf/fedora.conf.

[fedora]  
location = /usr/local/overlay/fedora  
sync-type = git  
sync-uri = https://github.com/pavlix/gentoo-fedora  
auto-sync = yes

       

Komentáře

[Honz]

Too loooong....
http://www.youtube.com/watch?v=QrtydD2u1N0&feature=kp

[kotrcka]

end tú inglyš.

[pavlix]

Předpokládám, že se mi právě ozvali dva zapálení fedoří maintaineři.

[kotrcka]

jes. ajm mejntejnyng máj oun mäšín.

[kavol]

You won't be able to do local builds easily as they rely on the local RPM metadata. I didn't package the tools necessary to do a local mockbuild as I didn't need it.

I believe I've seen a tool that is able to mirror the package database in different formats, but I wouldn't recommend(*) using that; mock is the way to go

(*) really, don't do that, I've seen all kinds of troubles having their origin in build environment differences that haunt our RHEL developers

[pavlix]

I like local builds because they are much quicker to tweak and quilt setup doesn't always do the job well. Mock is too slow and too complicated for many developer tasks. That could be more or less fixed by adding --nodeps to fedpkg compile.

[kavol]

Mock is too slow and too complicated for many developer tasks.

well ... in that case you should rethink the workflow probably, because I usually do not need to run mockbuild more than once for each release, so I'm not bothered by the slowness

and I'm not getting the "too complicated" part - for packager, it's just a matter of `fedpkg mockbuild` ... for the mock package maintainer, don't know how much it is tied to Fedora (e.g. the mentioned python version issue etc.) but I thought it is pretty universal?

[pavlix]

well ... in that case you should rethink the workflow probably

I love when people ask me to use a less effective workflow just because they think it's the right thing to do.

because I usually do not need to run mockbuild more than once for each release, so I'm not bothered by the slowness

I thought it was pretty clear that I'm not talking about basic maintainance tasks like bumping a release, rebuilding and leaving it until a new one comes. But to be honest, I'm not even using a local mockbuild for that, instead I'm using koji build --scratch.

and I'm not getting the "too complicated" part - for packager, it's just a matter of `fedpkg mockbuild`

I'm not specifically talking about cases fedpkg mockbuild and waiting for a resulting RPM works well.

The use case for local builds is when you need to repeatedly try to compile the source with relatively small changes until the result of the compilation is good. From SUSE folks I know they're using quilt setup for that but that doesn't work for all packages, although it has the advantage of keeping the patches ready for editing. You could theoretically use rhpkg prep but that doesn't do the configure stage which is not a simple cut-n-paste as it uses some system-level rpmbuild variables.

When you're done with that, a slow final koji build is a natural followup, as you need to handle the dependencies properly and you want to get the installable RPM. Sometimes a local mock build can be useful as an alternative to the koji build, but that's another topic.

[kavol]

well ... in that case you should rethink the workflow probably

I love when people ask me to use a less effective workflow just because they think it's the right thing to do.

I can hardly imagine that a workflow that does not rely on a slow tool (be it mock or koji) that much could be less effective than the one which makes extensive use of it

I thought it was pretty clear that I'm not talking about basic maintainance tasks like bumping a release, rebuilding and leaving it until a new one comes. But to be honest, I'm not even using a local mockbuild for that, instead I'm using koji build --scratch.

but that's the only point where do you need to actually build the package, be it local rpmbuild/mock/koji (the last being the slowest in a typical case, so ...) - I'm not getting why do you talk about mock slowness then ...

I'm not specifically talking about cases fedpkg mockbuild and waiting for a resulting RPM works well.

I'm completely lost then ...

The use case for local builds is when you need to repeatedly try to compile the source with relatively small changes until the result of the compilation is good.

ok, but "compile the source" != "build the package" ...

seems we're too generic here, I'd need some concrete example to understand what are you trying to achieve ... but I can live without that happily

[pavlix]

I don't know how much actual development development you're doing except the fact that it's not your everyday job. But for a developer, build an RPM package from dist-git is a very tiny portion of his job. And for nontrivial packages, it's still a small portion of the packaging as well. No matter that the final step is to build a set of RPM packages.

You don't actually need to have a specific example, just remember that your favorite package maintainers and developers need to touch and test the code that isn't present in the dist-git verbatim but is produced by the %prep phase and also part of the %configure phase as RPM is pretty dumb and doesn't distinguish between configuration and compilation, as some other build systems do.

When I want to check whether something works while doing my own little tweaks, I need to step in the build process when it's about to call make, or I can live with stepping in just after. That's possible with mock as well, but it's incredibly slow. And it's just as incredibly slow to reset the state with mock, as its caching capabilities don't seem to be very good.

That you can live without that is fair. Bud to advise an active developer to use a slower way using mock just because you're not familiar with the faster way is just ridiculous.

[kolcon]

az to bude ke cteni a ne jeden dlouhy souvisly blok textu, s dlouhymi souvetimi, u ktereho je problem poznat, kde to vlastne zacina a kde konci, a o cem to vlastne jako je, tak si to pak mozna prectu ;)

[Bedňa]

To sa čudeješ, vieš koľko Jílkových blogových zápiskov tu musel prečítať, to musí zanechať stopy Mňa by skôr od Pavlixa zaujímalo prečo chce balíčkovať bez riešenia závislostí na konkrétnej vetve Fedorky, alebo mi niečo uniklo?

[pavlix]

To sa čudeješ, vieš koľko Jílkových blogových zápiskov tu musel prečítať, to musí zanechať stopy

Kdo se pročetl až na konec, tak ví, že ten článek je needitovaný výblitek, protože na nějaké vyšperkované povídání jsem neměl čas. Už tak jsem šel spát dost pozdě.

Mňa by skôr od Pavlixa zaujímalo prečo chce balíčkovať bez riešenia závislostí na konkrétnej vetve Fedorky, alebo mi niečo uniklo?

Pokud reaguješ na článek, tak uniklo. Ty zabalené nástroje komunikují s fedoří build infrastrukturou, která závislosti řeší tak jak je u RPM distribuce zvykem. Pokud na komentář výše, tak fedpkg compile používám proto, abych bez zbytečného zdržování dostal adresář, ve kterém můžu dělat úpravy a pouštět make.

Popravdě řečeno, kontrola závislostí ve fedpkg compile je úplně na hovno. Ve Fedoře stejně předtím zavolám yum-builddep a tudíž mám závislosti nainstalované a zkontrolované. Mimo Fedoru nemá fedpkg compile data pro nějakou relevantní kontrolu.

[Bedňa]

Nedocvaklo mi to, trochu viac textu ako býva zvykom ;)

[David Jaša]

Předpokládám, že se mi právě ozvali dva zapálení fedoří maintaineři.

hm, třetí do party.

[pavlix]

:D