[arch-general] Question about automated builder

[Allan McRae]

On 28/01/11 03:36, Thomas S Hatch wrote:
> I have been passively working on a similar project called quarters, but I
> must admit that my motivation is somewhat low not knowing if the project is
> in demand. So here is my question, do we think that something like this
> would be a benefit to Arch? Is this the type of project that should merit my
> attention?

An automated build system, in particular for the big rebuilds we do, has 
been discussed among the developers many times.  So there is definitely 
interest in such a system.

> Also, if we do think that this would be a good thing to have for Arch, I
> would like feedback on what types of features the system would have and how
> it would behave. Right now I am following the idea of supporting a
> distributed build system so that we can have any number of build servers in
> the fray working away to produce Arch packages for us. I am also attempting
> to build it in such a way that a database is not required and that the
> interface would be amazingly simple (this is Arch after all). This would
> mean that by mearly checking into svn a package would be built, and then an
> interface would pop up for the right people to sign it off, and once it has
> been signed off it would move over.

I have never liked the idea of automatic building after a SVN commit. 
There a plenty of times that commits are made to SVN in preparation for 
an update at a later time.  Instead, I would prefer a command-line queue 
submission tool (local or remote...).  You could have it flexible enough 
to take a "makepkg --source" tarball, or allow configuration for 
updating then building from a SVN/git/etc repo.

Thinking about what I do in packaging (and as someone who does not use 
the AUR), I see the following situations that need to be covered:


1) single package build:  This would probably be the majority use case


2) multi-package build:  Sometimes packages come in pairs/groups where 
updating one requires updating the next.  e.g. tcl and tk.  That 
requires building tcl, adding the new package to the chroot and then 
building tk.

Note there could be cases where this is more complicated...   e.g. 
Updating package A,B,C,D.   A is required for B, B is required for C and 
B is required for D.   Note that installing C in the chroot after 
building would be unclean for the build of D.  Think updates to desktop 
environments as an example of this.

makechrootpkg has the facility to add packages to a local repo after 
being built, so that could be used to implement this.


3) big rebuild:  e.g. for library soname bump.  There are two types of 
builds here.   A) the initial builds and B) all the packages requiring 
rebuild because of those in A.   For B, automatic pkgrel bumping would 
be good.  Also, some automatic determination of the build order for B 
would be very helpful.  This would require error handling as if a 
package low in the dependency chain fails, the rest of that chain should 
not be built.  Again, adding the packages to a local repo as the rebuild 
progresses will allow to build packages cleanly against those that are 
already rebuilt.   Also not that there could be no packages in list A 
(e.g. if we just want to rebuild all of [core]).


4) flexibility to add makepkg paramaters:  I'm not sure how widespread 
use this would get...
e.g. the toolchain is built in the following order:
linux-api-headers->glibc->binutils->gcc->binutils->glibc
The first glibc and binutils builds will be built with the --nocheck 
option in the future.  The gcc and final binutils and glibc builds are 
built with -L to save the testsuite build logs.  I guess you would make 
-L the default anyway and have them available at the end of the build 
for inspection.


Anyway, #3 is what I consider most important in an automated build 
system designed to make the job of packaging in Arch easier.  It is also 
the level where I would start using such a system....   Building a 
single package locally is a two "makechrootpkg" commands (one per 
architecture) and I do not have to transfer the resultant package to 
test.  So I doubt I would use a build system for that (unless build time 
was massive) as I can not see the automated build being more efficient 
for me.

Allan