[arch-general] PKGBUILD.proto improvements

C Anthony Risinger anthony at xtfx.me
Thu Jul 21 17:42:21 EDT 2011


i wanted to share a couple of loose thoughts on the standard practices
for PKGBUILDs ... specifically development ones ... with an immediate
focus on git ones (for this instance, but applies to any)

awhile back i tried to make a super cheap way of doing git checkouts for builds:

https://bbs.archlinux.org/viewtopic.php?id=86366
(and this would have worked if i would have thought to make git use
itself as an alternative object pool ...)

... the motivation of this message is to encourage adding some
routines to the makepkg library, similar to msg/msg2/warning/etc, to
handle development checkouts.  basically i see people using _gitroot
and _gitname for different stuff ... namely _gitname.  i'd like to see
a routine that pulls the repo to a known location so it doenst get
blown away every update, and so the user doesnt have to manually
manage this.  everyone does it differently, puts the repository in all
sorts of different locations/etc, and it just isnt very pretty IMO.

... so, i suggest a permanent setup based off my out setup and
experiences, in reference of this PKGBUILD in particular:

http://aur.archlinux.org/packages/pyjamas-engine-pythonwebkit/PKGBUILD

) allow _git* variables to be altered by the environment
) add _gitspec variable which supercedes _gitname for checkout stage
(so you can do relative checkouts or SHA1 based checkouts)

... in the routine ...

) use a targeted fetch command instead of a blind clone.  not only is
this more flexible, but it can significantly reduce download size (for
kernel its a ~50% reduction IIRC).  the methoud from my builds are
very simple, greatly resembling --mirror mode in git, but ONLY for the
exact branch you are building
) store repositories in a known list of locations.  my PKGBUILD searches these:

/var/abs/PKGBUILD.devel/${pkgname}.git
/var/abs/local/PKGBUILD.devel/${pkgname}.git
${SRCDEST}/PKGBUILD.devel/${pkgname}.git
${startdir}/PKGBUILD.devel/${pkgname}.git
~/PKGBUILD.devel/${pkgname}.git

... and chooses the first one that already exists (PKGBUILD.devel
location only BEFORE appending repo name), OR the first one with write
access to parent dir (it will create PKGBUILD.devel for you if it
can).  i personally used the first one (/var/abs/PKGBUILD.devel), with
`chmod 1777` to sticky bit it like /tmp.  this not only allows for
reuse, but also allows for safely+easily+consistently bind-mounting
into a chroot for use with mkchrootpkg ...

) use a proxy mechanism in the event a repository is found but it is
read-only.  this lets you read-only bind mount a repo (think
mkchrootpkg), and it will simply create a new repository, copy the
refs, and setup the object directory as an alternative for the proxy
repo.  thus the proxy has the ability to download new objects as
needed, but starts from the same spot as the bind-mounted repo.

these techniques save me a great deal of time and pain.  that package
in particular is a custom webkit build, requiring over 1GB to clone
... mkchrootpkg tried to blow it away once, bit it was read-only :-)
... i intend to possibly improve on this process by reducing a
targeted fetch to a *shallow*  targeted fetch , ie. the minimum amount
of objects required to build.  i have seen "clones" go down to 50%,
sometimes even 10% or less, by using this over a naive `git clone
$xyz`.

i'd like to see *something* adopted so we can end the madness :-) ...
below are a couple key excerpts from the noted PKGBUILD, for your
reference.

C Anthony

-------------------------------------------------
[locate repo]
-------------------------------------------------

            # Devel directory fragment
            : ${_dir_devel:=PKGBUILD.devel}

            # Local/custom repo?
            if [[ -z ${_gitrepo} ]]; then
                search=(/var/abs{,/local}/"${_dir_devel}"
                        "${SRCDEST%%/}/${_dir_devel}"
                        "${startdir%%/}/${_dir_devel}"
                        ~/"${_dir_devel}")
                for d in "${search[@]%%/}"; do
                    if [[ -e ${d} ]] || [[ ! -e ${d} && -w ${d%/*} ]]; then
                        mkdir -p "${d}" 2>&- &&
                            _gitrepo_proxy="${srcdir}/${pkgname}.git" &&
                            _gitrepo="${d}/${pkgname}.git" &&
                            break
                    fi
                done
            fi

-------------------------------------------------
[create new repo, or new proxy, if needed]
-------------------------------------------------

    mkdir -p "${w}"
    if [[ ! -e ${g}/objects ]]; then
        msg "[git] Creating NEW repository ... "
        git --git-dir="${g}" --work-tree="${w}" init
    elif [[ ! -w ${g}/objects ]]; then
        warning "[git] Repository read-only, setting up proxy ... "
        git --git-dir="${_gitrepo_proxy}" --work-tree="${w}" init
        echo "${g}/objects" > "${_gitrepo_proxy}/objects/info/alternates"
        cp -r "${g}/refs" "${_gitrepo_proxy}"
        g="${_gitrepo_proxy}"
    fi

-------------------------------------------------
[perform targeted fetch]
-------------------------------------------------

    msg "[git] Syncing ... ${_gitroot} -> ${g}"
    git --git-dir="${g}" --work-tree="${w}" fetch -fu "${_gitroot}"
"+${_gitname}:${_gitname}"
    msg "[git] Reading ... ${_gitspec:-${_gitname}} -> ${w}"
    git --git-dir="${g}" --work-tree="${w}" read-tree --reset -u
"${_gitspec:-${_gitname}}"

-------------------------------------------------


More information about the arch-general mailing list