pbulk-HOWTO
From NetBSD Wiki
pbulk is the parallel bulk build framework that has been added to pkgsrc in 2007. This article describes how to use it as an unprivileged (non-root) user.
Contents |
Environment
I am doing the pbulk as the user bulk on my local system.
I have the following directories:
- /home/scratch/distfiles: Lots of distfiles that have already been downloaded.
- /home/bulk: My home directory.
- /home/scratch/pbulk/packages: The binary packages will end up here.
- /home/scratch/pbulk/log: The build logs will end up here.
Installing the outer pkgsrc
First, you need to create the "outer" pkgsrc installation that contains all the tools needed for the bulk build. The packages of the "real" pkgsrc installation will be created independently of the outer installation.
cd $PKGSRCDIR
./bootstrap/bootstrap \
--unprivileged \
--prefix=$HOME/pbulk-outer \
--workdir=/tmp/pbulk-outer
rm -rf /tmp/pbulk-outer
Now you should edit the ~/pbulk-outer/etc/mk.conf file so that it fits your local environment. I added these lines:
DISTDIR= /home/scratch/distfiles WRKOBJDIR= /tmp/pbulk-outer PKG_DEVELOPER= yes
The next step is to install the needed tools into the outer pkgsrc installation:
(cd $PKGSRCDIR/pkgtools/pbulk && ~/pbulk-outer/bin/bmake install)
Configuring pbulk
The pbulk package has a configuration file in ~/pbulk-outer/etc/pbulk.conf which also needs to be adjusted to the local environment:
- base_url needs to be adjusted, although the pbulk code should not need to know it at all, maybe except for generating the mail that the report has been completed.
- bootstrapkit is a tar archive containing the "base" pkgsrc installation. It looks as if we have to create this file manually. It will soon be created as ~/bootstrapkit.
- I also had to change almost all pathnames, so that they lie within /home/bulk, since I don't have write access for the other locations.
Creating the bootstrap kit
$PKGSRCDIR/bootstrap/bootstrap --unprivileged --workdir=/tmp/pbulk-inner rm -rf /tmp/pbulk-inner
Now I edited ~/pkg/etc/mk.conf and added the following:
DISTDIR= /home/scratch/distfiles WRKOBJDIR= /tmp/pbulk PKG_DEVELOPER= yes PACKAGES= /home/scratch/pbulk/packages/NetBSD/i386/head
(cd / && pax -w -z -f ~/bootstrap.tar.gz home/bulk/pkg)
Let's start
This seems to be all that is needed for preparing the bulk build. So let's try to start it: If you see errors during the scanning phase "bmake: permission denied" you may need to set your $PATH to mention ~/pbulk-outer/bin.
~/pbulk-outer/bin/bulkbuild
... some hours later ...
... pbulk-scan: Duplicate package: websvn-1.61nb7 pbulk-scan: Duplicate package: websvn-1.61nb7 pbulk-scan: Duplicate package: websvn-1.61nb7 pbulk-scan: Duplicate package: websvn-1.61nb7 pbulk-scan: Duplicate package: websvn-1.61nb7 pbulk-scan: Duplicate package: zopeedit-0.7nb1 Resolving... Building... Starting build of digest-20070803 Successfully finished build of digest-20070803 Starting build of checkperms-1.7 Failed to build of checkperms-1.7 Starting build of libkver-0.6 Successfully finished build of libkver-0.6 Starting build of pkg_install-info-4.5nb3 Successfully finished build of pkg_install-info-4.5nb3 Starting build of rdigest-20070803 Failed to build of rdigest-20070803 Building pkg_summary... cd: can't cd to /home/scratch/pbulk/packages/NetBSD/i386/head/All
That doesn't look good. It's because I didn't keep packages in pbulk.conf and PACKAGES in mk.conf in sync. My fault. :(
Resuming an already started pbulk build
Since I don't want to regenerate the database, I need to find a way to skip the generation phase. It looks as if the following works, but it doesn't feel right to me:
$ : "Is the following necessary at all?"
$ rm /home/scratch/pbulk/log/{pbuild,error,success}
$ ~/pbulk-outer/libexec/pbulk/build
The packages are being built now, and I am surprised how many of them are built successfully.
