rpm
packages for mass deploymentpuppet
,
yum
, or apt
for
a large number of Linux
hosts via a boot
sequence that builds rpm
packages.
If you want to build packages for HPUX
or Solaris
then you can use this as a guide
for the order you need to build and install the packages.
make
(1), and have
a working knowledge of rsync
and
rpm
or yum
.
You may build the first package, msrc_base
,
as a mortal user. To build any that follow you must be
able to run rpm -i
as the superuser (aka
root
). For example you must install
msrc_base
before you can build
install_base
on which every other package and
product depends.
You can build all the this in a chroot
, but that
is pretty hard to setup; so I'm not going to explain it here.
You may be able to install the packages in a mortals account, but I
would just use the relocate
spells for that, and not bother with the rpm
build process, since that processes is better automated.
msrc
toolstar
-balls
from a local rsync
service.
rsync
service provides the same source tar
-balls (as on my
website) to all the build engines in the build cluster.
This cache allows the build cluster to:
rsync
services
msrcmux
and muxcat
via
the relocation recipe. That system
moves at the same speed as the local master source cache, by virtue of
using the current files from /usr/msrc
in real-time.
Rsync
is pretty safe to run as a mortal, but
I don't like to run it a the superuser if I don't have to. I can always
use op
, sudo
,
mock
or cut-and-paste to escalate the build or install process, as needed.
make
recipes in the
build process, then you can't trust the binary files produced by them.
That's all there is to it. To be prudent we build as a mortal to prevent
accidental shootings when testing new builds.
Rpm
works pretty much the same way everywhere.
dpkg
or other packaging tools.
Some convert from one format to another (e.g. alien
).
But rpm
is old enough and buggy enough that it
works (breaks) the same way most everywhere. So we'll use that here.
With all that said, it is time to make some
rpm
packages.
rsync
for the source tar
archivestar
-balls. For the time being use the source tar
archives from my ftp
site. A you update
the code you'll roll new ones with level2s
and
level3s
, for your own applications and maybe mine.
If you don't have such a service build one now. We need to update
(or install) /etc/rsyncd.conf
.
Later you'll push
this into a level 2 configuration directory, which will install the correct
rsyncd.conf
across all your hosts.
You'll need to enable rsync
from boot with
chkconfig
, so use sudo
or su
as you see fit. If you can't
do that, you can start a mortal instance on a high port (like 2081).
In that case you'll need to
add -p
2081
option to
the rsync
commands below.
Here is a sample one I use to get started.
Also as the superuser you may need to# Allow build platforms to pull master source down -- ksb # ... #motd file = /usr/msrc/.motd syslog facility = local6 [msrc] list = yes comment = NPCGuild master source tree path = /usr/msrc uid = ftp gid = ftp read only = true
mkdir
the
root of your master source cache. I used /usr/msrc
,
but you could pick another place. I'm going to assume that you made it
group writable, so we can update the source tar
-balls as you. I'm also going
to assume you downloaded them into a temporary directory already.
You may have to copy the source
archives into place, I'll just mv
them, as we
don't need the copies in /tmp
.
Pick a group (like source
) that limits changes
to the admins or maintainers and use that group to do your work.
That way we don't have to carry a loaded root
shell around all the time.
Start# mkdir -m2775 -p /usr/msrc/level2 /usr/msrc/level3 # chgrp source /usr/msrc /usr/msrc/level2 /usr/msrc/level3 ... mortal$ mv -i *_base-*.tgz* /usr/msrc/level3/ mortal$ mv -i *-*.tgz /usr/msrc/level2/
rsync
under --daemon
, or
chkconfig
it on
:
# chkconfig rsync on
The check to see that it lists the packages you need:
mortal$ rsync localhost::msrc/level2 2>&1 | less list of compressed package archives mortal$ rsync localhost::msrc/level3 2>&1 | less list of compressed product archives mortal$
This also helps later when you build your own packages and products:
just add the tar
archives to the correct
directory, and use the same techniques in this spell to build local
rpm
packages.
rsync
repository to
a build host. It could be the same host, but usually there are
build servers for different platform combinations (OS, hardware platform,
and patch level).
On the build server of your choice we'll boot-strap
the rpm
build, from just the source
tar
-balls and
the rsync
repository. Get a mortal shell on
a build host then make a scratch directory to work under, then
set MPS
to the name of
the rsync
repository:
sulaco$ ssh build mortal$ mkdir /tmp/rpB mortal$ cd /tmp/rpB mortal$ export MPS=msrc.example.com mortal$ rsync $MPS:: ...rsync list of modules
msrc_base
package under the
rpmboot
subdirectory, so we'll have to dig
a little to fetch it. It uses the same trick the relocate spell uses to
allow it to work
both
with and without being run through m4
.
Fetch the msrc_base
package to find and
unpack the recipe file.
To fetch the recipe file, use this (longish) spell:
We'll just check to see that we got the correct recipe, the default target in the recipe outputs some clues:mortal$ rsync -L $MPS::msrc/level3/msrc_base-Two.tgz ./my.tgz mortal$ tar xzf my.tgz --exclude=Pkgs \*/rpmboot/Makefile.host mortal$ mv msrc_base*/rpmboot/Makefile.host Makefile mortal$ rmdir -p msrc_base*/*/ mortal$ rm -f my.tgz mortal$ ls Makefile
mortal$ make This assumes you have an rsync repo with the source tarballs at ${MPS} Presently makes value for MPS is "msrc.example.com" As the superuser "make root.setup" for required rpm support directories. ...
If you got that output you are ready to rock and roll.
screen
or xterm
sessions open on the build host:
tar
-balls to build
rpm
as it is built
screen
(tmux
) now,
or create 3 xterm
's. In of them is a superuser
shell (or add sudo
to each command); the other 2
are mortal shells one in /tmp/rpB
, one in
the mortal's home directory.
In the superuser shell we'll make the setup target to root:
If that still vetches about missing gdbm-devel, then you can't build# make -f /tmp/rpB/Makefile root.setup ... : root actions done
oue
.
As the mortal we need to setup an rpm
build environment. I started with an empty home and added my shell
startup files, and my ssh
directory.
Make sure $MPS
is set, it might have been lost
when you started a new shell. Then we'll setup for rpm
:
I symbolic linkbuild$ export MPS=msrc.example.com build$ mkdir -p rpmbuild/SOURCES build$ touch /home/ksb/.rpmmacros build$ ln -s rpmbuild rpm build$ make -f /tmp/rpB/Makefile prep build$ ls _bin/ _ftp/ my.tgz build$ ls _bin level2s level3s muxcat
rpmbuild
to
rpm
, because some older versions of
rpm
are totally broken.
Move back to the mortal shell in /tmp/rpB
to
rebuild the setup
target to assure that
everything is set. Then we as the recipe to fetch all the
tar
-balls we need:
mortal$ make -f /tmp/rpB/Makefile MPS=$MPS fetch | tee DoIt.sh | ${PAGER:-less} cp _ftp/msrc_base-2.44.tgz /root/rpmbuild/SOURCES/ level3s rpm msrc_base-2.44.tgz cp _ftp/install_base-8.67.tgz /root/rpmbuild/SOURCES/ level3s rpm install_base-8.67.tgz cp _ftp/oue-2.31.tgz /root/rpmbuild/SOURCES/ level2s rpm oue-2.31.tgz cp _ftp/... ...many more pairs here... : Silent Running : fetch complete
If you don't get the "fetch complete" message, then a package
failed to download. Set 'TRACE=set -x;' before the
MPS
assignment on the make
command line to trace the failure. Fix that and remove
the whole _ftp
subdirectory before you retry
this process; this forces the download of all the packages again, which is
what you want to get a complete manifest of commands in
DoIt.sh
.
After you've paged to the end, you have captured the output in
DoIt.sh
. That script, when run as the
superuser, looks like it builds the packages. But it
won't work as given, because it doesn't know the name of
the packages it built to rpm -i
or
yum localinstall
install them.
This is because rpm
builds package names and
paths in different ways under various Linux distributions. I gave up
trying to glob match all the possible places, names, and formats.
So we are going to use your human pattern recognition to find the name
of the package in the output from the build process.
As a bonus we'll build as a mortal and only install a the superuser.
tar
-ballscp
command and the level2s
command into the mortal
shell to build the package and output the name.
Then I rpm -i
(or yum localinstall
)
that name in the superuser shell.
Not that I'm proud of that solution, but it works and it lets you debug anything that goes wrong as you build the packages.
In the mortal shell we just less
the
DoIt.sh
list:
We are just going to use that as an ordered list of commands for the mortal shell. So copy the first line and we're off!build$ less ~/DoIt.sh level3s rpm msrc_base-2.44.tgz level3s rpm install_base... level3s rpm oue-...
Here is an example session snippet where I'm building everything:
I copy the downloaded files to SOURCES
, but it
looks like modern rpm
doesn't need that step
anymore.
The pair of "Wrote: /home/ksb/..." lines are the important part. That's the source and binary package path. Note that the error message gets the path to the new package wrong: it includes thebuild$ cp _ftp/* rpmbuild/SOURCES/ build$ level3s rpm _ftp/msrc_base-2.44.tgz 1144 -rw-r--r-- 1 ksb ksb 1168232 Feb 4 13:29 /home/ksb/rpmbuild/SOURCES/msrc_base-2.44.tgz Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.okwgpg + umask 022 + cd /home/ksb/rpmbuild/BUILD ... lots of configure and compile output here ... Requires(pre): /bin/sh Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/ksb/rpmbuild/BUI LDROOT/msrc_base-2.44-1.el6.el6.x86_64 Wrote: /home/ksb/rpmbuild/SRPMS/msrc_base-2.44-1.el6.el6.src.rpm Wrote: /home/ksb/rpmbuild/RPMS/x86_64/msrc_base-2.44-1.el6.el6.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.MLdrRs + umask 022 + cd /home/ksb/rpmbuild/BUILD + cd msrc_base-2.44 + exit 0 error: open of /home/ksb/rpmbuild/RPMS/x86_64/_ftp/msrc_base-2.44-1.el6.x86_64.rpm failed: \ No such file or directory build$
_ftp
prefix and the actual
path has an (extra?) .el6
in it.
So you need to copy the path in the second "Wrote" line.
(This is the part that is hard to automate across Linux distributions and
versions of rpm
.)
To continue the build process you'll need to install the binary package with something like:
Then cut the next build command from the pager from, paste it into the build session. Then find the "Wrote.*RPMS" path, install it from the superuser shell with# yum localinstall /home/ksb/rpmbuild/RPMS/x86_64/msrc_base-2.44-1.el6.el6.x86_64.rpm
yum
or rpm
.
# yum localinstall /home/ksb/rpmbuild/RPMS/x86_64/install_base-8.69-1.el6.el6.x86_64.rpm
There are 2 base packages that are build with level3s
,
the other products are built with level2s
, because
they are separate products. There are other packages, but we'll not install
them yet. Those are usually not installed, or may be installed later:
entomb_base
file entombing I like a lot
conserver
is less popular than the conserver.com release
qsys
is an old job scheduler
rpm
as it is built.
After about 60 iterations you should have all the products built and
installed. It shouldn't take long at all. I use
a screen
and page flip between the 3 sessions.
Package build failures can be debugged in another screen. If the first
4 build you are more than likely OK.
rpm
's you just built to another
host. If you like you can rebuild with a gpg
key
to make signed packages. But I've never gotten that to work, sadly.
When you are done you should cleanup the download cache:
# cd .. # make -f /tmp/rpB/Makefile reset rm -f _bin/muxcat _bin/level2s _bin/level3s my.tgz rm -rf _bin _ftp # exit Shift to your mortal shell here. mortal$ pwd /tmp/rpB mortal$ ls Makefile mortal$ cd .. && rm -rf rpB mortal$ exit Shift to your mortal shell here. build$ exit
yum
repositoryroot
's home directory. Put those where you
need them: some apt
, yum
, or
other package repository.
Move along to the next build host, collect the fruits of your labors and
stash them in a yum
repo.
I would copy the recipe file from /tmp/rpB/Makefile
to the next host (as it is the same). Only do that after you've gotten
used to the process. Also check that the source tar
has not changed on you.
rsync
daemon, then you can kill it (or chkconfig
it
off
).
I leave the rsync
running, because other engineers
may want to build test packages at any time.
$Id: rpmboot.html,v 2.4 2014/02/04 20:22:15 ksb Exp $ by ksb.