Build rpm packages for mass deployment

Tactics to deploy my tools with puppet, yum, or apt for a large number of Linux hosts via a boot sequence that builds rpm packages. If you want to build packages for HPUX or Solaris then you can use this as a guide for the order you need to build and install the packages.

To understand this document

This document assumes you are familiar with the standard UNIX™ recipe builder, make(1), and have a working knowledge of rsync and rpm or yum. You may build the first package, msrc_base, as a mortal user. To build any that follow you must be able to run rpm -i as the superuser (aka root). For example you must install msrc_base before you can build install_base on which every other package and product depends.

You can build all the this in a chroot, but that is pretty hard to setup; so I'm not going to explain it here.

You may be able to install the packages in a mortals account, but I would just use the relocate spells for that, and not bother with the rpm build process, since that processes is better automated.

To create locally built RPMs for the msrc tools

I'm assuming some things here with may sound odd, but follow along and it will make more sense at the end.
We always fetch tar-balls from a local rsync service.
This rsync service provides the same source tar-balls (as on my website) to all the build engines in the build cluster. This cache allows the build cluster to: These are all good things, and you should love it as much as I do. If you need to build from the current (unstable) sources, use msrcmux and muxcat via the relocation recipe. That system moves at the same speed as the local master source cache, by virtue of using the current files from /usr/msrc in real-time.
We run the fetch process as a mortal login.
Rsync is pretty safe to run as a mortal, but I don't like to run it a the superuser if I don't have to. I can always use op, sudo, mock or cut-and-paste to escalate the build or install process, as needed.
We build as a mortal, or as the superuser.
If you don't trust the make recipes in the build process, then you can't trust the binary files produced by them. That's all there is to it. To be prudent we build as a mortal to prevent accidental shootings when testing new builds.
We install each package as we go (before we build the next)
I ordered the builds so that we always build prerequisite packages first, then prerequisite products then the leaf products.
The packages are to be used elsewhere.
The build host is not the (only) host that will consume the packages, because the build host is just a builder, not a durable asset (see next).
Build hosts are disposable assets.
Rebuild them from scratch after you use them to avoid finger files someone may have put in-place to test new features or bug fixes.
Rpm works pretty much the same way everywhere.
Some people use dpkg or other packaging tools. Some convert from one format to another (e.g. alien). But rpm is old enough and buggy enough that it works (breaks) the same way most everywhere. So we'll use that here.
The clues you get from reading this recipe should translate into any other packaging application.

With all that said, it is time to make some rpm packages.

Enable rsync for the source tar archives

We need to enable local build hosts to fetch the locally blessed source tar-balls. For the time being use the source tar archives from my ftp site. A you update the code you'll roll new ones with level2s and level3s, for your own applications and maybe mine.

If you don't have such a service build one now. We need to update (or install) /etc/rsyncd.conf. Later you'll push this into a level 2 configuration directory, which will install the correct rsyncd.conf across all your hosts.

You'll need to enable rsync from boot with chkconfig, so use sudo or su as you see fit. If you can't do that, you can start a mortal instance on a high port (like 2081). In that case you'll need to add -p2081 option to the rsync commands below.

Here is a sample one I use to get started.

# Allow build platforms to pull master source down -- ksb
# ...

#motd file = /usr/msrc/.motd
syslog facility = local6

[msrc]
list = yes
comment = NPCGuild master source tree
path = /usr/msrc
uid = ftp
gid = ftp
read only = true
Also as the superuser you may need to mkdir the root of your master source cache. I used /usr/msrc, but you could pick another place. I'm going to assume that you made it group writable, so we can update the source tar-balls as you. I'm also going to assume you downloaded them into a temporary directory already.

You may have to copy the source archives into place, I'll just mv them, as we don't need the copies in /tmp. Pick a group (like source) that limits changes to the admins or maintainers and use that group to do your work. That way we don't have to carry a loaded root shell around all the time.

# mkdir -m2775 -p /usr/msrc/level2 /usr/msrc/level3
# chgrp source /usr/msrc /usr/msrc/level2 /usr/msrc/level3
...
mortal$ mv -i *_base-*.tgz* /usr/msrc/level3/
mortal$ mv -i *-*.tgz /usr/msrc/level2/
Start rsync under --daemon, or chkconfig it on:
# chkconfig rsync on

The check to see that it lists the packages you need:

mortal$ rsync localhost::msrc/level2 2>&1 | less
list of compressed package archives
mortal$ rsync localhost::msrc/level3 2>&1 | less
list of compressed product archives
mortal$ 

This also helps later when you build your own packages and products: just add the tar archives to the correct directory, and use the same techniques in this spell to build local rpm packages.

The recipe file for a build host

At this point we shift from the rsync repository to a build host. It could be the same host, but usually there are build servers for different platform combinations (OS, hardware platform, and patch level).

On the build server of your choice we'll boot-strap the rpm build, from just the source tar-balls and the rsync repository. Get a mortal shell on a build host then make a scratch directory to work under, then set MPS to the name of the rsync repository:

sulaco$ ssh build
mortal$ mkdir /tmp/rpB
mortal$ cd /tmp/rpB
mortal$ export MPS=msrc.example.com
mortal$ rsync $MPS::
...rsync list of modules

Unpack the recipe file

The platform recipe file we need is in the msrc_base package under the rpmboot subdirectory, so we'll have to dig a little to fetch it. It uses the same trick the relocate spell uses to allow it to work both with and without being run through m4.

Fetch the msrc_base package to find and unpack the recipe file. To fetch the recipe file, use this (longish) spell:

mortal$ rsync -L $MPS::msrc/level3/msrc_base-Two.tgz ./my.tgz
mortal$ tar xzf my.tgz --exclude=Pkgs \*/rpmboot/Makefile.host
mortal$ mv msrc_base*/rpmboot/Makefile.host Makefile
mortal$ rmdir -p msrc_base*/*/
mortal$ rm -f my.tgz
mortal$ ls
Makefile
We'll just check to see that we got the correct recipe, the default target in the recipe outputs some clues:
mortal$ make
This assumes you have an rsync repo with the source tarballs at ${MPS}
Presently makes value for MPS is "msrc.example.com"
As the superuser "make root.setup" for required rpm support directories.
...

If you got that output you are ready to rock and roll.

Let's go

I'd keep 3 screen or xterm sessions open on the build host: So start a screen (tmux) now, or create 3 xterm's. In of them is a superuser shell (or add sudo to each command); the other 2 are mortal shells one in /tmp/rpB, one in the mortal's home directory.

In the superuser shell we'll make the setup target to root:

# make -f /tmp/rpB/Makefile root.setup
...
: root actions done
If that still vetches about missing gdbm-devel, then you can't build oue.

As the mortal we need to setup an rpm build environment. I started with an empty home and added my shell startup files, and my ssh directory. Make sure $MPS is set, it might have been lost when you started a new shell. Then we'll setup for rpm:

build$ export MPS=msrc.example.com
build$ mkdir -p rpmbuild/SOURCES
build$ touch /home/ksb/.rpmmacros
build$ ln -s rpmbuild rpm
build$ make -f /tmp/rpB/Makefile prep
build$ ls
_bin/  _ftp/  my.tgz
build$ ls _bin
level2s  level3s  muxcat
I symbolic link rpmbuild to rpm, because some older versions of rpm are totally broken.

Move back to the mortal shell in /tmp/rpB to rebuild the setup target to assure that everything is set. Then we as the recipe to fetch all the tar-balls we need:

mortal$ make -f /tmp/rpB/Makefile MPS=$MPS fetch | tee DoIt.sh | ${PAGER:-less}
cp _ftp/msrc_base-2.44.tgz /root/rpmbuild/SOURCES/
level3s rpm msrc_base-2.44.tgz
cp _ftp/install_base-8.67.tgz /root/rpmbuild/SOURCES/
level3s rpm install_base-8.67.tgz
cp _ftp/oue-2.31.tgz /root/rpmbuild/SOURCES/
level2s rpm oue-2.31.tgz
cp _ftp/...
...many more pairs here...
: Silent Running
: fetch complete

If you don't get the "fetch complete" message, then a package failed to download. Set 'TRACE=set -x;' before the MPS assignment on the make command line to trace the failure. Fix that and remove the whole _ftp subdirectory before you retry this process; this forces the download of all the packages again, which is what you want to get a complete manifest of commands in DoIt.sh.

After you've paged to the end, you have captured the output in DoIt.sh. That script, when run as the superuser, looks like it builds the packages. But it won't work as given, because it doesn't know the name of the packages it built to rpm -i or yum localinstall install them.

This is because rpm builds package names and paths in different ways under various Linux distributions. I gave up trying to glob match all the possible places, names, and formats. So we are going to use your human pattern recognition to find the name of the package in the output from the build process.

As a bonus we'll build as a mortal and only install a the superuser.

Building from the downloaded tar-balls

I fell-back to the hackery: I just cut+paste the cp command and the level2s command into the mortal shell to build the package and output the name. Then I rpm -i (or yum localinstall) that name in the superuser shell.

Not that I'm proud of that solution, but it works and it lets you debug anything that goes wrong as you build the packages.

In the mortal shell we just less the DoIt.sh list:

build$ less ~/DoIt.sh
level3s rpm msrc_base-2.44.tgz
level3s rpm install_base...
level3s rpm oue-...
We are just going to use that as an ordered list of commands for the mortal shell. So copy the first line and we're off!

Here is an example session snippet where I'm building everything: I copy the downloaded files to SOURCES, but it looks like modern rpm doesn't need that step anymore.

build$ cp _ftp/* rpmbuild/SOURCES/
build$ level3s rpm _ftp/msrc_base-2.44.tgz
1144 -rw-r--r-- 1 ksb ksb 1168232 Feb  4 13:29 /home/ksb/rpmbuild/SOURCES/msrc_base-2.44.tgz

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.okwgpg
+ umask 022
+ cd /home/ksb/rpmbuild/BUILD
... lots of configure and compile output here ...
Requires(pre): /bin/sh
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/ksb/rpmbuild/BUI
LDROOT/msrc_base-2.44-1.el6.el6.x86_64
Wrote: /home/ksb/rpmbuild/SRPMS/msrc_base-2.44-1.el6.el6.src.rpm
Wrote: /home/ksb/rpmbuild/RPMS/x86_64/msrc_base-2.44-1.el6.el6.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.MLdrRs
+ umask 022
+ cd /home/ksb/rpmbuild/BUILD
+ cd msrc_base-2.44
+ exit 0
error: open of /home/ksb/rpmbuild/RPMS/x86_64/_ftp/msrc_base-2.44-1.el6.x86_64.rpm failed: \
	No such file or directory
build$ 
The pair of "Wrote: /home/ksb/..." lines are the important part. That's the source and binary package path. Note that the error message gets the path to the new package wrong: it includes the _ftp prefix and the actual path has an (extra?) .el6 in it. So you need to copy the path in the second "Wrote" line. (This is the part that is hard to automate across Linux distributions and versions of rpm.)

To continue the build process you'll need to install the binary package with something like:

# yum localinstall /home/ksb/rpmbuild/RPMS/x86_64/msrc_base-2.44-1.el6.el6.x86_64.rpm
Then cut the next build command from the pager from, paste it into the build session. Then find the "Wrote.*RPMS" path, install it from the superuser shell with yum or rpm.
# yum localinstall /home/ksb/rpmbuild/RPMS/x86_64/install_base-8.69-1.el6.el6.x86_64.rpm

There are 2 base packages that are build with level3s, the other products are built with level2s, because they are separate products. There are other packages, but we'll not install them yet. Those are usually not installed, or may be installed later:

Continue with the level 2 products

Just keep up the copy+paste for each line in the list. Be sure to install each rpm as it is built.

After about 60 iterations you should have all the products built and installed. It shouldn't take long at all. I use a screen and page flip between the 3 sessions. Package build failures can be debugged in another screen. If the first 4 build you are more than likely OK.

Cleanup

First save the rpm's you just built to another host. If you like you can rebuild with a gpg key to make signed packages. But I've never gotten that to work, sadly.

When you are done you should cleanup the download cache:

# cd ..
# make -f /tmp/rpB/Makefile reset
rm -f _bin/muxcat _bin/level2s _bin/level3s my.tgz
rm -rf _bin _ftp
# exit
	Shift to your mortal shell here.
mortal$ pwd
/tmp/rpB
mortal$ ls
Makefile
mortal$ cd .. && rm -rf rpB
mortal$ exit
	Shift to your mortal shell here.
build$ exit

You are done with this build host

Take a break.

Export your work to your yum repository

That leaves only the packages (source and binary) in the cache under root's home directory. Put those where you need them: some apt, yum, or other package repository.

Move along to the next build host, collect the fruits of your labors and stash them in a yum repo. I would copy the recipe file from /tmp/rpB/Makefile to the next host (as it is the same). Only do that after you've gotten used to the process. Also check that the source tar has not changed on you.

After you've done all the build hosts

When they are all done, if you started an rsync daemon, then you can kill it (or chkconfig it off).

I leave the rsync running, because other engineers may want to build test packages at any time.

See also

The mortal relocation build process.


$Id: rpmboot.html,v 2.4 2014/02/04 20:22:15 ksb Exp $ by .