The Lazy Engineer’s Guide to Using Mock

In my previous post, I laid out four rules to yield superior RPMs and reduce headaches in their deployment. In this post, I’ll go into more detail around the infrastructure we’ve put in place here at Bronto to make following the rules easy, especially rule three: building safely using Mock.

Larry Wall advocates that laziness is a primary virtue of good technologists. As a result, we want to do as little work as possible, taking advantage of the work of others whenever we can. At Bronto, we rely on the efforts of the CentOS Project for the operating system for our production servers and Puppet Labs for the software that manages those servers. For building packages, the path of least resistance would be a puppet module that manages building RPMs using Mock on RHEL-like machines, with bonus points if we don’t have to write the module ourselves. Fortunately for us, a puppet module already exists that does pretty much what we want. It provides a huge head start to building RPMs, and we use it as a base for our build servers with only minimal modifications.

Out of the box, Jeff McCune’s mockbuild puppet module sets up the mockbuild user and group, installs the needed packages, and sets up the usual RPM-building directories (BUILD, RPMS, SRPMS, SOURCES, and SPECS) in mockbuild’s home directory of /var/lib/mockbuild. With normal usage, you’d copy your spec file into SPECS, your code into SOURCES, and then run mock to actually build the RPMs, doing this all by hand.

Obviously, we want to put some automation around this process to increase efficiency and decrease error rates. We also want to make sure that our spec files and local patches are checked into version control and that we have a known place for downloaded source code. Since we have a well-defined code review and deployment process for our puppet code, we decided to check our code in there and deploy to the build servers using puppet. We also have a puppet file server set up to hold the original sources since we don’t want to check tarballs into our version control system. We accomplished this by adding the following puppetry:

  file { "${home}/sources":
    ensure  => directory,
    owner   => 'mockbuild',
    group   => 'mockbuild',
    mode    => '0644',
    recurse => true,
    purge   => true,
    force   => true,
    source  => 'puppet:///software/mockbuild/sources/',
  }

  file { "${home}/files":
    ensure  => directory,
    recurse => true,
    purge   => true,
    force   => true,
    source  => "puppet:///modules/mockbuild/${::lsbmajdistrelease}",
  }

This takes care of getting the files onto the build servers, but we still need to copy the spec files and the sources into the right places to actually build the RPMs. We do this via a small Makefile in /var/lib/mockbuild:

##
## Automation for building RPMs.
##
## Synopsis:
##
##  sudo -Hu mockbuild make <project_name>
##  sudo -Hu mockbuild make clean
##
## To create i386 RPMs on x86_64 hosts, you must specify an i386 build environment
##
##  sudo -Hu mockbuild make <project_name> ENV=epel-5-i386
##
## Author:
##
##  Kevin Kreamer <kevin.kreamer@bronto.com>
##

MOCKBUILD_HOME=/var/lib/mockbuild
ENV=default

RPMS=$(shell find $(MOCKBUILD_HOME)/files -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)
.PHONY: clean $(RPMS)

$(RPMS):
        /usr/bin/find $(MOCKBUILD_HOME)/sources/$@ $(MOCKBUILD_HOME)/files/$@ -maxdepth 1 -type f -not -name \*.spec -exec cp {} $(MOCKBUILD_HOME)/rpmbuild/SOURCES \;
        cp $(MOCKBUILD_HOME)/files/$@/$@.spec $(MOCKBUILD_HOME)/rpmbuild/SPECS
        sudo /usr/sbin/mock -r $(ENV) --resultdir $(MOCKBUILD_HOME)/rpmbuild/SRPMS/ --buildsrpm --spec $(MOCKBUILD_HOME)/rpmbuild/SPECS/$@.spec --sources $(MOCKBUILD_HOME)/rpmbuild/SOURCES/
        sudo /usr/sbin/mock -r $(ENV) --resultdir $(MOCKBUILD_HOME)/rpmbuild/RPMS --rebuild $(MOCKBUILD_HOME)/rpmbuild/SRPMS/$@*.src.rpm

clean:
        /bin/rm -Rf $(MOCKBUILD_HOME)/rpmbuild/SOURCES/* $(MOCKBUILD_HOME)/rpmbuild/SPECS/* $(MOCKBUILD_HOME)/rpmbuild/SRPMS/* $(MOCKBUILD_HOME)/rpmbuild/BUILD/*
        /bin/rm -f $(MOCKBUILD_HOME)/rpmbuild/RPMS/*.src.rpm $(MOCKBUILD_HOME)/rpmbuild/RPMS/*.log

with its associated puppetry:

  file { "${home}/Makefile":
    ensure => 'present',
    owner  => 'mockbuild',
    group  => 'mockbuild',
    mode   => '0644',
    source => 'puppet:///modules/mockbuild/Makefile',
  }

The Makefile assumes that each spec file lives in a subdirectory of the same name along with its associated sources. It copies them into SPECS and SOURCES as appropriate and then kicks off mock to build the RPMs. make clean is provided to clean out the rpmbuild directory for building other RPMs. The Makefile is written extensibly to automatically handle additional subdirectories with additional spec files.

With this infrastructure in place, building an RPM with all of Mock’s goodness is just a simple make command away.

Four Rules of Building RPMs

At Bronto, we’re heavily invested in open source technologies. We have dozens of MySQL shards, we run Hadoop and HBase, and we’ve built out a production infrastructure on CentOS.

One of the benefits of open source is that the world keeps improving the software you base your business on. You do need to regularly get those improvements and upgrades into production, though. As a result, we tend to roll our own RPMs somewhat often. Here are a few high-level rules we use to (mostly) achieve the zen nirvana of a stable environment of latest version software. Continue reading