Introduction
I’m starting a small series of blog posts to talk about one of the important concepts in IPS – self-assembly. We cover this in the IPS Developer Guide but don’t provide many examples as yet.
In the IPS Developer Guide, we introduced the concept of self-assembly as:
Any collection of installed software on a system should be able to build itself into a working configuration when that system is booted, by the time the packaging operation completes, or at software runtime.
Lots of software ships with default configuration in sample files, often installed in /etc. During packaging, these files are commonly marked as "user editable", with an attribute defining how those user edits should be treated in the case where the shipped example file gets updated in new release of the package.
In IPS, those user editable files are marked with a preserve attribute, which is documented in the pkg(5) man page.
However, what happens if we want to allow another package to deliver new configuration instead of simply allowing user edits?
By default, IPS will report an error if two packages try to deliver the same file.
In these blog posts, we’ll take a sample package, and show how it can be modified to allow us to deliver new add-on packages that deliver different configuration.
Before getting into a more complicated true self-assembly scenario (in the next post), we’ll cover a very simple one first.
In this first post, we’ll talk about the overlay attribute. Technically, this example doesn’t actually cover self-assembly. Instead, it shows how IPS allows packages to re-deliver configuration files already delivered by another package.
First, let’s introduce our example package.
Our example package
We’ll use a package that already exists as our example: the Squid web proxy.
In our examples, we’re going to delivering a new version of Squid that allows us to achieve our goal of being able to deliver add-on packages to supply configuration.
To be clear, I’m not suggesting all administrators ought to do this – by using their own private copy of a package shipped by Oracle, they face the burden of maintaining this version themselves: future upgrades from the solaris publisher will not automatically update their version. By default, publishers in IPS are sticky – so packages installed from one publisher may not be updated by a new version of that package from another publisher.
Publisher stickiness may be overridden, but then the administrator risks that their carefully crafted package gets updated by a version of the package from Oracle. In addition, the presence of a local version of the package may also prevent updates from occurring.
However, when I was looking for an example of the modifications that need to be made to a package which doesn’t normally participate in self-assembly, Squid fits the bill nicely.
Let’s look at the choices that were made when Squid was being packaged for Solaris, concentrating on how its configuration files are handled.
Using the following command, we can show the actions associated with the squid.conf files that are delivered in the package:
$ pkg contents -H -r -o action.raw -a path=etc/squid/squid.conf* squid | pkgfmt
Here is the output from the command:
file 7d8f133b331e7460fbbdca593bff31446f8a3bad path=etc/squid/squid.conf \
owner=root group=webservd mode=0644 \
chash=272ed7f686ce409a121f427a5b0bf75aed0e2095 \
original_name=SUNWsquid:etc/squid/squid.conf pkg.csize=1414 pkg.size=3409 \
preserve=renamenew
file 7d8f133b331e7460fbbdca593bff31446f8a3bad \
path=etc/squid/squid.conf.default owner=root group=bin mode=0444 \
chash=272ed7f686ce409a121f427a5b0bf75aed0e2095 pkg.csize=1414 pkg.size=3409
file 971681745b21a3d88481dbadeea6ce7f87b0070a \
path=etc/squid/squid.conf.documented owner=root group=bin mode=0444 \
chash=b9662e497184c97fff50b1c249a6e153c51432e1 pkg.csize=60605 \
pkg.size=200255
We can see that the package delivers three files:
- etc/squid/squid.conf
- This is the default configuration file that squid uses. You can see that it has a preserve attribute, with a value set to renamenew User edits to this file are allowed, and will be preserved on upgrade, and any new versions of the file (delivered by an updated Squid package) will be renamed.
- etc/squid/squid.conf.default
- Squid also ships with a second copy of the configuration file (notice how the hashes are the same as the previous version) with a different name – presumably to use as a record of the original configuration.
- etc/squid/squid.conf.documented
- Finally we have another copy of the configuration file, this time with more comments included, to better explain the configuration.
Adding an overlay attribute
In IPS, two packages are allowed to deliver the same file if:
- one package contains a file with the attribute overlay=allow
- another package contains the same file, with the attribute overlay=true
In both cases, all other file attributes (owner, mode, group) must match. The overlay attribute is covered in Chapter 3 of the IPS Developer Guide and is also documented in the pkg(5) man page.
Since our sample package doesn’t deliver its configuration file, etc/squid/squid.conf, with an overlay attribute, we’ll need to modify the package.
First, we download the package in a raw form, suitable for republishing later, and show where pkgrecv(1) stores the manifest:
$ pkgrecv -s http://pkg.oracle.com/solaris/release --raw -d squid-proto squid@3.1.8,5.11-0.175.0.0.0.2.537 Processing packages for publisher solaris ... Retrieving and evaluating 1 package(s)... PROCESS ITEMS GET (MB) SEND (MB) Completed 1/1 18.0/18.0 0.0/0.0 $ find squid-proto -name manifest squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z/manifest
Next, we’ll define a simple pkgmogrify(1) transform to add an overlay=allow attribute.
We’ll also remove the solaris publisher from the FMRI, as we intend to republish this package to our own repository. (This transform is discussed in more detail in Chapter 14 of the IPS Developer Guide)
The transform looks like:
<transform set name=pkg.fmri -> edit value pkg://[^/]+/ pkg://mypublisher/> <transform file path=etc/squid/squid.conf$ -> set overlay allow>
Here’s how we run it:
$ pkgmogrify squid-overlay.mog \
squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z/manifest \
> squid-overlay.mf
Finally we can republish our package:
$ pkgsend -s myrepository publish \
-d squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z \
squid-overlay.mf
WARNING: Omitting signature action 'signature 2ce2688faa049abe9d5dceeeabc4b17e7b72e792
.
.
.
pkg://mypublisher/web/proxy/squid@3.1.8,5.11-0.175.0.0.0.2.537:20111108T220909Z
PUBLISHED
We get a warning when republishing it saying that we’re dropping the signature action (I’ve trimmed the output here).
Package signing is always performed on a repository using pkgsign(1), never on a manifest. Since the package’s timestamp is always updated on publication, that would cause any hardcoded signatures to be invalid. Package signing is covered in more detail in Chapter 11 of the IPS Developer Guide.
This gets us part of the way towards our goal: we’ve now got a version of Squid that can allow other packages to deliver a new copy of etc/squid/squid.conf.
Notice that we’ve left the version alone on our copy of Squid, so it still complies with the same package version constraints that were on the original version of Squid that was shipped with Solaris.
Writing Configuration Packages
At this point, we can start writing packages to deliver new versions of our configuration file.
First let’s install our modified squid package. We’ll add our local repository to the system, and make sure we search for packages there before the solaris publisher, so that our packages are discovered first.
$ pfexec pkg set-publisher --search-before=solaris -p ./myrepository
Updated publisher(s): mypublisher
$ pfexec pkg install squid
Packages to install: 1
Create boot environment: No
Create backup boot environment: No
Services to change: 1
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 1519/1519 8.5/8.5
PHASE ACTIONS
Install Phase 1704/1704
PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2
Next, we’ll create our configuration package. Perhaps the only thing we want to change, is the default port that Squid listens on. Let’s write a new squid.conf file that uses port 8080 instead of 3128:
Our original squid configuration shows:
$ grep 3128 /etc/squid/squid.conf # Squid normally listens to port 3128 http_port 3128
We’ll write our new configuration:
$ mkdir -p squid-conf-proto/etc/squid
$ cat /etc/squid/squid.conf | sed -e 's/3128/8080/g' \
> squid-conf-proto/etc/squid/squid.conf
$ grep 8080 squid-conf-proto/etc/squid/squid.conf
# Squid normally listens to port 8080
http_port 8080
Now, we’ll create a package for the file. We’ll make the package depend on our Squid package. For this package, since the Squid package already delivers the dir action needed for etc/squid we’ll just deliver the file-action for our new squid.conf.
$ cat > squid-conf.mf
set name=pkg.fmri value=config/web/proxy/squid-configuration@1.0
set name=pkg.summary value="My Company Inc. Default squid.conf settings"
file path=etc/squid/squid.conf owner=root group=webservd mode=0644 \
overlay=true preserve=renameold
depend type=require fmri=web/proxy/squid@3.1.8
^D
Notice that we have specified overlay=true to indicate that this action should overlay any existing file, and have specified preserve=renameold to indicate that we want the old file renamed if one exists.
$ pkgsend -s myrepository publish -d squid-conf-proto squid-conf.mf pkg://mypublisher/config/web/proxy/squid-configuration@1.0,5.11:20111108T150325Z PUBLISHED
We can now install this package to our system, and check to make sure our changes have appeared:
$ pfexec pkg install squid-configuration
Packages to install: 1
Create boot environment: No
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 1/1 0.0/0.0
PHASE ACTIONS
Install Phase 4/4
PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2
The following unexpected or editable files and directories were
salvaged while executing the requested package operation; they
have been moved to the displayed location in the image:
etc/squid/squid.conf -> /var/pkg/lost+found/etc/squid/squid.conf-20111108T071810Z
$ grep 8080 /etc/squid/*
/etc/squid/squid.conf:# Squid normally listens to port 8080
/etc/squid/squid.conf:http_port 8080
$ pkg list squid squid-configuration
NAME (PUBLISHER) VERSION IFO
config/web/proxy/squid-configuration 1.0 i--
web/proxy/squid 3.1.8-0.175.0.0.0.2.537 i--
Conclusion
This was a pretty simple case – we’ve simply modified an existing package, and delivered a single new package allowing a single configuration package to deliver a change to the file.
This wasn’t really self-assembly per se, since the configuration is still hard-coded, but it is a common use-case, and provides a good introduction to our next example.
However, what happens if we want to deliver a further change to this file, from another package? Trying the same approach again, creating a new package "pkg:/config/web/proxy/squid-configuration-redux" then trying to install it,
we see:
$ pkgsend -s myrepository publish -d squid-conf-proto squid-conf-redux.mf pkg://mypublisher/config/web/proxy/squid-configuration-redux@1.0,5.11:20111108T152449Z PUBLISHED $ pfexec pkg install squid-configuration-redux Creating Plan | pkg install: The following packages all deliver file actions to etc/squid/squid.conf: pkg://mypublisher/web/proxy/squid@3.1.8,5.11-0.175.0.0.0.2.537:20111108T151647Z pkg://mypublisher/config/web/proxy/squid-configuration-redux@1.0,5.11:20111108T152449Z pkg://mypublisher/config/web/proxy/squid-configuration@1.0,5.11:20111108T151420Z These packages may not be installed together. Any non-conflicting set may be, or the packages must be corrected before they can be installed.
So IPS only allows one configuration package to be installed at a time. We’ll uninstall our configuration package, revert the old squid.conf content, then install our new configuration package:
$ pfexec pkg uninstall squid-configuration
Packages to remove: 1
Create boot environment: No
Create backup boot environment: No
PHASE ACTIONS
Removal Phase 3/3
PHASE ITEMS
Package State Update Phase 1/1
Package Cache Update Phase 1/1
Image State Update Phase 2/2
$ pfexec pkg revert /etc/squid/squid.conf
Packages to update: 1
Create boot environment: No
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 1/1 0.0/0.0
PHASE ACTIONS
Update Phase 1/1
PHASE ITEMS
Image State Update Phase 2/2
$ pfexec pkg install squid-configuration-redux
Packages to install: 1
Create boot environment: No
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 1/1 0.0/0.0
PHASE ACTIONS
Install Phase 4/4
PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2
The following unexpected or editable files and directories were
salvaged while executing the requested package operation; they
have been moved to the displayed location in the image:
etc/squid/squid.conf -> /var/pkg/lost+found/etc/squid/squid.conf-20111108T072930Z
We see that the new configuration file has been installed.
In the next post in this series, we’ll provide a more complex example of
self-assembly.
Tim Foster's Web Log 
Pingback: Self assembly – Part 2: multiple packages delivering configuration « Tim Foster's Web Log