We’ve just released Oracle Solaris 11.2 beta, and with it comes a considerable number of improvements in the packaging system, both for Solaris administrators and for developers who publish packages for Solaris.

Other than general bug fixes and performance improvements, I thought a few changes would be worth mentioning in a bit more detail, so here goes!

Admin changes

One of the focuses we had for this release was to simplify common administrative tasks in the packaging system, particularly for package repository management. Most of the changes in this section reflect that goal.

Mirror support

We’ve now made it extremely easy to create local mirrors of package repositories.

The following command will create a new repository in a new ZFS dataset in /var/share/pkg/repositories and will create a cron job which will periodically do a pkgrecv from all publishers configured on the system, keeping the local mirror up to date:

# svcadm enable pkg/mirror

If that’s too much content, the mirror service uses the notion of a “reference image” in which you can configure the origins which should be mirrored (“/” is the default reference image). All SSL keys/certs are obtained by the service from the properties on the reference image.

So to mirror just the official release repository, you could do:

# pkg image-create /export/ref-image-release
# pkg -R /export/ref-image-release set-publisher -p http://pkg.oracle.com/solaris/release
# svccfg -s pkg/mirror:default setprop config/ref_image = /export/ref-image-release
# svccfg -s pkg/mirror:default refresh
# svcadm enable pkg/mirror

Of course, if you want to maintain several local repositories, each mirroring a different repository with separate mirror-update schedules, you can easily create a new instance of the pkg/mirror service to do that.

More settings are available in the config property group in the SMF service, and they should be self-explanatory.

pkgrecv –clone

The mirror service mentioned above is an additive mirror of one or more origins, receiving into a single pkg(5) repository from one or more upstream repositories.

For better performance we have also included a very fast way to copy a single repository. The --clone operation for pkgrecv(1) gives you an exact copy of a repository, optionally limiting the pkgrecv to specific publishers.

Scalable repository server

In the past, when serving repositories over HTTP where there was a high expected load, we’ve recommended using an Apache front-end, and reverse-proxying to several pkg.depotd(1M) processes using a load balancer.

We felt that this was a rather involved setup just to get a performant repository server, so for this release we’re introducing a new repository server which serves pkg(5) content directly from Apache.

Here’s what that looks like:

Screenshot-ips repository server

Here, you can see a single pkg/depot, with associated httpd.worker processes, along with a series of pkg/server instances which correspond to the screenshot above:

#  svcs -p pkg/depot pkg/server
STATE          STIME    FMRI
online         18:44:21 svc:/application/pkg/depot:default
               18:44:20    713363 httpd.worker
               18:45:29    713595 httpd.worker
               18:45:29    713596 httpd.worker
               18:45:29    713597 httpd.worker
               18:47:54    713605 httpd.worker
               20:36:36    836614 httpd.worker
online         18:44:24 svc:/application/pkg/server:on12-extra
online         18:45:13 svc:/application/pkg/server:on12-nightly
online         18:45:25 svc:/application/pkg/server:pkg-gate

You can see that we only have processes associated with the pkg/depot service: the pkg/server instances here have properties set to say that they should not run a pkg.depotd(1M) instance, but instead should only be used for configuration of the pkg/depot server.

We can mix and match pkg/server instances which are associated with the pkg/depot service and instances which have their own pkg.depotd(1M) process.

The new pkg/depot service does not allow write access or publication, but otherwise responds to pkgrepo(1) commands as you would expect:

$ pkgrepo -s http://kura/pkg-gate info
PUBLISHER        PACKAGES STATUS           UPDATED
pkg5-nightly     12       online           2014-04-28T02:11:36.909420Z
pkg5-localizable 1        online           2014-04-28T02:11:40.244604Z

pkgrepo verify, fix

These were actually included in an S11.1 SRU, but they’re worth repeating here. We now have pkgrepo(1) subcommands to allow an administrator to verify and fix a repository, checking that all packages and files in the repository are valid, looking at repository permissions, verifying both package metadata and the files delivered by the package.

pkgrepo contents

You can now query the contents of a given package using the pkgrepo command (previously, you had to have a pkg(5) image handy in order to use “pkg contents”)

pkgrecv -m all-timestamps is the default

For most commands, you’d expect the most-commonly used operation to be the default. Well, for pkgrecv, when specifying a package name without the timestamp portion of the FMRI, we’ll now receive all packages matching that name, rather than just the latest one – which is what most of our users want by default. There are other -m arguments that allow you to change the way packages are matched, allowing you to choose the old behaviour.

SSL support for pkgrepo, pkgrecv, pkgsend

It’s now possible to specify keys and certificates when communicating with HTTPS repositories for these commands.

pkgsurf

pkgsurf(1) is a tool that implements a something we’d always wanted: a way to streamline our publication processes.

When publishing new builds of our software, we’d typically publish all packages for every build, even if the packaged content hadn’t changed, resulting in a lot of packaging bloat in the repository.

The repository itself was always efficient when dealing with package contents, since files are stored by hash in the repository. However, with each publication cycle, we’d get more package versions accumulating in the repository, with each new package referencing the same content. This would inflate package catalogs, and cause clients to do more work during updates, as they’d need to download the new package metadata each time.

pkgsurf(1) allows us to compare the packages we’ve just published with the packages in a reference repository, replacing any packages that have not changed with the original package metadata. The upshot of this is a greatly reduced number of packages accumulating in, say, a nightly build repository, resulting in less work for clients to do when systems are updated where no actual package content has changed between builds.

This is really more of a package developer change, rather than a package administrative change, but it’s in this section because having fewer package versions to deal with makes administrators happy.

pkg exact-install

This is a fast way to bring a system back to a known state, installing all packages supplied on the command line (and their dependencies) and removing all other packages from the system. This command can be very helpful when trying to bring a system back into compliance with a set of allowed packages.

While the operation itself is fairly straightforward, coming up with a name for it was complex, and we spent quite some time trying to decide on a name! It turned out “exact-install”, the original suggestion, was the most descriptive. The old computer science adage of “There are only two hard things in Computer Science: cache invalidation and naming things.”[1] remains safely intact.

–ignore-missing

Several pkg(1) subcommands now take a --ignore-missing argument, which prevents pkg(1) from reporting an error and returning when one of the packages presented on the command line wasn’t present in the image being operated upon.

Zones changes

The packaging system in Solaris has always been well-integrated with Solaris Zones, and with 11.2, we’ve improved that integration.

recursive linked-images operations

A common operation on systems with zones is to install or update a package in the global zone and all attached non-global zones. While pkg(1) has always ensured that packages in zones and non-global zones have always been compatible, apart from “pkg update” (with no arguments) most package operations would only apply on the global zone unless parent/child dependencies were specified on the package being installed or updated.

With Solaris 11.2, we now have a flag, -r, that can be used with pkg install, pkg uninstall and pkg update that will recurse into the zones on the system to perform that same packaging operation. The -z and -Z options can be supplied to select specific zones into which we should recurse, or exclude certain zones from being operated upon.

Actuators run for booted NGZ operations

This is really a side-effect of the work mentioned in the previous paragraph, but it bears repeating: actuators now fire in non-global zones as a result of package operations initiated in the global zone which needed to also operate in non-global zones.

Synchronous actuators

This applies only to global zones in this release (and non-global zones if you issued the pkg operation from within the zone, not recursive operations initiated from the global zone), but since we’ve just talked about actuators, now seems like a good time to mention it.

There are now --sync-actuators and --sync-actuators-timeout arguments for several pkg(1) subcommands that cause us to wait until all actuators have fired before returning, or to wait a specified amount of time before returning. That way, you can be sure that any self-assembly operations have completed before the pkg(1) client returns.

Kernel zones

Mike has written more about about kernel zones but I thought I’d make a small note about them with respect to the packaging system.

While the packaging system is well-integrated with traditional zones, it’s intentionally not integrated with kernel zones. That is, other than the initial installation of a kernel zone, there are no IPS interactions between a kernel zone and the global zone in which it’s hosted. The kernel zone is a separate IPS image, potentially running a different version of the operating system than the global zone.

Misc changes

system attributes support

The packaging system now has support for delivering files with system attributes (those visible using ls -/ c). See pkg(5) and chmod(1) for more details.

Multiple hash algorithm support

This is really a behind-the-scenes change, and for 11.2 it has no visible effects, but since I spent quite a while working on it, I thought it was worth mentioning :-) So far, the packaging system has used SHA-1 for all hashes calculated on package content and metadata throughout its codebase. We recognized that we’d want to support additional hash algorithms in the future, but at the same time ensure that old clients were compatible with packages published using algorithms other than SHA-1.

With this work, we revisited the use of SHA-1 in pkg(5) and made sure that the hash algorithm could be easily changed in the future, and that older clients using packages published with multiple hash algorithms would automatically choose the most favorable algorithm when verifying that package.

There’s work ahead to allow the publication of packages with more than one hash algorithm, but we’ve laid the foundations now for that work to happen.

To close

That’s been a quick roundup of the changes that we have in IPS in 11.2. I hope you’ve found it interesting.

On a personal note, I’ve had a lot of fun working on some of these features (I didn’t work on all of them). Of late I’ve spent most of my time working on the OS/Net build system, and have a new role helping that consolidation along towards its next major release (“major” in a similar sense to “major motion picture”, not “SunOS 6.0″ :-) so I won’t have as much time to spend on IPS for a while. I’ll try to dip my toe in, from time to time though!


[1] (and off-by-one errors) via http://martinfowler.com/bliki/TwoHardThings.html.

About these ads