Original image by veggiefrog, on Flickr

I thought it might be a good idea to put together a post about some of the IPS changes that appear in Solaris 11.1. To make it more of a challenge, everything I’m going to talk about here, begins with the letter ‘P‘.

Performance

We’ve made great progress in speeding up IPS. I think performance bugs tend to come in a few different flavours: difficult to solve or subtle bugs, huge and obvious ones, bugs that can be solved by doing tasks in parallel and bugs that are really all about the perception of performance, rather than actual performance. We’ve come across at least one of each of those flavours during the course of our work on 11.1.

Shawn and Brock spent time digging into general packaging performance, carefully analyzing the existing code and testing changes to improve performance and reduce memory usage. Ultimately, their combined efforts resulted in a 30% boost to pkg(5) performance across the board, which I think was pretty impressive.

Other performance bugs were much easier to spot and fix. For example, 'pkg history' performance on systems with lots of boot environments was attrocious: my laptop with 1796 pkg history entries was taking 3 minutes to run 'pkg history' with S11 IPS bits, and after the fix, the command runs in 11 seconds, another good performance improvement, albeit one of lesser significance.

I’ll mention some other performance fixes in the next two sections.

Parallel zones

Apart from trying to perform operations more quickly, a typical way to address performance problems is to make the system faster by doing things in parallel. In this case, in the previous release, 'pkg update' in a global zone that contains many non-global zones was quite slow because we worked on one zone at a time. For S11.1, Ed did some excellent work to add the ‘-C‘ flag to several pkg(1) subcommands, allowing multiple zones to be updated at once.

Ed’s work wasn’t simply just to perform multiple operations in parallel, but also to improve what was being done along the way – it was a lot of change, and it was well worth it.

With the work we’ve done in the past on the system-repository, these parallel updates are network-efficient, with caching of packaged content for zones being provided by the system repository.

Progress-tracking

Sometimes you can make a system appear faster by making the user interface provide more feedback on what is being performed. Dan added some wonderful new progress tracking code to all of the pkg(5) tools, changing the tools to use that API.

So, if the older "Planning /-|-\ " spinner was frustrating you, then you’ll definitely enjoy the changes here. It’s hard to show an example of the curses-terminal-twiddling in this blog post, so here’s what you’d see when piping the output (the progress tracking code can tell when it’s talking to a terminal, and it adjusts the output accordingly): 

root@kakariki:~# pkg install squid | tee
 Startup: Refreshing catalog 'solaris' ... Done
 Startup: Caching catalogs ... Done
Planning: Solver setup ... Done
Planning: Running solver ... Done
Planning: Finding local manifests ... Done
Planning: Fetching manifests: 0/1  0% complete
Planning: Fetching manifests: 1/1  100% complete
Planning: Package planning ... Done
Planning: Merging actions ... Done
Planning: Checking for conflicting actions ... Done
Planning: Consolidating action changes ... Done
Planning: Evaluating mediators ... Done
Planning: Planning completed in 10.32 seconds
           Packages to install:  1
       Create boot environment: No
Create backup boot environment: No
            Services to change:  1

Download:    0/1715 items  0.0/8.8MB  0% complete 
Download:  230/1715 items  0.3/8.8MB  3% complete (59.9k/s)
Download:  505/1715 items  0.5/8.8MB  6% complete (55.6k/s)
.
. [ ed. I removed a few lines here ]
.
Download: 1417/1715 items  8.3/8.8MB  94% complete (140k/s)
Download: 1653/1715 items  8.7/8.8MB  99% complete (85.2k/s)
Download: Completed 8.78 MB in 76.46 seconds (117k/s)
 Actions:    1/1903 actions (Installing new actions)
 Actions: Completed 1903 actions in 3.23 seconds.
Finalize: Updating package state database ...  Done
Finalize: Updating image state ...  Done
Finalize: Creating fast lookup database ...  Done

Proxy configuration

I suppose this could also be seen as a performance bug (though the link is tenuous, I admit)

Behind the scenes, pkg(5) tools use libcurl to provide HTTP and HTTPS transport facilities, and we inherit the support that libcurl provides for web proxies. Typically a user would set a $http_proxy environment variable before running their IPS command.

At home, I run a custom web-proxy, through which I update all of my Solaris development machines (most of my systems reside in NZ, but many of my repositories are in California, so using a local caching proxy is a big performance win for me)

Now, I could use pkgrecv(1) to pull updates to a local repository every build, and while this is great for users who want to maintain a “golden master” repository, it’s not an ideal solution for a user like me who updates their systems every two weeks: the upstream repository tends to have a bunch of packages that I will never care about, I’m unlikely to ever need to worry about sparc binaries at home, and I’m never sure which packages I’ll want to install, so I prefer the idea of a transparent repository cache, than having to populate and maintain a complete local repository.

Unfortunately, quite often I’d find myself forgetting to set $http_proxy before running ‘pkg update‘, and I’d end up using more bandwidth than I needed to, and when using repositories that were only accessible with different proxies, things tended to get a bit messy.

So, to scratch that itch, we came up with the "--proxy" argument to "pkg set-publisher", which allows us to associate proxies with origins on your system. The support is provided at the individual origin level, so you can use different proxies for different URLs (handy if you have some publishers that live on the internet, and others that live on your intranet)

To make things easier for zones administrators, the system-repository inherits that configuration automatically, so there’s no need to set the ‘config/http_proxy‘ option in the SMF service anymore (however, if you do set it, the service will use that value to override all --proxy settings on individual origins)

As part of this work, we also changed the output of "pkg publisher", removing those slightly confusing "proxy://http://foobar" URIs. Now, in a non-global zone, we show something like this: 

root@kakariki:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris        (syspub)     origin   online T <system-repository>
solaris        (syspub)     origin   online F <system-repository>
solaris        (syspub)     origin   online F http://localhost:8080/

This particular zone is one that’s running on a system which has a HTTP origin and a file-based origin in the global zone, and a HTTP origin that has been manually added to the nonglobal zone. The “P” column indicates whether a proxy is being used for each origin (“T” standing for “true”, indicating HTTP access going through the system repository, and “F” standing for “false”, showing the file-based publisher being served directly from the system-repository itself, as well as the zone-specific repository running on port 8080 in that zone)

We print more details about the configuration using the "pkg publisher <publisher>" command: 

root@kakariki:~# pkg publisher solaris

            Publisher: solaris
                Alias: 
           Origin URI: http://ipkg.us.oracle.com/solaris11/dev/
                Proxy: http://localhost:1008
              SSL Key: None
             SSL Cert: None
           Origin URI: http://localhost:1008/solaris/db0fe5e2fa5d0cabc58d864c318fdd112587cd89/
              SSL Key: None
             SSL Cert: None
           Origin URI: http://localhost:8080/
              SSL Key: None
             SSL Cert: None
          Client UUID: 69d11a50-0e13-11e2-bc49-0208209b8274
      Catalog Updated: October  4, 2012 10:56:53 AM 
              Enabled: Yes

P5p archive support and zones

This isn’t related to performance (unless you count a completely missing piece of functionality as being a particularly severe form of performance bug!) When implementing the system-repository for S11, we ran out of runway and had to impose a restriction on the use of “p5p” archives when the system had zones configured. This work lifts those restrictions.

The job of the system-repository is to allow the zone to access all of the pkg(5) repositories that are configured in the global zone, and to ensure that any changes in the publisher configuration in the global zone are reflected in every non-global zone automatically.

To do this, it uses a basic caching proxy for HTTP and HTTPS-based publishers, and a series of Apache RewriteRule directives to provide access to the file-based repositories configured in the global zone.

P5p files were more problematic: these are essentially archives of pkg(5) repositories that can be configured directly using ‘pkg set-publisher‘. The problem was, that no amount of clever RewriteRules would be able to crack open a p5p archive, and serve its contents the the non-global zone.

We considered a few different options on how to provide this support, but ended up with a solution that uses mod_wsgi (which is now in Solaris, as a result) to serve the contents directly. See /etc/pkg/sysrepo/sysrepo_p5p.py if you’re interested in how that works, but there’s no administrator interaction needed when using p5p archives, everything is taken care of by the system-repository service itself.

Pruning and general care-taking

According to hg(1), we’ve made 209 putbacks containing 276 bug fixes and RFEs to the pkg-gate since S11. So aside from all of the performance and feature work mentioned here, Solaris 11.1 comes with a lot of other IPS improvements – definitely a good reason to update to this release.

If you’re running on an Illumos-based distribution and you don’t have these bits in your distribution, I think now would be an excellent time to sync your hg repositories and pull these new changes. Feel free to ping us on #pkg5 on irc.freenode.net if you’ve any questions about porting, or anything else really – we’re a friendly bunch.

You can see a list of the changes we’ve made at 

http://src.opensolaris.org/source/history/pkg/gate/

Per-BE /var subdirectories (/var/share)

OK, that’s a slightly contrived name for this feature (only used here so it could begin with ‘P’) We’ve been calling this “separate /var/share” while it was under development.

Technically, this isn’t an IPS change, it’s a change in the way we package the operating system, but it’s a concrete example of one of the items in the IPS developer guide on how to migrate data across directories during package operations using the ‘salvage-from‘ attribute for ‘dir‘ actions.

This change moves several directories previously delivered under /var onto a new dataset, rpool/VARSHARE, allowing boot environments to carry less baggage around as part of each BE clone, sharing data where that makes sense. Bart came up with the mechanism and prototype to perform the migration of data that should be shared, and I finished it off and managed the putback.

For this release, the following directories are shared:

  • /var/audit
  • /var/cores
  • /var/crash (previously unpackaged!)
  • /var/mail
  • /var/nfs
  • /var/statmon

Have a look at /lib/svc/method/fs-minimal to see how this migration was performed. Here’s what pkg:/system/core-os looks like when delivering actions that salvage content: 

$ pkg contents -H -o action.raw -a 'path=var/.migrate*' core-os | pkgfmt
dir  path=var/.migrate owner=root group=sys mode=0755
dir  path=var/.migrate/audit owner=root group=root mode=0755 reboot-needed=true \
    salvage-from=/var/audit
dir  path=var/.migrate/cores owner=root group=sys mode=0755 reboot-needed=true \
    salvage-from=/var/cores
dir  path=var/.migrate/crash owner=root group=sys mode=0700 reboot-needed=true \
    salvage-from=/var/crash
dir  path=var/.migrate/mail owner=root group=mail mode=1777 reboot-needed=true \
    salvage-from=/var/mail
$ pkg contents -H -o action.raw -a 'target=../var/share/*' core-os | pkgfmt
link path=var/audit target=../var/share/audit
link path=var/cores target=../var/share/cores
link path=var/crash target=../var/share/crash
link path=var/mail target=../var/share/mail

As part of this work, we also wrote a new section 5 man page, datasets(5) which is well worth reading. It describes the default ZFS datasets that are created during installation, and explains how they interact with system utilities such as swap(1M), beadm(1M), useradd(1M), etc.

Putting the dev guide on docs.oracle.com

Finally, it’s worth talking a bit about the devguide. We wrote the IPS Developer Guide in time for the initial release of Solaris 11, but didn’t quite make the deadline for the official docs.oracle.com documentation release, leading us to publish it ourselves on OTN and opensolaris.org. Since then, we’ve had a complaints about the perceived lack of developer documentation for IPS, which was unfortunate.

So, for Solaris 11.1, Alta has converted the guide into Docbook, and done some cleanup on the text (the content is largely the same) and it will be available on docs.oracle.com in all its monochrome glory.

I think that’s all of the Solaris 11.1 improvements I’ll talk about for now – if you’ve questions on any of these, feel free to add comments below, mail us on pkg-discuss or pop in to #pkg5 to say hello. I’ll update this post with links to the official Solaris 11.1 documentation once it becomes available.