I thought it might be a good idea to put together a post about some of the IPS changes that appear in Solaris 11.1. To make it more of a challenge, everything I’m going to talk about here, begins with the letter ‘P‘.
We’ve made great progress in speeding up IPS. I think performance bugs tend to come in a few different flavours: difficult to solve or subtle bugs, huge and obvious ones, bugs that can be solved by doing tasks in parallel and bugs that are really all about the perception of performance, rather than actual performance. We’ve come across at least one of each of those flavours during the course of our work on 11.1.
Shawn and Brock spent time digging into general packaging performance, carefully analyzing the existing code and testing changes to improve performance and reduce memory usage. Ultimately, their combined efforts resulted in a 30% boost to
pkg(5) performance across the board, which I think was pretty impressive.
Other performance bugs were much easier to spot and fix. For example,
'pkg history' performance on systems with lots of boot environments was attrocious: my laptop with 1796 pkg history entries was taking 3 minutes to run
'pkg history' with S11 IPS bits, and after the fix, the command runs in 11 seconds, another good performance improvement, albeit one of lesser significance.
I’ll mention some other performance fixes in the next two sections.
Apart from trying to perform operations more quickly, a typical way to address performance problems is to make the system faster by doing things in parallel. In this case, in the previous release,
'pkg update' in a global zone that contains many non-global zones was quite slow because we worked on one zone at a time. For S11.1, Ed did some excellent work to add the ‘
-C‘ flag to several
pkg(1) subcommands, allowing multiple zones to be updated at once.
Ed’s work wasn’t simply just to perform multiple operations in parallel, but also to improve what was being done along the way – it was a lot of change, and it was well worth it.
With the work we’ve done in the past on the system-repository, these parallel updates are network-efficient, with caching of packaged content for zones being provided by the system repository.
Sometimes you can make a system appear faster by making the user interface provide more feedback on what is being performed. Dan added some wonderful new progress tracking code to all of the
pkg(5) tools, changing the tools to use that API.
So, if the older
"Planning /-|-\ " spinner was frustrating you, then you’ll definitely enjoy the changes here. It’s hard to show an example of the curses-terminal-twiddling in this blog post, so here’s what you’d see when piping the output (the progress tracking code can tell when it’s talking to a terminal, and it adjusts the output accordingly):
root@kakariki:~# pkg install squid | tee Startup: Refreshing catalog 'solaris' ... Done Startup: Caching catalogs ... Done Planning: Solver setup ... Done Planning: Running solver ... Done Planning: Finding local manifests ... Done Planning: Fetching manifests: 0/1 0% complete Planning: Fetching manifests: 1/1 100% complete Planning: Package planning ... Done Planning: Merging actions ... Done Planning: Checking for conflicting actions ... Done Planning: Consolidating action changes ... Done Planning: Evaluating mediators ... Done Planning: Planning completed in 10.32 seconds Packages to install: 1 Create boot environment: No Create backup boot environment: No Services to change: 1 Download: 0/1715 items 0.0/8.8MB 0% complete Download: 230/1715 items 0.3/8.8MB 3% complete (59.9k/s) Download: 505/1715 items 0.5/8.8MB 6% complete (55.6k/s) . . [ ed. I removed a few lines here ] . Download: 1417/1715 items 8.3/8.8MB 94% complete (140k/s) Download: 1653/1715 items 8.7/8.8MB 99% complete (85.2k/s) Download: Completed 8.78 MB in 76.46 seconds (117k/s) Actions: 1/1903 actions (Installing new actions) Actions: Completed 1903 actions in 3.23 seconds. Finalize: Updating package state database ... Done Finalize: Updating image state ... Done Finalize: Creating fast lookup database ... Done
I suppose this could also be seen as a performance bug (though the link is tenuous, I admit)
Behind the scenes,
pkg(5) tools use libcurl to provide HTTP and HTTPS transport facilities, and we inherit the support that libcurl provides for web proxies. Typically a user would set a
$http_proxy environment variable before running their IPS command.
At home, I run a custom web-proxy, through which I update all of my Solaris development machines (most of my systems reside in NZ, but many of my repositories are in California, so using a local caching proxy is a big performance win for me)
Now, I could use
pkgrecv(1) to pull updates to a local repository every build, and while this is great for users who want to maintain a “golden master” repository, it’s not an ideal solution for a user like me who updates their systems every two weeks: the upstream repository tends to have a bunch of packages that I will never care about, I’m unlikely to ever need to worry about sparc binaries at home, and I’m never sure which packages I’ll want to install, so I prefer the idea of a transparent repository cache, than having to populate and maintain a complete local repository.
Unfortunately, quite often I’d find myself forgetting to set
$http_proxy before running ‘
pkg update‘, and I’d end up using more bandwidth than I needed to, and when using repositories that were only accessible with different proxies, things tended to get a bit messy.
So, to scratch that itch, we came up with the
"--proxy" argument to
"pkg set-publisher", which allows us to associate proxies with origins on your system. The support is provided at the individual origin level, so you can use different proxies for different URLs (handy if you have some publishers that live on the internet, and others that live on your intranet)
To make things easier for zones administrators, the system-repository inherits that configuration automatically, so there’s no need to set the ‘
config/http_proxy‘ option in the SMF service anymore (however, if you do set it, the service will use that value to override all
--proxy settings on individual origins)
As part of this work, we also changed the output of
"pkg publisher", removing those slightly confusing
"proxy://http://foobar" URIs. Now, in a non-global zone, we show something like this:
root@kakariki:~# pkg publisher PUBLISHER TYPE STATUS P LOCATION solaris (syspub) origin online T <system-repository> solaris (syspub) origin online F <system-repository> solaris (syspub) origin online F http://localhost:8080/
This particular zone is one that’s running on a system which has a HTTP origin and a file-based origin in the global zone, and a HTTP origin that has been manually added to the nonglobal zone. The “P” column indicates whether a proxy is being used for each origin (“T” standing for “true”, indicating HTTP access going through the system repository, and “F” standing for “false”, showing the file-based publisher being served directly from the system-repository itself, as well as the zone-specific repository running on port 8080 in that zone)
We print more details about the configuration using the
"pkg publisher <publisher>" command:
root@kakariki:~# pkg publisher solaris Publisher: solaris Alias: Origin URI: http://ipkg.us.oracle.com/solaris11/dev/ Proxy: http://localhost:1008 SSL Key: None SSL Cert: None Origin URI: http://localhost:1008/solaris/db0fe5e2fa5d0cabc58d864c318fdd112587cd89/ SSL Key: None SSL Cert: None Origin URI: http://localhost:8080/ SSL Key: None SSL Cert: None Client UUID: 69d11a50-0e13-11e2-bc49-0208209b8274 Catalog Updated: October 4, 2012 10:56:53 AM Enabled: Yes
P5p archive support and zones
This isn’t related to performance (unless you count a completely missing piece of functionality as being a particularly severe form of performance bug!) When implementing the system-repository for S11, we ran out of runway and had to impose a restriction on the use of “
p5p” archives when the system had zones configured. This work lifts those restrictions.
The job of the system-repository is to allow the zone to access all of the
pkg(5) repositories that are configured in the global zone, and to ensure that any changes in the publisher configuration in the global zone are reflected in every non-global zone automatically.
To do this, it uses a basic caching proxy for HTTP and HTTPS-based publishers, and a series of Apache RewriteRule directives to provide access to the file-based repositories configured in the global zone.
P5p files were more problematic: these are essentially archives of
pkg(5) repositories that can be configured directly using ‘
pkg set-publisher‘. The problem was, that no amount of clever RewriteRules would be able to crack open a p5p archive, and serve its contents the the non-global zone.
We considered a few different options on how to provide this support, but ended up with a solution that uses mod_wsgi (which is now in Solaris, as a result) to serve the contents directly. See
/etc/pkg/sysrepo/sysrepo_p5p.py if you’re interested in how that works, but there’s no administrator interaction needed when using p5p archives, everything is taken care of by the system-repository service itself.
Pruning and general care-taking
hg(1), we’ve made 209 putbacks containing 276 bug fixes and RFEs to the pkg-gate since S11. So aside from all of the performance and feature work mentioned here, Solaris 11.1 comes with a lot of other IPS improvements – definitely a good reason to update to this release.
If you’re running on an Illumos-based distribution and you don’t have these bits in your distribution, I think now would be an excellent time to sync your hg repositories and pull these new changes. Feel free to ping us on
irc.freenode.net if you’ve any questions about porting, or anything else really – we’re a friendly bunch.
You can see a list of the changes we’ve made at
Per-BE /var subdirectories (/var/share)
OK, that’s a slightly contrived name for this feature (only used here so it could begin with ‘P’) We’ve been calling this “separate /var/share” while it was under development.
Technically, this isn’t an IPS change, it’s a change in the way we package the operating system, but it’s a concrete example of one of the items in the IPS developer guide on how to migrate data across directories during package operations using the ‘
salvage-from‘ attribute for ‘
This change moves several directories previously delivered under
/var onto a new dataset,
rpool/VARSHARE, allowing boot environments to carry less baggage around as part of each BE clone, sharing data where that makes sense. Bart came up with the mechanism and prototype to perform the migration of data that should be shared, and I finished it off and managed the putback.
For this release, the following directories are shared:
- /var/crash (previously unpackaged!)
Have a look at
/lib/svc/method/fs-minimal to see how this migration was performed. Here’s what
pkg:/system/core-os looks like when delivering actions that salvage content:
$ pkg contents -H -o action.raw -a 'path=var/.migrate*' core-os | pkgfmt dir path=var/.migrate owner=root group=sys mode=0755 dir path=var/.migrate/audit owner=root group=root mode=0755 reboot-needed=true \ salvage-from=/var/audit dir path=var/.migrate/cores owner=root group=sys mode=0755 reboot-needed=true \ salvage-from=/var/cores dir path=var/.migrate/crash owner=root group=sys mode=0700 reboot-needed=true \ salvage-from=/var/crash dir path=var/.migrate/mail owner=root group=mail mode=1777 reboot-needed=true \ salvage-from=/var/mail $ pkg contents -H -o action.raw -a 'target=../var/share/*' core-os | pkgfmt link path=var/audit target=../var/share/audit link path=var/cores target=../var/share/cores link path=var/crash target=../var/share/crash link path=var/mail target=../var/share/mail
As part of this work, we also wrote a new section 5 man page,
datasets(5) which is well worth reading. It describes the default ZFS datasets that are created during installation, and explains how they interact with system utilities such as
Putting the dev guide on docs.oracle.com
Finally, it’s worth talking a bit about the devguide. We wrote the IPS Developer Guide in time for the initial release of Solaris 11, but didn’t quite make the deadline for the official docs.oracle.com documentation release, leading us to publish it ourselves on OTN and opensolaris.org. Since then, we’ve had a complaints about the perceived lack of developer documentation for IPS, which was unfortunate.
So, for Solaris 11.1, Alta has converted the guide into Docbook, and done some cleanup on the text (the content is largely the same) and it will be available on docs.oracle.com in all its monochrome glory.
I think that’s all of the Solaris 11.1 improvements I’ll talk about for now – if you’ve questions on any of these, feel free to add comments below, mail us on pkg-discuss or pop in to
#pkg5 to say hello. I’ll update this post with links to the official Solaris 11.1 documentation once it becomes available.