I thought it might be a good idea to put together a post about some of the IPS changes that appear in Solaris 11.1. To make it more of a challenge, everything I’m going to talk about here, begins with the letter ‘P‘.
We’ve made great progress in speeding up IPS. I think performance bugs tend to come in a few different flavours: difficult to solve or subtle bugs, huge and obvious ones, bugs that can be solved by doing tasks in parallel and bugs that are really all about the perception of performance, rather than actual performance. We’ve come across at least one of each of those flavours during the course of our work on 11.1.
Shawn and Brock spent time digging into general packaging performance, carefully analyzing the existing code and testing changes to improve performance and reduce memory usage. Ultimately, their combined efforts resulted in a 30% boost to
pkg(5) performance across the board, which I think was pretty impressive.
Other performance bugs were much easier to spot and fix. For example,
'pkg history' performance on systems with lots of boot environments was attrocious: my laptop with 1796 pkg history entries was taking 3 minutes to run
'pkg history' with S11 IPS bits, and after the fix, the command runs in 11 seconds, another good performance improvement, albeit one of lesser significance.
I’ll mention some other performance fixes in the next two sections.
Apart from trying to perform operations more quickly, a typical way to address performance problems is to make the system faster by doing things in parallel. In this case, in the previous release,
'pkg update' in a global zone that contains many non-global zones was quite slow because we worked on one zone at a time. For S11.1, Ed did some excellent work to add the ‘
-C‘ flag to several
pkg(1) subcommands, allowing multiple zones to be updated at once.
Ed’s work wasn’t simply just to perform multiple operations in parallel, but also to improve what was being done along the way – it was a lot of change, and it was well worth it.
With the work we’ve done in the past on the system-repository, these parallel updates are network-efficient, with caching of packaged content for zones being provided by the system repository.
Sometimes you can make a system appear faster by making the user interface provide more feedback on what is being performed. Dan added some wonderful new progress tracking code to all of the
pkg(5) tools, changing the tools to use that API.
So, if the older
"Planning /-|-\ " spinner was frustrating you, then you’ll definitely enjoy the changes here. It’s hard to show an example of the curses-terminal-twiddling in this blog post, so here’s what you’d see when piping the output (the progress tracking code can tell when it’s talking to a terminal, and it adjusts the output accordingly):
root@kakariki:~# pkg install squid | tee Startup: Refreshing catalog 'solaris' ... Done Startup: Caching catalogs ... Done Planning: Solver setup ... Done Planning: Running solver ... Done Planning: Finding local manifests ... Done Planning: Fetching manifests: 0/1 0% complete Planning: Fetching manifests: 1/1 100% complete Planning: Package planning ... Done Planning: Merging actions ... Done Planning: Checking for conflicting actions ... Done Planning: Consolidating action changes ... Done Planning: Evaluating mediators ... Done Planning: Planning completed in 10.32 seconds Packages to install: 1 Create boot environment: No Create backup boot environment: No Services to change: 1 Download: 0/1715 items 0.0/8.8MB 0% complete Download: 230/1715 items 0.3/8.8MB 3% complete (59.9k/s) Download: 505/1715 items 0.5/8.8MB 6% complete (55.6k/s) . . [ ed. I removed a few lines here ] . Download: 1417/1715 items 8.3/8.8MB 94% complete (140k/s) Download: 1653/1715 items 8.7/8.8MB 99% complete (85.2k/s) Download: Completed 8.78 MB in 76.46 seconds (117k/s) Actions: 1/1903 actions (Installing new actions) Actions: Completed 1903 actions in 3.23 seconds. Finalize: Updating package state database ... Done Finalize: Updating image state ... Done Finalize: Creating fast lookup database ... Done
I suppose this could also be seen as a performance bug (though the link is tenuous, I admit)
Behind the scenes,
pkg(5) tools use libcurl to provide HTTP and HTTPS transport facilities, and we inherit the support that libcurl provides for web proxies. Typically a user would set a
$http_proxy environment variable before running their IPS command.
At home, I run a custom web-proxy, through which I update all of my Solaris development machines (most of my systems reside in NZ, but many of my repositories are in California, so using a local caching proxy is a big performance win for me)
Now, I could use
pkgrecv(1) to pull updates to a local repository every build, and while this is great for users who want to maintain a “golden master” repository, it’s not an ideal solution for a user like me who updates their systems every two weeks: the upstream repository tends to have a bunch of packages that I will never care about, I’m unlikely to ever need to worry about sparc binaries at home, and I’m never sure which packages I’ll want to install, so I prefer the idea of a transparent repository cache, than having to populate and maintain a complete local repository.
Unfortunately, quite often I’d find myself forgetting to set
$http_proxy before running ‘
pkg update‘, and I’d end up using more bandwidth than I needed to, and when using repositories that were only accessible with different proxies, things tended to get a bit messy.
So, to scratch that itch, we came up with the
"--proxy" argument to
"pkg set-publisher", which allows us to associate proxies with origins on your system. The support is provided at the individual origin level, so you can use different proxies for different URLs (handy if you have some publishers that live on the internet, and others that live on your intranet)
To make things easier for zones administrators, the system-repository inherits that configuration automatically, so there’s no need to set the ‘
config/http_proxy‘ option in the SMF service anymore (however, if you do set it, the service will use that value to override all
--proxy settings on individual origins)
As part of this work, we also changed the output of
"pkg publisher", removing those slightly confusing
"proxy://http://foobar" URIs. Now, in a non-global zone, we show something like this:
root@kakariki:~# pkg publisher PUBLISHER TYPE STATUS P LOCATION solaris (syspub) origin online T <system-repository> solaris (syspub) origin online F <system-repository> solaris (syspub) origin online F http://localhost:8080/
This particular zone is one that’s running on a system which has a HTTP origin and a file-based origin in the global zone, and a HTTP origin that has been manually added to the nonglobal zone. The “P” column indicates whether a proxy is being used for each origin (“T” standing for “true”, indicating HTTP access going through the system repository, and “F” standing for “false”, showing the file-based publisher being served directly from the system-repository itself, as well as the zone-specific repository running on port 8080 in that zone)
We print more details about the configuration using the
"pkg publisher <publisher>" command:
root@kakariki:~# pkg publisher solaris Publisher: solaris Alias: Origin URI: http://ipkg.us.oracle.com/solaris11/dev/ Proxy: http://localhost:1008 SSL Key: None SSL Cert: None Origin URI: http://localhost:1008/solaris/db0fe5e2fa5d0cabc58d864c318fdd112587cd89/ SSL Key: None SSL Cert: None Origin URI: http://localhost:8080/ SSL Key: None SSL Cert: None Client UUID: 69d11a50-0e13-11e2-bc49-0208209b8274 Catalog Updated: October 4, 2012 10:56:53 AM Enabled: Yes
P5p archive support and zones
This isn’t related to performance (unless you count a completely missing piece of functionality as being a particularly severe form of performance bug!) When implementing the system-repository for S11, we ran out of runway and had to impose a restriction on the use of “
p5p” archives when the system had zones configured. This work lifts those restrictions.
The job of the system-repository is to allow the zone to access all of the
pkg(5) repositories that are configured in the global zone, and to ensure that any changes in the publisher configuration in the global zone are reflected in every non-global zone automatically.
To do this, it uses a basic caching proxy for HTTP and HTTPS-based publishers, and a series of Apache RewriteRule directives to provide access to the file-based repositories configured in the global zone.
P5p files were more problematic: these are essentially archives of
pkg(5) repositories that can be configured directly using ‘
pkg set-publisher‘. The problem was, that no amount of clever RewriteRules would be able to crack open a p5p archive, and serve its contents the the non-global zone.
We considered a few different options on how to provide this support, but ended up with a solution that uses mod_wsgi (which is now in Solaris, as a result) to serve the contents directly. See
/etc/pkg/sysrepo/sysrepo_p5p.py if you’re interested in how that works, but there’s no administrator interaction needed when using p5p archives, everything is taken care of by the system-repository service itself.
Pruning and general care-taking
hg(1), we’ve made 209 putbacks containing 276 bug fixes and RFEs to the pkg-gate since S11. So aside from all of the performance and feature work mentioned here, Solaris 11.1 comes with a lot of other IPS improvements – definitely a good reason to update to this release.
If you’re running on an Illumos-based distribution and you don’t have these bits in your distribution, I think now would be an excellent time to sync your hg repositories and pull these new changes. Feel free to ping us on
irc.freenode.net if you’ve any questions about porting, or anything else really – we’re a friendly bunch.
You can see a list of the changes we’ve made at
Per-BE /var subdirectories (/var/share)
OK, that’s a slightly contrived name for this feature (only used here so it could begin with ‘P’) We’ve been calling this “separate /var/share” while it was under development.
Technically, this isn’t an IPS change, it’s a change in the way we package the operating system, but it’s a concrete example of one of the items in the IPS developer guide on how to migrate data across directories during package operations using the ‘
salvage-from‘ attribute for ‘
This change moves several directories previously delivered under
/var onto a new dataset,
rpool/VARSHARE, allowing boot environments to carry less baggage around as part of each BE clone, sharing data where that makes sense. Bart came up with the mechanism and prototype to perform the migration of data that should be shared, and I finished it off and managed the putback.
For this release, the following directories are shared:
- /var/crash (previously unpackaged!)
Have a look at
/lib/svc/method/fs-minimal to see how this migration was performed. Here’s what
pkg:/system/core-os looks like when delivering actions that salvage content:
$ pkg contents -H -o action.raw -a 'path=var/.migrate*' core-os | pkgfmt dir path=var/.migrate owner=root group=sys mode=0755 dir path=var/.migrate/audit owner=root group=root mode=0755 reboot-needed=true \ salvage-from=/var/audit dir path=var/.migrate/cores owner=root group=sys mode=0755 reboot-needed=true \ salvage-from=/var/cores dir path=var/.migrate/crash owner=root group=sys mode=0700 reboot-needed=true \ salvage-from=/var/crash dir path=var/.migrate/mail owner=root group=mail mode=1777 reboot-needed=true \ salvage-from=/var/mail $ pkg contents -H -o action.raw -a 'target=../var/share/*' core-os | pkgfmt link path=var/audit target=../var/share/audit link path=var/cores target=../var/share/cores link path=var/crash target=../var/share/crash link path=var/mail target=../var/share/mail
As part of this work, we also wrote a new section 5 man page,
datasets(5) which is well worth reading. It describes the default ZFS datasets that are created during installation, and explains how they interact with system utilities such as
Putting the dev guide on docs.oracle.com
Finally, it’s worth talking a bit about the devguide. We wrote the IPS Developer Guide in time for the initial release of Solaris 11, but didn’t quite make the deadline for the official docs.oracle.com documentation release, leading us to publish it ourselves on OTN and opensolaris.org. Since then, we’ve had a complaints about the perceived lack of developer documentation for IPS, which was unfortunate.
So, for Solaris 11.1, Alta has converted the guide into Docbook, and done some cleanup on the text (the content is largely the same) and it will be available on docs.oracle.com in all its monochrome glory.
I think that's all of the Solaris 11.1 improvements I'll talk about for now - if you've questions on any of these, feel free to add comments below, mail us on pkg-discuss or pop in to
#pkg5 to say hello. I'll update this post with links to the official Solaris 11.1 documentation once it becomes available.
At Sun, and now Oracle, we have a server called Jurassic. The machine was previously hosted in MPK17, the office in Melo Park, CA where many of the Solaris kernel developers worked (now occupied by a bunch of kids improving the lot of humanity through careful application of advertising technology) It now lives in the Santa Clara offices, where many of us moved to.
Every two weeks, jurassic is updated to the latest development builds of Solaris. Less frequently, it gets a forklift upgrade to more recent hardware to improve test coverage on that platform. The “Developing Solaris” document has this to say about jurassic:
You should assume that once you putback your change, the rest of the world will be running your code in production. More specifically, if you happen to work in MPK17, within three weeks of putback, your change will be running on the building server that everyone in MPK17 depends on. Should your change cause an outage during the middle of the day, some 750 people will be out of commission for the order of an hour. Conservatively, every such outage costs Sun $30,000 in lost time [ed. note from timf: I strongly suspect this is lower now: newer jurassic hardware along with massive improvements in Solaris boot time, along with bootable ZFS means that we can reboot jurassic with the last stable Solaris bits very quickly and easily nowadays, though that's not an excuse to putback a changeset that causes jurassic to tip over] — and depending on the exact nature of who needed their file system, calendar or mail and for what exactly, it could cost much, much more.
If this costs us so much, why do we do it? In short, to avoid the Quality Death Spiral. The Quality Death Spiral is much more expensive than a handful of jurassic outages — so it’s worth the risk. But you must do your part by delivering FCS quality all the time.
Does this mean that you should contemplate ritual suicide if you introduce a serious bug? Of course not — everyone who has made enough modifications to delicate, critical subsystems has introduced a change that has induced expensive downtime somewhere. We know that this will be so because writing system software is just so damned tricky and hard. Indeed, it is because of this truism that you must demand of yourself that you not integrate a change until you are out of ideas of how to test it. Because you will one day introduce a bug of such subtlety that it will seem that no one could have caught it.
And what do you do when that awful, black day arrives? Here’s a quick coping manual from those of us who have been there:
But most importantly, you must ask yourself: what could I have done differently? If you honestly don’t know, ask a fellow engineer to help you. We’ve all been there, and we want to make sure that you are able to learn from it. Once you have an answer, take solace in it; no matter how bad you feel for having introduced a problem, you can know that the experience has improved you as an engineer — and that’s the most anyone can ask for.
So, naturally, my home directory in CA is on jurassic, and whenever I’m using lab machines in California, I too am subject to whatever bits are running on jurassic.
However, I don’t live in California – I work remotely from New Zealand, and as good as NFSv4 is, I don’t fancy accessing all my content over the Pacific link.
I strongly believe in the sentiment expressed in the Developing Solaris document though, so my solution is to run a “mini-jurassic” at home, a solution I expect most other remote Solaris developers use.
My home server was previously my desktop machine – a little 1.6ghz Atom 330 box that I wrote about a while ago. Since Oracle took over, I now run a much more capable workstation with a Xeon E31270 @ 3.40GHz, a few disks and a lot more ram :) Despite the fact the workstation also runs bits from bi-weekly builds of Solaris, it doesn’t do enough to even vaguely stress the hardware, so when I got it at the beginning of the year, I repurposed my old Atom box as a mini-jurassic.
Here are the services I’ve got running at the moment:
… well, obviously. The box is pretty limited in that it’s maxed out at 4gb RAM, and non-ECC ram at that (I know – I’ll definitely be looking for an ECC-capable board next time, though I haven’t looked to see if there are any mini-ITX, low-power boards out at the moment)
With only three disks available, I use a single disk for the bootable root pool and a pair of disks, mirrored, for the main data pool. I periodically use ZFS to send/recv important datasets from the mirror to other machines on my home network. I suspect whenever I next upgrade the system, I’ll buy more disks and use a 3-way mirror: space hasn’t been a problem yet, the main data-pool is just using 1.5TB disks, and I’m only at 24% capacity.
I run the old zfs-auto-snapshot service on the system so that I always have access to daily, hourly, and every-15-minute snapshots of the datasets I really care about.
- I serve my home directory from here, which automounts onto my laptop and workstation. It also shares to my mac. Whenever I have to travel, I use ZFS to send/receive all of the datasets that make up my home directory over to my laptop, then send them back when I return.
- The windows laptop mounts its guest Z: drive via the CIFS server sharing a single dataset from the data pool (with a quota on that dataset, just in case) This is also shared to my mac.
- An Immutable Zone
- Immutable zones are a new feature in Solaris 11. I have a very stripped-down zone, which is internet-facing, running FeedPlus, a simple cron-job that runs a Python script and a minimal web-server. The zone has resource-controls set to give it only 256mb of ram to prevent it from taking over the world. I really ought to configure Crossbow to limit bandwidth as well.
- A read/write zone
The standard flavour of zones have been around for a while now. This runs the web server for the house, sharing music and video content. All of the content actually resides in the global-zone, but is shared into a zone using ZFS clones of the main datasets, which means that even if someone goes postal in the zone, all of my data is safe.
The zone also runs my IRC logger for #pkg5 on Freenode (helpful when you work in a different timezone)
- IPS updates
The system gets upgraded every two weeks, creating a new boot environment both for the zones as well as the global zone. It updates through a caching HTTP proxy which runs on my workstation, helping to further minimise bandwidth when I update all of my local machines once new bits become available (though IPS is already pretty good at keeping bandwidth to a minimum, only downloading the files that change during each update)
I tend to run several other stable and experimental bits and pieces on my home systems, both on the little Atom box, as well as my workstation. These mostly relate to my day-job improving IPS in Solaris, and those have already proved to be worth their weight, both in terms of shaking bugs out, as well as making my life a lot easier as a remote worker. I hope to write more about some of those in a future post sometime.
As more capabilities get added to Solaris, as with the jurassic server in California, I try as much as I can to find ways to exercise those new bits, because as it says on the jurassic web page:
Every problem we find and fix here is a problem which a customer will not see.
and that’s a good thing.
I found that, despite Twitterfeed claiming a 30 minute interval between checking the RSS/Atom URL it was configured to repost to Twitter, updates weren’t being pulled that frequently and often a G+ update I’d write that was relatively timely would finally appear several hours later. I decided it’d be better to work out how to post directly to Twitter.
For webapps, this didn’t appear to be too complex, but it’s a bit more involved for CLI applications (OAuth, yuck!) python-twitter helps a lot though, and it seems to be working so far. For now, I’ve got this running in a cron job that fires every 5 minutes, and while it’s definitely not as network-efficient as it could be, it’s doing the job just fine.
As before, I’m really hoping Google will come along with proper G+ to Twitter bridging (and in that direction: yes it’s lossy, but 140 characters just seems so restricting these days!) I’m also hoping that more of the folks I follow on Twitter will ditch both Facebook and Twitter, and give G+ another go – we’ll see.
Comments and bug reports welcome over on the FeedPlus Github page.
I’m excited about today’s launch of Solaris 11 – I’ve been contributing to Solaris for quite a while now, pretty much since 1996, but my involvement in S11 has been the most fun I’ve had in all releases so far.
I’ve talked before about some of the work I’ve done on IPS over the last two years – pkg history, pkgdepend (and here), pkglint and pkgsend and most recently, helping to put together the IPS Developer Guide.
Today, I’m going to talk about the system repository and how I helped.
How zones differ from earlier releases
Zones that use IPS are different than those in Solaris 10, in that they are always full-root: every zone contains its own local copy of each package, they don’t inherit packaged content from the global zone as "sparse" zones did in Solaris 10.
This simplifies a lot of zone-related functionality: for the most part, administrators can treat a zone as if it were a full Solaris instance, albeit a very small one. By default new zones in S11 are tiny. However, packaging with zones is a little more complex, and the system aims to hide that complexity
Some packages in the zone always need to be kept in sync with those packages in the global zone. For example, anything which delivers a kernel module and a userland application that interfaces with it must be kept in sync between the global zone and any non-global zones on the system.
In earlier OpenSolaris releases, after each global-zone update, each non-global zone had to be updated by hand, attaching and detaching each zone. During that detach/attach the ipkg brand scripts determined which packages were now in the global zone, and updated the non-global zone accordingly.
In addition, in OpenSolaris, the packaging system itself didn’t have any way of ensuring that every publisher in the global zone was also available in the non-global zone, making updates difficult if switching publishers.
Zones in Solaris 11
In Solaris 11, zones are now first-class citizens of the packaging system. Each zone is installed as a linked image, connected to the parent image, which is the global zone.
During packaging operations in the global zone, IPS recurses into any non-global zones to ensure that packages which need to be kept in sync between the global and non-global zones are kept in sync.
For this to happen, it’s important for the zone to have access to all of the IPS repositories that are available from the global zone.
This is problematic for a few reasons:
- the zone might not be on the same subnet as the global zone
- the global-zone administrator might not want to distribute SSL keys/certs for the repos to all zone administrators
- the global zone might change its publisher configuration, requiring publisher configuration change in every non-global zone
The System Repository
The system repository, and accompanying zones-proxy services was our solution to the list of problems above.
The SMF Services responsible are:
The first two services run in the global zone, the last one runs in the non-global zones.
With these services, the system repository shares publisher configuration to all non-global zones on the system, and also acts as a conduit to the publishers configured in the global zone. Inside the non-global zone, these proxied global-zone publishers are called system publishers.
When performing packaging operations inside a zone that accesses those publishers, Solaris proxies access through the system repository. While proxying, the system repository also caches any file-content that was
downloaded. If there are lots of zones all downloading the same packaged content, that will be efficiently managed.
If you don’t care about how all this works behind the scenes, then you can stop reading now.
There’s three parts to making all of the above work, apart from the initial linked image functionality that Ed worked on, which was fundamental to all of the system repository work.
- IPS client/repository support
- Zones proxy
- System repository
IPS client/repository support
Brock managed the heavy lifting here. This work involved:
- defining an interchange format that IPS could use to pass publisher configuration between the global and non-global zones
- refreshing the system repository service on every parent image publisher change
- allowing local publisher configuration to merge with system publisher configuration
- ensuring that system-provided publishers could not have their order changed
- allowing an image to be created that has no publishers
- toggling use of the system publisher
The zones proxy client, when started in the non-global zone creates a socket which listens on an inet port on 127.0.0.1. It passes the file descriptor for this socket to the zones proxy daemon via a door call.
The zones proxy daemon then listens for connections on the file descriptor. When the zone proxy daemon receives a connection, it proxies the connection to the system repository.
This allows the zone to access the system repository without any additional networking configuration needed (which I think is pretty neat – nicely done Krister!)
The system repository itself consists of two components:
- A Python program, /usr/lib/pkg.sysrepo
- A custom Apache 2.2 instance
Brock initially prototyped some httpd.conf configurations, and I worked on the code to write them automatically, produce the response that the system repository would use to inform zones of the configured publishers, and also worked out how to proxy access to file-based publishers in the global zone, which was an interesting problem to solve.
When you start the system-repository service in the global zone, pkg.sysrepo(1) determines the enabled, configured publishers then creates a response file served to non-global zones that want to discover the publishers configured in the global zone. It then uses a Mako template from /etc/pkg/sysrepo/sysrepo_httpd.conf.mako to generate an Apache configuration file.
The configuration file describes a basic caching proxy, providing limited access to the URLs of each publisher, as well as allowing URL rewrites to serve any file-based repositories. It uses the SSL keys and certificates from the global zone, and allows proxies access to those from the non-global zone over http.
(remember, data served by the system repository between the zone and non-global zone goes over the zones proxy socket, so http is fine here: access from the proxy to the publisher still goes over https)
The system repository service then starts an Apache instance, and a daemon to keep the proxy cache down to its configured maximum size. More detail on the options available to tune the system repository are in pkg.sysrepo(1) man page.
The practical upshot of all this, is that all zones can access all publishers configured on the global zone, and if that configuration changes, the zones publishers automatically change too. Of course, non-global zones can add their own publishers, but aren’t allowed to change the order, or disable any system
Here’s what the pkg publisher output looks like in a non-global zone:
root@puroto:~# pkg publisher PUBLISHER TYPE STATUS URI solaris (non-sticky, syspub) origin online proxy://http://pkg.oracle.com/solaris11/release/ mypublisher (syspub) origin online http://localhost:1008/mypublisher/89227627f3c003d11b1e4c0b5356a965ef7c9712/ test (syspub) origin online http://localhost:1008/test/eec48b7c8b107bb3ec9b9cf0f119eb3d90b5303e/
and here’s the system repository running in the global zone:
$ ps -fu pkg5srv | grep httpd pkg5srv 206 2334 0 12:02:02 ? 0:00 /usr/apache2/2.2/bin/64/httpd.worker -f /system/volatile/pkg/sysrepo/sysrepo_ht pkg5srv 204 2334 0 12:02:02 ? 0:00 /usr/apache2/2.2/bin/64/httpd.worker -f /system/volatile/pkg/sysrepo/sysrepo_ht pkg5srv 205 2334 0 12:02:02 ? 0:00 /usr/apache2/2.2/bin/64/httpd.worker -f /system/volatile/pkg/sysrepo/sysrepo_ht pkg5srv 939 2334 0 12:46:32 ? 0:00 /usr/apache2/2.2/bin/64/httpd.worker -f /system/volatile/pkg/sysrepo/sysrepo_ht
Personally, I’ve found this capability to be incredibly useful. I work from home, and have a system with an internet-facing non-global zone, and a global zone accessing our corporate VPN. My non-global zone is able to securely access new packages when it needs to (and I get to test my own code at the same time!)
Performing a pkg update from the global zone ensures that all zones are kept in sync, and will update all zones automatically (though, as mentioned in the Zones administration guide, pkg update <list of packages> will simply update the global zone, and ensure that during that update only the packages that cross the kernel/userland boundary are updated in each zone.)
Working on zones and the system repository was a lot of fun – hope you find it useful.
I’m starting a small series of blog posts to talk about one of the important concepts in IPS – self-assembly. We cover this in the IPS Developer Guide but don’t provide many examples as yet.
In the IPS Developer Guide, we introduced the concept of self-assembly as:
Any collection of installed software on a system should be able to build itself into a working configuration when that system is booted, by the time the packaging operation completes, or at software runtime.
Lots of software ships with default configuration in sample files, often installed in /etc. During packaging, these files are commonly marked as "user editable", with an attribute defining how those user edits should be treated in the case where the shipped example file gets updated in new release of the package.
In IPS, those user editable files are marked with a preserve attribute, which is documented in the pkg(5) man page.
However, what happens if we want to allow another package to deliver new configuration instead of simply allowing user edits?
By default, IPS will report an error if two packages try to deliver the same file.
In these blog posts, we’ll take a sample package, and show how it can be modified to allow us to deliver new add-on packages that deliver different configuration.
Before getting into a more complicated true self-assembly scenario (in the next post), we’ll cover a very simple one first.
In this first post, we’ll talk about the overlay attribute. Technically, this example doesn’t actually cover self-assembly. Instead, it shows how IPS allows packages to re-deliver configuration files already delivered by another package.
First, let’s introduce our example package.
Our example package
We’ll use a package that already exists as our example: the Squid web proxy.
In our examples, we’re going to delivering a new version of Squid that allows us to achieve our goal of being able to deliver add-on packages to supply configuration.
To be clear, I’m not suggesting all administrators ought to do this – by using their own private copy of a package shipped by Oracle, they face the burden of maintaining this version themselves: future upgrades from the solaris publisher will not automatically update their version. By default, publishers in IPS are sticky – so packages installed from one publisher may not be updated by a new version of that package from another publisher.
Publisher stickiness may be overridden, but then the administrator risks that their carefully crafted package gets updated by a version of the package from Oracle. In addition, the presence of a local version of the package may also prevent updates from occurring.
However, when I was looking for an example of the modifications that need to be made to a package which doesn’t normally participate in self-assembly, Squid fits the bill nicely.
Let’s look at the choices that were made when Squid was being packaged for Solaris, concentrating on how its configuration files are handled.
Using the following command, we can show the actions associated with the squid.conf files that are delivered in the package:
$ pkg contents -H -r -o action.raw -a path=etc/squid/squid.conf* squid | pkgfmt
Here is the output from the command:
file 7d8f133b331e7460fbbdca593bff31446f8a3bad path=etc/squid/squid.conf \ owner=root group=webservd mode=0644 \ chash=272ed7f686ce409a121f427a5b0bf75aed0e2095 \ original_name=SUNWsquid:etc/squid/squid.conf pkg.csize=1414 pkg.size=3409 \ preserve=renamenew file 7d8f133b331e7460fbbdca593bff31446f8a3bad \ path=etc/squid/squid.conf.default owner=root group=bin mode=0444 \ chash=272ed7f686ce409a121f427a5b0bf75aed0e2095 pkg.csize=1414 pkg.size=3409 file 971681745b21a3d88481dbadeea6ce7f87b0070a \ path=etc/squid/squid.conf.documented owner=root group=bin mode=0444 \ chash=b9662e497184c97fff50b1c249a6e153c51432e1 pkg.csize=60605 \ pkg.size=200255
We can see that the package delivers three files:
- This is the default configuration file that squid uses. You can see that it has a preserve attribute, with a value set to renamenew User edits to this file are allowed, and will be preserved on upgrade, and any new versions of the file (delivered by an updated Squid package) will be renamed.
- Squid also ships with a second copy of the configuration file (notice how the hashes are the same as the previous version) with a different name – presumably to use as a record of the original configuration.
- Finally we have another copy of the configuration file, this time with more comments included, to better explain the configuration.
Adding an overlay attribute
In IPS, two packages are allowed to deliver the same file if:
- one package contains a file with the attribute overlay=allow
- another package contains the same file, with the attribute overlay=true
Since our sample package doesn’t deliver its configuration file, etc/squid/squid.conf, with an overlay attribute, we’ll need to modify the package.
First, we download the package in a raw form, suitable for republishing later, and show where pkgrecv(1) stores the manifest:
$ pkgrecv -s http://pkg.oracle.com/solaris/release --raw -d squid-proto firstname.lastname@example.org,5.11-0.175.0.0.0.2.537 Processing packages for publisher solaris ... Retrieving and evaluating 1 package(s)... PROCESS ITEMS GET (MB) SEND (MB) Completed 1/1 18.0/18.0 0.0/0.0 $ find squid-proto -name manifest squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z/manifest
Next, we’ll define a simple pkgmogrify(1) transform to add an overlay=allow attribute.
We’ll also remove the solaris publisher from the FMRI, as we intend to republish this package to our own repository. (This transform is discussed in more detail in Chapter 14 of the IPS Developer Guide)
The transform looks like:
<transform set name=pkg.fmri -> edit value pkg://[^/]+/ pkg://mypublisher/> <transform file path=etc/squid/squid.conf$ -> set overlay allow>
Here’s how we run it:
$ pkgmogrify squid-overlay.mog \ squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z/manifest \ > squid-overlay.mf
Finally we can republish our package:
$ pkgsend -s myrepository publish \ -d squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z \ squid-overlay.mf WARNING: Omitting signature action 'signature 2ce2688faa049abe9d5dceeeabc4b17e7b72e792 . . . pkg://email@example.com,5.11-0.175.0.0.0.2.537:20111108T220909Z PUBLISHED
We get a warning when republishing it saying that we’re dropping the signature action (I’ve trimmed the output here).
Package signing is always performed on a repository using pkgsign(1), never on a manifest. Since the package’s timestamp is always updated on publication, that would cause any hardcoded signatures to be invalid. Package signing is covered in more detail in Chapter 11 of the IPS Developer Guide.
This gets us part of the way towards our goal: we’ve now got a version of Squid that can allow other packages to deliver a new copy of etc/squid/squid.conf.
Notice that we’ve left the version alone on our copy of Squid, so it still complies with the same package version constraints that were on the original version of Squid that was shipped with Solaris.
Writing Configuration Packages
At this point, we can start writing packages to deliver new versions of our configuration file.
First let’s install our modified squid package. We’ll add our local repository to the system, and make sure we search for packages there before the solaris publisher, so that our packages are discovered first.
$ pfexec pkg set-publisher --search-before=solaris -p ./myrepository Updated publisher(s): mypublisher $ pfexec pkg install squid Packages to install: 1 Create boot environment: No Create backup boot environment: No Services to change: 1 DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1519/1519 8.5/8.5 PHASE ACTIONS Install Phase 1704/1704 PHASE ITEMS Package State Update Phase 1/1 Image State Update Phase 2/2
Next, we’ll create our configuration package. Perhaps the only thing we want to change, is the default port that Squid listens on. Let’s write a new squid.conf file that uses port 8080 instead of 3128:
Our original squid configuration shows:
$ grep 3128 /etc/squid/squid.conf # Squid normally listens to port 3128 http_port 3128
We’ll write our new configuration:
$ mkdir -p squid-conf-proto/etc/squid $ cat /etc/squid/squid.conf | sed -e 's/3128/8080/g' \ > squid-conf-proto/etc/squid/squid.conf $ grep 8080 squid-conf-proto/etc/squid/squid.conf # Squid normally listens to port 8080 http_port 8080
Now, we’ll create a package for the file. We’ll make the package depend on our Squid package. For this package, since the Squid package already delivers the dir action needed for etc/squid we’ll just deliver the file-action for our new squid.conf.
$ cat > squid-conf.mf set name=pkg.fmri firstname.lastname@example.org set name=pkg.summary value="My Company Inc. Default squid.conf settings" file path=etc/squid/squid.conf owner=root group=webservd mode=0644 \ overlay=true preserve=renameold depend type=require email@example.com ^D
Notice that we have specified overlay=true to indicate that this action should overlay any existing file, and have specified preserve=renameold to indicate that we want the old file renamed if one exists.
$ pkgsend -s myrepository publish -d squid-conf-proto squid-conf.mf pkg://firstname.lastname@example.org,5.11:20111108T150325Z PUBLISHED
We can now install this package to our system, and check to make sure our changes have appeared:
$ pfexec pkg install squid-configuration Packages to install: 1 Create boot environment: No Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1/1 0.0/0.0 PHASE ACTIONS Install Phase 4/4 PHASE ITEMS Package State Update Phase 1/1 Image State Update Phase 2/2 The following unexpected or editable files and directories were salvaged while executing the requested package operation; they have been moved to the displayed location in the image: etc/squid/squid.conf -> /var/pkg/lost+found/etc/squid/squid.conf-20111108T071810Z $ grep 8080 /etc/squid/* /etc/squid/squid.conf:# Squid normally listens to port 8080 /etc/squid/squid.conf:http_port 8080 $ pkg list squid squid-configuration NAME (PUBLISHER) VERSION IFO config/web/proxy/squid-configuration 1.0 i-- web/proxy/squid 3.1.8-0.175.0.0.0.2.537 i--
This was a pretty simple case – we’ve simply modified an existing package, and delivered a single new package allowing a single configuration package to deliver a change to the file.
This wasn’t really self-assembly per se, since the configuration is still hard-coded, but it is a common use-case, and provides a good introduction to our next example.
However, what happens if we want to deliver a further change to this file, from another package? Trying the same approach again, creating a new package "pkg:/config/web/proxy/squid-configuration-redux" then trying to install it,
$ pkgsend -s myrepository publish -d squid-conf-proto squid-conf-redux.mf pkg://email@example.com,5.11:20111108T152449Z PUBLISHED $ pfexec pkg install squid-configuration-redux Creating Plan | pkg install: The following packages all deliver file actions to etc/squid/squid.conf: pkg://firstname.lastname@example.org,5.11-0.175.0.0.0.2.537:20111108T151647Z pkg://email@example.com,5.11:20111108T152449Z pkg://firstname.lastname@example.org,5.11:20111108T151420Z These packages may not be installed together. Any non-conflicting set may be, or the packages must be corrected before they can be installed.
So IPS only allows one configuration package to be installed at a time. We’ll uninstall our configuration package, revert the old squid.conf content, then install our new configuration package:
$ pfexec pkg uninstall squid-configuration Packages to remove: 1 Create boot environment: No Create backup boot environment: No PHASE ACTIONS Removal Phase 3/3 PHASE ITEMS Package State Update Phase 1/1 Package Cache Update Phase 1/1 Image State Update Phase 2/2 $ pfexec pkg revert /etc/squid/squid.conf Packages to update: 1 Create boot environment: No Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1/1 0.0/0.0 PHASE ACTIONS Update Phase 1/1 PHASE ITEMS Image State Update Phase 2/2 $ pfexec pkg install squid-configuration-redux Packages to install: 1 Create boot environment: No Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1/1 0.0/0.0 PHASE ACTIONS Install Phase 4/4 PHASE ITEMS Package State Update Phase 1/1 Image State Update Phase 2/2 The following unexpected or editable files and directories were salvaged while executing the requested package operation; they have been moved to the displayed location in the image: etc/squid/squid.conf -> /var/pkg/lost+found/etc/squid/squid.conf-20111108T072930Z
We see that the new configuration file has been installed.
In the next post in this series, we’ll provide a more complex example of
In the previous post in this series, we showed how it was possible to take a single package and republish it, such that other packages could overwrite a default configuration file.
The example we used was the Squid web proxy, allowing configuration packages to overwrite /etc/squid/squid.conf with new contents.
There was a limitation using that approach: only one package could contribute to that configuration at a time, replacing the entire shipped configuration.
Recall, that we define self-assembly in Chapter 1 of the IPS Developer Guide as:
Any collection of installed software on a system should be able to build itself into a working configuration when that system is booted, by the time the packaging operation completes, or at software runtime.
In this post, we’ll cover a more advanced case than last time: true self-assembly, where the configuration can be delivered by multiple add-on packages, if necessary. In particular, we’ll continue to talk about Squid, a package that isn’t normally capable of self-assembly, and will show how we fix that.
How does self-assembly work?
The main premise with self-assembly, is that configuration for an application must be built from a composed view of all fragments of the entire configuration that are present on the system. That can be done either by the application itself, in which case nothing else is required on the part of the application packager, or it can be done with an add-on service to assemble the entire configuration file from the delivered fragments.
When a new package delivers another fragment of the configuration, then the application must have its configuration rebuilt to include that fragment.
Similarly, when a fragment is removed from the system, again, the application must have its configuration rebuilt from the remaining fragments on the system.
A good example of self-assembly is in the Solaris package for pkg:/web/server/apache-22. Solaris ships a default httpd.conf file that has an Include directive that references /etc/apache2/2.2/conf.d.
Packages can deliver a new file to that directory, and use a refresh_fmri actuator causing the system to automatically to refresh the Apache instance
either after a pkg install operation has completed, or after a
pkg remove operation has completed, causing the webserver to rebuild its configuration.
The reason behind self-assembly, is to replace postinstall, preinstall, preremove, postremove and class action scripts, needed by other packaging systems. Install-time scripting was a common source of errors during application packaging because the scripting had to work in multiple scenarios.
For example, scripts had to correctly run
- against alternate image roots, perhaps running on a system that didn’t have
the necessary tools support to correctly run the script
- within the confines of a LiveCD environment
- when making edits to an offline zone
With IPS, we eliminated those forms of install-time scripting, concentrating on an atomic set of actions (discussed in Chapter 3 of the IPS Developer Guide) that performed common packaging tasks, and allowing for actuators (discussed in Chapter 9 of the IPS Developer Guide) to run during packaging operations.
Actuators enable self-assembly to work on live systems by restarting or refreshing the necessary SMF services. Since the same SMF services they point to run during boot as well, we don’t need to do anything when performing operations on alternate images: the next time the image is booted, our self-assembly is completed.
Making Squid self-assembly aware
As in the previous post, we will start by downloading and modifying our Squid package.
This time, we intend to remove the etc/squid/squid.conf file entirely – our self-assembly service will be constructing this file instead for us. Recall that
Squid delivers some of its configuration files with the following actions:
file 7d8f133b331e7460fbbdca593bff31446f8a3bad path=etc/squid/squid.conf \ owner=root group=webservd mode=0644 \ chash=272ed7f686ce409a121f427a5b0bf75aed0e2095 \ original_name=SUNWsquid:etc/squid/squid.conf pkg.csize=1414 pkg.size=3409 \ preserve=renamenew file 7d8f133b331e7460fbbdca593bff31446f8a3bad \ path=etc/squid/squid.conf.default owner=root group=bin mode=0444 \ chash=272ed7f686ce409a121f427a5b0bf75aed0e2095 pkg.csize=1414 pkg.size=3409 file 971681745b21a3d88481dbadeea6ce7f87b0070a \ path=etc/squid/squid.conf.documented owner=root group=bin mode=0444 \ chash=b9662e497184c97fff50b1c249a6e153c51432e1 pkg.csize=60605 \ pkg.size=200255
Since squid.conf.default is already shipped and is identical to the
squid.conf file that is also delivered, we can use that for the basis of our self-assembly of the squid.conf configuration file.
We download a copy of the package with the following command:
$ pkgrecv -s http://pkg.oracle.com/solaris/release --raw -d squid-proto email@example.com,5.11-0.175.0.0.0.2.537
which pulls the content into the squid-proto directory.
We’ll use a series of pkgmogrify(1) transforms to edit the package contents, similar to the ones we used in the previous post. We will remove the file action that delivers squid.conf using a drop transform operation, and will also deliver a new directory, etc/squid/conf.d. Here is the transform file that accomplishes that:
<transform set name=pkg.fmri -> edit value pkg://[^/]+/ pkg://mypublisher/> <transform file path=etc/squid/squid.conf$ -> drop> dir path=etc/squid/conf.d owner=root group=bin mode=0755
We can create a new manifest using this transform using pkgmogrify(1):
$ pkgmogrify squid\-assembly.mog \ squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z/manifest \ > squid-assembly.mf
A self-assembly SMF service
In order for self-assembly to happen during packaging operations, we need to use an actuator discussed in Chapter 9 of the IPS Developer Guide.
The actuator is a special tag on any IPS action that points to an SMF service. The SMF service is made up of two components:
- The SMF manifest
- The SMF method script
This self-assembly SMF service is going to be responsible for building the contents of /etc/squid/squid.conf. We’ll talk about each component in the following section:
This is what the SMF manifest of our self-assembly service looks like:
<?xml version="1.0"?> <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type='manifest' name='Squid:self-assembly'> <service name='config/network/http/squid-assembly' type='service' version='1'> <single_instance /> <dependency name='fs-local' grouping='require_all' restart_on='none' type='service'> <service_fmri value='svc:/system/filesystem/local:default' /> </dependency> <dependent name='squid_self-assembly-complete' grouping='optional_all' restart_on='none'> <service_fmri value='svc:/milestone/self-assembly-complete' /> </dependent> <instance enabled='true' name='default'> <exec_method type='method' name='start' exec='/lib/svc/method/squid-self-assembly' timeout_seconds='30'/> <exec_method type='method' name='stop' exec=':true' timeout_seconds='0'/> <property_group name='startd' type='framework'> <propval name='duration' type='astring' value='transient' /> </property_group> </instance> </service> </service_bundle>
This defines a service instance that we intend to use whenever we deliver new configuration file fragments to the system.
For that to happen, any configuration file fragment added or removed must include a restart_fmri actuator.
For example, a package might deliver a configuration file fragment:
file path=etc/squid/squid.conf/myconfig.conf owner=root group=bin mode=0644 \ restart_fmri=svc:/config/network/http/squid-assembly:default \ restart_fmri=svc:/network/http/squid:default
The other vital thing needed, is an SMF dependency on the SMF service delivered by the Squid package. We need to add this, so that the Squid application will only be able to start once our self-assembly service has finished producing our configuration file.
First, we’ll create a proto area for the files we’re going to add to our Squid package, and copy the default SMF manifest:
$ mkdir -p squid-assembly-proto/lib/svc/manifest/network $ cp /lib/svc/manifest/network/http\-squid.xml squid-assembly-proto/lib/svc/manifest/network
Next, we edit the http-squid.xml SMF manifest, adding the following:
<!-- Wait for the Squid self-assembly service to complete --> <dependency name='squid-assembly' grouping='require_all' restart_on='none' type='service'> <service_fmri value='svc:/config/network/http/squid-assembly:default'/> </dependency>
Now that we’ve done this, our next step, is writing the method script for our self-assembly service.
The SMF method script
We need to write a script, such that when it is run, we end up with /etc/squid.conf containing all changes, as defined in all configuration fragments installed on the system.
This step can be as simple or complex as you’d like it to be – essentially we’re performing postinstall scripting here, but on our terms: we know exactly the environment the script is running in – that of a booted OS where our package is installed (defined by the depend actions that accompany the package)
Here is a sample script, written in Python (as short as I could make it, so there’s very little error checking involved here) which takes squid.conf.default copies it to squid.conf, then applies a series of edits to it.
We’ll save the script as /lib/svc/method/squid-self-assembly.
#!/usr/bin/python2.6 import shutil import os import re import logging # define the paths we'll work with MASTER="/etc/squid/squid.conf.default" CONF_FILE="/etc/squid/squid.conf" CONF_DIR="/etc/squid/conf.d/" # verbose logging for now logging.basicConfig(level=logging.DEBUG) def apply_edits(fragment): """Takes edit operations in the path "fragment", and applies them to CONF_FILE in order. The syntax of our config file is intentionally basic. We support the following operations: # lines that start with a hash are comments add <line to add to the config file> remove <regular expression to remove> """ squid_config = file(CONF_FILE).readlines() squid_config = "".join(squid_config) # read our list of operations operations = open(fragment, "r").readlines() operations = [line.rstrip() for line in operations] for op in operations: if op.startswith("add"): addition = op[len("add") + 1:] logging.debug("adding line %s" % addition) squid_config += "\n" + addition elif op.startswith("remove"): exp = op[len("remove") + 1:] squid_config = re.sub(exp, "", squid_config) logging.debug("removing expression %s" % exp) elif op.startswith("#"): pass conf = open(CONF_FILE, "w") conf.write(squid_config + "\n") conf.close() # first, remove any existing configuration if os.path.exists(CONF_FILE): os.unlink(CONF_FILE) # now copy the master template file in, on # which all edits are based shutil.copy(MASTER, CONF_FILE) os.chmod(CONF_FILE, 0644) fragments =  # now iterate through the contents of /etc/squid/conf.d # looking for configuration fragments, and apply the changes # find in a defined order. We do not look in subdirectories. for dirpath, dirnames, filenames in os.walk("/etc/squid/conf.d/"): fragments = sorted(filenames) break for fragment in fragments: logging.debug("Applying edits from %s" % fragment) apply_edits(os.path.join(CONF_DIR, fragment))
Testing the self-assembly script
We can now test the self-assembly script. For the most part, this testing can be done outside the confines of the pkg(1) command – we simply need to ensure
that our self-assembly script runs properly.
First, we’ll check that the squid.conf file isn’t present, run the script, then determine that the contents are the same as squid.conf.default
# ls /etc/squid/squid.conf /etc/squid/squid.conf: No such file or directory # /lib/svc/method/squid-self-assembly # digest -a sha1 /etc/squid/squid.conf.default /etc/squid/squid.conf (/etc/squid/squid.conf.default) = 7d8f133b331e7460fbbdca593bff31446f8a3bad (/etc/squid/squid.conf) = 7d8f133b331e7460fbbdca593bff31446f8a3bad #
Next, we’ll try a simple configuration fragment:
# cat > /etc/squid/conf.d/change_http_port.conf # The default configuration uses port 3128, our organisation uses 8080 # We'll remove that default, add a comment, and add a http_port directive remove # Squid normally listens to port 3128 remove http_port 3128 add # Our organisation requires Squid to operate on port 8080 add http_port 8080 ^D
Then we’ll test the self-assembly script again:
# /lib/svc/method/squid-self-assembly DEBUG:root: --- applying edits from change_http_port.conf --- DEBUG:root:removing expression # Squid normally listens to port 3128 DEBUG:root:removing expression http_port 3128 DEBUG:root:adding line # Our organisation requires Squid to operate on port 8080 DEBUG:root:adding line http_port 8080
We can verify that the changes have been made:
# grep "port 8080" /etc/squid/squid.conf # Our organisation requires Squid to operate on port 8080 http_port 8080
Now, we’ll add another configuration fragment:
# cat > /etc/squid/conf.d/connect_ports.conf # We want to allow users to connect to gmail and irc # over our proxy server. add # We need to allow access to gmail and irc add acl Connect_ports port 5222 # gmail chat add acl Connect_ports port 6667 # irc chat add http_access allow CONNECT Connect_ports ^D
and see what happens when we run the self-assembly script:
# /lib/svc/method/squid-self-assembly DEBUG:root: --- applying edits from change_http_port.conf --- DEBUG:root:removing expression # Squid normally listens to port 3128 DEBUG:root:removing expression http_port 3128 DEBUG:root:adding line # Our organisation requires Squid to operate on port 8080 DEBUG:root:adding line http_port 8080 DEBUG:root: --- applying edits from connect_ports.conf --- DEBUG:root:adding line # We need to allow access to gmail and irc DEBUG:root:adding line acl Connect_ports port 5222 # gmail chat DEBUG:root:adding line acl Connect_ports port 6667 # irc chat DEBUG:root:adding line http_access allow CONNECT Connect_ports
Again, we can verify that the edits have been made correctly:
# grep "port 8080" /etc/squid/squid.conf # Our organisation requires Squid to operate on port 8080 http_port 8080 # egrep gmail\|irc /etc/squid/squid.conf # We need to allow access to gmail and irc acl Connect_ports port 5222 # gmail chat acl Connect_ports port 6667 # irc chat
And finally, we can see what happens if we remove one of our fragments:
# rm /etc/squid/conf.d/connect_ports.conf # /lib/svc/method/squid-self-assembly DEBUG:root: --- applying edits from change_http_port.conf --- DEBUG:root:removing expression # Squid normally listens to port 3128 DEBUG:root:removing expression http_port 3128 DEBUG:root:adding line # Our organisation requires Squid to operate on port 8080 DEBUG:root:adding line http_port 8080 # # grep "port 8080" /etc/squid/squid.conf # Our organisation requires Squid to operate on port 8080 http_port 8080 # egrep gmail\|irc /etc/squid/squid.conf #
As expected, the configuration file no longer contains the directives configured by connect_ports.conf, since that was removed from the system, but still
contains the changes from change_http_port.conf
Delivering the SMF service
The bulk of the hard work has been done now – to recap:
- we have modified the Squid package to drop the shipped squid.conf file
- we have an SMF service that can perform self assembly, generating
squid.conf files from installed fragments on the system
- we have added a dependency to the Squid SMF service on our self-assembly SMF service
All that remains, is to ensure that the self-assembly service gets included in
the Squid package.
For that, we’ll add a few more lines to the pkgmogrify(1) transform that we talked about earlier, so that it looks like:
<transform set name=pkg.fmri -> edit value pkg://[^/]+/ pkg://mypublisher/> <transform file path=etc/squid/squid.conf$ -> drop> dir path=etc/squid/conf.d owner=root group=bin mode=0755 file path=lib/svc/method/squid/squid-self-assembly group=bin mode=0555 owner=root file path=lib/svc/manifest/network/http-squid-assembly.xml group=sys \ mode=0444 owner=root restart_fmri=svc:/system/manifest-import:default
Now we can transform our original Squid package, and publish it to our repository:
$ pkgmogrify squid-assembly.mog \ squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z/manifest \ > squid\-assembly.mf $ pkgsend -s myrepository publish -d squid-assembly-proto \ -d squid-proto/web%2Fproxy%2Fsquid/3.1.8%2C5.11-0.175.0.0.0.2.537%3A20111019T121425Z \ squid\-assembly.mf WARNING: Omitting signature action 'signature 2ce2688faa049abe9d5dceeeabc4b17e7b72e792 . . pkg://firstname.lastname@example.org,5.11-0.175.0.0.0.2.537:20111108T201820Z PUBLISHED $
Installing that package, we discover a svc:/config/network/http/squid-assembly service, and verify that when we drop unpackaged files into /etc/squid/conf.d, and restart the self-assembly service, we see what we expect:
# more /var/svc/log/config-network-http-squid-assembly\:default.log [ Nov 8 12:19:50 Enabled. ] [ Nov 8 12:19:50 Rereading configuration. ] [ Nov 8 12:19:50 Executing start method ("/lib/svc/method/squid-self-assembly"). ] [ Nov 8 12:19:50 Method "start" exited with status 0. ] [ Nov 8 12:23:42 Stopping because service restarting. ] [ Nov 8 12:23:42 Executing stop method (null). ] [ Nov 8 12:23:42 Executing start method ("/lib/svc/method/squid-self-assembly"). ] DEBUG:root: --- applying edits from change_port.conf --- DEBUG:root:removing expression # Squid normally listens to port 3128 DEBUG:root:removing expression http_port 3128 DEBUG:root:adding line # Our organisation requires Squid to operate on port 8080 DEBUG:root:adding line http_port 8080 [ Nov 8 12:23:42 Method "start" exited with status 0. ]
We have verified that Squid is performing self-assembly perfectly.
Delivering new configuration fragments
Now that we have a service that’s capable of performing self-assembly, we need to know how to deliver configuration fragments in new packages.
This is simply a case of delivering config files to /etc/squid/conf.d, and applying the correct actuator tags to the manifest.
An example manifest that delivers this would be:
set name=pkg.fmri value=pkg:/email@example.com set name=pkg.summary value="Our organisations squid configurations" file path=etc/squid/conf.d/change_http.conf owner=root group=bin mode=0644 \ restart_fmri=svc:/config/network/http/squid-assembly:default \ restart_fmri=svc:/network/http:squid
When we publish, then install this manifest, we see:
# pkg install squid-configuration@2 Packages to install: 1 Create boot environment: No Create backup boot environment: No Services to change: 2 DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1/1 0.0/0.0 PHASE ACTIONS Install Phase 3/3 PHASE ITEMS Package State Update Phase 1/1 Image State Update Phase 2/2
We can quickly verify that the Squid configuration has changed:
$ curl localhost:8080 | grep squid/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2904 100 2904 0 0 1633k 0 --:--:-- --:--:-- --:--:-- 2835k <p>Generated Tue, 08 Nov 2011 23:00:27 GMT by tcx2250-13 (squid/3.1.8)</p>
And we can backout the configuration by removing the package, and again check that the Squid configuration has changed:
# curl localhost:8080 | grep squid % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (7) couldn't connect to host # curl localhost:3128 | grep squid/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3140 100 3140 0 0 1779k 0 --:--:-- --:--:-- --:--:-- 3066k <p>Generated Tue, 08 Nov 2011 23:03:37 GMT by tcx2250-13 (squid/3.1.8)</p>
We won’t go into details here, but clearly, multiple packages could deliver
configuration fragments at the same time, and they would all contribute to the
configuration of our service.
This has been a pretty fast example of the self-assembly idiom, but we hope this has been useful, and shows complex scripting operations can be performed in IPS.
There may more work to do to make the Squid application fully self-assembly aware – we’ve only covered the main configuration file and have’t looked at whether we also want to allow the other files in /etc/squid to participate in self-assembly. If we did want to do that, it would be a case of ensuring that:
- we ship a master template for each configuration file
- modify our self-assembly SMF service to copy each template into place
- ensure our script can perform edits on that file
Of course, there’s other ways in which a self-assembly service could perform edits – we could use SMF to deliver properties to the service, which are then accessed by a self-assembly script, and placed into a configuration file, but perhaps that’s an example for another day.
Over the last while, we’ve been writing a replacement for the venerable Application Packaging Developer’s Guide.
I started at Sun in the European Localisation group, and writing tools to create (and sometimes hack) SVR4 packages for Solaris was one of the things I had to do quite a bit – I found myself consulting the Application Packaging Developer’s Guide frequently. So, having the opportunity to help write its replacement was pretty cool.
Bart did a great job in defining the structure of the new book, and writing a lot of the initial content. Brock wrote some comprehensive content on package signing, SMF and zones, and I’ve been working on the guide since then, using some of the original documentation that Stephen wrote.
Unlike the previous guide, we have fewer examples in this guide. We feel that our man pages are better than the SVR4 packaging man pages were, and already contain useful examples. This guide is meant to compliment our man pages, not replace them.
The guide is a lot shorter than the old book – currently 56 pages, as opposed to the 190 pages in the document it replaces. Some of this is because of the fewer examples we have, but also we don’t have to write about patch creation, complex postinstall or class-action scripting or relocatable packages. IPS is simpler than SVR4 in many ways, though there is a learning curve, which this book aims to help with.
If you use IPS, or have had experience in packaging software for Solaris previously, I’d be interested in hearing your thoughts on the content so far.
Updated: I’ve updated the PDF with the latest content as of today, and made sure there’s a datestamp on the file, so you can tell which version you’re reading. Changelog below (though I’ve not put the source back to the gate, yet, so these changesets will likely be collapsed)
changeset: 2576:6b14edfc38b5 tag: tip user: Tim Foster date: Fri Oct 21 09:49:15 2011 +1300 description: Add another example to chpt14, republishing under a new publisher name changeset: 2575:0cf619c409b1 user: Tim Foster date: Thu Oct 20 15:51:22 2011 +1300 description: complete the SVR4 conversion example in appendix2 changeset: 2574:3a671202fd35 user: Tim Foster date: Thu Oct 20 12:09:48 2011 +1300 description: Rewrite the var/share discussion in chapter 10 Add macros and use them throughout the book Add a datestamp and logo to the cover page use hg identify for versioning add more text explaining why we don't "cross the streams" Add note on Immutable Zones to chapter1 Nobody uses fax machines anymore, do they? Add text on pkg.human-version to chapter 3 Add content for the user action Add content for the group action Introduce the constraints/freezing section of chapter 6 Reference constraints/freezing section in chapter 7 Describe XXX sections more clearly
Updated: I’ve added more stuff, replaced the links, but not yet pushed these changes to our external gate. Changelog below.
timf@linn hg log -r 2577: changeset: 2577:83de3ed97341 user: Tim Foster date: Fri Oct 21 16:23:37 2011 +1300 description: Break up chpt5 Move XXX for versioning disussion to chpt3 Fix table of contents changeset: 2578:df045bbafc98 user: Tim Foster date: Fri Oct 21 17:19:22 2011 +1300 description: Write content for mediated links, with an example changeset: 2579:1c3d87d950e6 tag: tip user: Tim Foster date: Fri Oct 21 17:26:50 2011 +1300 description: Move XXX for versioning disussion to chpt3 (fix build)
Updated: Almost at the final version – there’s a few small changes to make, but I’ve updated the links to the pdf of that version now, which hasn’t yet been pushed to the gate. Too many commit messages to indicate what’s changed unfortunately.
Updated: More minor changes and some style/branding tweaks. I’ve updated the pdf links above (2011-11-07:16:40:12-NZDT 8e2ee40e0bfb tip)
Updated: wikis.sun.com seems to be having issues. So in the meantime, there’s a local copy here
It wasn’t perfect, but was good enough to do the job.
I wasn’t able to find any good alternatives out there, so quickly rolled my own. It takes the id of a G+ user, and spits out an “atom.xml” file into the directory you choose, which you can then send to anything that can consume Atom.
The code is up on GitHub if anyone’s interested. My code isn’t perfect either, but will do the job until something better comes along: FeedPlus.
Update, Dec 13th 2011: I’ve updated the app to use the offical G+ API now, rather than relying on the json data that Russell’s app relied on. Hopefully this will make it a little more stable. Next stop, add an option to have it post to Twitter directly, rather than just writing atom.
Summary? I think it’s fabulous.
I like the idea of Circles, I like the user interface and have yet to start to accumulate the amount of crap I was tending to get on Facebook (pointless games, updates from people who clicked on spammy “see who’s been watching your profile” apps, that sort of thing)
It’s missing some stuff so far that for me makes Twitter the better option if you’re looking to follow interesting people, or look for posts about particular topics but don’t necessarily know who those people are, or who’s writing about a particular topic of interest: #hashtags. Being able to tag posts would be useful too, because while I like being able to view just a set of posts from a given Circle of friends, not everything those friends write about is necessarily relevant to me.
I’ve a few other nits with G+ that I’d like to see improved as well, including better client support on Android, a public API (which will help towards the first item, no doubt) but I’m happy to be patient.
It’s early days with Google+, though what I’ve seen so far makes me think it won’t be long before they’ll have the extra features that seem like an obvious addition to any social networking application.
So, what’s my plan with Twitter and Facebook?
Well, the week-long experiment on Google plus wasn’t long enough for me to choose whether or not to keep using one of those other services. My plan for now, is to hang on to my Twitter account, @timfoster and essentially treat it as read-only, except for direct messages and any replies I feel like posting to other folks on Twitter.
I’m going to try to use +Tim Foster for everything else.
As for Facebook, I’m going to give it up completely – I really only started using it to connect with the people that I know who don’t use Twitter, and seeing as there’s a better application out there now, it seems like Facebook has outlived its usefulness. I’d honestly rather one corporation had a detailed map of me than two (not that I can be particularly precious about this: I’ve got an Android phone, use Gmail – so in a sense, I’ve already signed my soul over to Google)
So, I’m closing my Facebook account as of next week, and stopping the Twitter re-publication now (I rarely ever actually post on Facebook, usually it’s just my Twitter feed getting redirected there)
If you would still like to keep up with me, get onto Google Plus – I’ve a few invites left which I’ll give to you if I know you, but I’m guessing they’ve got to be close to opening the service for anyone who wants in, invite or not. Otherwise, join Twitter.
I ran the 2011 Wellington Marathon on the 19th of June – that was my 3rd marathon, and without doubt, the toughest one I’ve done to date. As with the other races that I’ve
run, here’s a writeup of my experiences.
There’s also an official race report, if you’re interested.
Going into the build-up for the race this year, my goal was to try to beat last year’s time, but really I wanted to see if I had a sub-3hr marathon in me, I knew it’d be a stretch, but I was going to give it a try.
With that in mind, I stepped up to the next level of training, getting serious about it. As with earlier years, I thought I’d stick with Hal Higdon’s training program, but this time going for the Advanced – I plan. Over and above the “Intermediate – 2″ plan that I followed last year, this one added speed-work, hill repeats, tempo runs, and an extra day of running.
So, for 18 weeks during the lead up to the marathon, I was running 6 days a week, which was extremely grueling – not just for me, but also for my long-suffering family. They were incredibly supportive during the training, thanks Bob, Ella & Calum!
As before, I took a unit-test-inspired approach, printing out the training plan and sticking it to our fridge, with an aim of turning it green. Days when I just couldn’t be bothered running would get a red mark, days when I was unable to run, either because I was sick or because of work-pressure would be highlighted in yellow. Otherwise, I got on with the training, and slowly coloured each square green when I could. Ella was happy to help colouring too, she did a great job!
Overall, I was happy with the training progress – here’s the completed chart. I was sick for about 10 days in total, and if you match up pkg(5) integration dates for the projects I was working on, you can probably spot where work got in the way of running :-)
During training, I was getting my Yasso repeats up to the point where I was definitely in the right ballpark for a sub-3hr finish.
As I got closer to race day though, I became uncertain whether I’d be able to run the pace I was aiming for over the full distance. The longest I’d run at full race-pace was 16k. Still, I reasoned that those 16k were very hilly during training, so perhaps on the flat, I’d be alright. There was only one way to find out.
Race day, like last year, was cold and wet – a marathon at the end of June is in the depths of winter here. We lined up on the concourse of the Westpac stadium at 7:30am ready for the off.
My first few kilometers were slow as I moved through the pack, and I only looked at my watch as we approached Te Papa seeing that I was a little behind. I picked up the pace a bit, before settling into my race-pace.
The wind around the Wellington bays was strong and gusty, and like last year, I drafted when I could and took my turn. Getting out towards Shelly Bay, my 10k time was looking good – there was another runner doing about the same pace as me so we hung on and kept each other going. At this point, I was running a pace of between 4:01 and 4:12/km (depending on wind and water stations, I guess) which should have had me finishing around 2:52:00 – bang on the pace I was hoping for.
Thinking about pacing before the race, I had eventually decided that I’d go out with a really aggressive pace, aiming for well under a 3hr finish, rather then being more conservative and potentially making it over the line in 3:01 or thereabouts. While just scraping over the line sub-3hr would have been great,
I’d hate to have put in all that effort, and only miss my goal by a few seconds.
The question I faced was, was I willing to potentially blow my chances of beating last year’s time at all or risking the dreaded “DNF” (Did Not Finish), just to avoid the anguish of barely missing a sub-3hr finish? Yes, I was.
At the half-way turn-around, things were still going well. I was keeping to my pace, and feeling quite good about progress so far, but they say the first half of a marathon is just about transport – the race really begins at the halfway mark.
My pace slowed slightly leading up to the 30km mark and then I hit the wall – or rather, my legs did.
With only 10k to go, I started feeling slight twinges in my calves, which made me extremely worried. I tried easing up on my pace a little, to give my legs a chance. Apart from my legs though, I still felt good at this point – I wasn’t tired or out of breath, and definitely had fight left in me.
Unfortunately, things only got worse from here. As I made it back along the route we’d taken earlier, now joined by the half-marathon runners, the cramps in my legs became worse. At water stations, I’d drop to walking pace, trying to struggle back up to at least a 5:00/km pace when I started running again, but the cramps persisted and continued to get worse, as did the pain.
Partly it was pain in my legs, which was getting really intense at this point, but also it was from seeing my goal-time slipping away from me: there was nothing I could do about it.
The last 5k were gruesome – I just wanted to finish the race at this point, and didn’t care what sort of time I’d finish in. Finally limping over line, I was happy it was over, but depressed at the same time.
As it turned out, I did beat last year’s time of 3:12:39, with a 3:11:37 finish. So, was all the training leading up to the race worth a finish time of only 62 seconds faster than last year?
Well, I can’t pretend that I wasn’t hugely disappointed with my result, but at the same time, I keep telling myself that a 3:11 marathon finish isn’t shabby by any measure. I know I’m faster than that, but I’m still weighing up whether I want to try to prove it or not next year.
In the meantime, I’m taking a break from running – give it a few months and perhaps I’ll be back on the roads again, but for now, I’m enjoying the extra free time I suddenly seem to have!
Race results, and splits from my watch during the race are below:
|Net place||25th of 370 total finishers||24th of 259 male finishers||16th of 136 M18-39 finishers|
|21.2||4:34||1:26.22 ( half way marker )|
|38,39,40,41,42.2||49:17 – that’s wrong, I forgot to stop my watch|
Here is a selection of the last three pairs of running shoes I’ve owned:
- they’re all Asics Nimbus x, and all from successive years. I’ve trained for and run a marathon in each of the last two pairs, and am currently putting my most recent pair through their paces with another race in the next few weeks. I’m running 6 days a week at the moment on the road, with a maximum of about 50 miles/week. I really like these shoes – honestly, no complaints and no injuries to speak of.
The thing is, it’s getting to be fairly obvious to me where my runners wear out the fastest – just on the outside edge of the heel, and at a fairly consistent area on the midsole. (yes, I know I should have changed the red & white pair well before they got into the state they’re in)
So, here’s my idea: could someone please come up with a mechanism that allows me to present these old shoes to my running shop the next time I’m getting a new pair so that they can do a quick 3d-scan of the old shoes, then send the scan off to the factory along with my weight and height, and a brief description of the sort of running I do.
There, they would take the scans, run them through a CAD system, do some magic, then 3d-print a new pair for me with slightly harder-wearing rubber on the bits that wear out most quickly for me. They’d send them back to the shop a few weeks later and I’d pick them up. I would even pay slightly more for this, if it means my runners don’t wear out as quickly, and they’ll have a loyal customer for life.
Yes, I admit to the possibility that perhaps harder-wearing parts of the sole might mean less cushioning in those parts, and that maybe the runners are designed to fall apart at the moment, in the interests of protecting my joints? Still, it’d be an interesting experiment I think.
I’ve had a few people asking for the focaccia recipe I mentioned here, so here goes.
Other than the description above, Bob mentions that you should mix the extra virgin olive oil together with the potato using a fork until it’s smooth before adding it to the dough.
Yes, potato in a focaccia recipe – some say this betrays our Irish roots, they’re probably right, but it makes for pretty fabulous bread.
One of the design-goals for IPS was that scripting should be moved out of the packaging system, preferring scripting to only ever occur on the environment the software was intended to run in (eg. after boot, rather than at install time) – Stephen talks more about that here.
Related to that, I’ve had a chance to do a bit more work on pkgsend(1) recently to help developers through this transition. With the putback of:
changeset: 2276:d774031d55bd user: Tim Foster date: Thu Mar 24 09:56:10 2011 +1300 description: 17826 pkgsend should warn for potentially problematic SVR4 packages 17891 pkgsend could convert more parameters from a SVR4 pkginfo file 17905 pkgsend should return a better warning for multi-package datastreams
pkgsend is now more helpful in how it deals with SVR4 packages.
Up till now, when converting packages from SVR4 to IPS, pkgsend has ignored any preinstall, postinstall, preremove, postremove and class-action scripts that may have been present in the SVR4 package. So while this would give users an installable IPS version of their packages, there’s a good chance that the packages wouldn’t have functioned properly, as the scripts would not have run.
With this small change, we now report errors when encountering scripting, giving the package developer a heads-up that a little more work is needed in order to properly convert their package and will tell them exactly what things need attention. For example:
timf@linn $(pkgsend open firstname.lastname@example.org) timf@linn pkgsend import ./SUNWsshdr pkgsend: Abandoning transaction due to errors. pkgsend: import: ERROR: script present in SUNWsshdr: postinstall pkgsend: import: ERROR: class action script used in SUNWsshdr: etc/ssh/sshd_config belongs to "sshdconfig" class
Investigating this particular sample package (an old S10 version of ssh) the postinstall script makes sysidconfig do ssh host-key-generation, (something that’s now done automatically by the SMF method script for ssh) and the sshdconfig script removes any “CheckMail” entries from any preserved sshd_config files previously installed.
Along with these changes, we’re also converting more information from the SVR4 pkginfo file – populating the package description and summary fields automatically, and creating
pkg.send.convert.* attributes for other pkginfo parameters that may have been defined.
The expectation is that package developers will change those attribute names to a name that better suits their needs, either by editing the manifest that
pkgsend generate produces for you from the SVR4 package, or by using
Hopefully these changes make pkgsend a little more helpful to developers when converting their SVR4 packages over to IPS.
I wrote a short introduction to that, here on pkg-discuss. It’s all very much work in progress as yet, but so far, is proving to be a lot of fun.
I should have written about this a few days ago, but better late than never.
With the putback of:
changeset: 2236:7b074b5316ec user: Tim Foster date: Tue Feb 22 10:00:49 2011 +1300 description: 16015 pkgdepend needs python runpath hints 16020 pkgdepend doesn't find native modules 17477 typo in pkgdepend man page 17596 python search path not generated correctly for xml.dom.minidom 17615 pkgdepend generate needs an exclusion mechanism 17619 pkgdepend generate is broken for 64-bit binaries when passing relative run paths
pkgdepend(1) has become better at being able to determine dependencies. I’d done some work on
pkgdepend before, and it was nice to visit the code again.
To those unfamiliar with the tool, I thought I’d write an introduction to it (which I should have written last time).
pkgdepend in a nutshell
pkgdepend is used before publishing an IPS package to discover what other packages are needed in order for the contents of that package to function properly. The packaging system then uses those dependencies whenever a package is installed to automatically install those dependencies for you.
During the creation of a package, the process of running
pkgdepend on your manifests is broken into two phases, each with its own subcommand.
The first is called ‘generate’. This is where the code examines each of the files you’re intending to publish in your package. Depending on the type of file it is, we look for clues in that file to see what other files it may depend on.
Those clues could be as simple as the path that comes after the ‘#!’ in UNIX scripts (so for a Perl script with ‘#!/usr/bin/perl’ at the top of it, obviously you need to have Perl installed in order to run the script) or could be complex, such as digging around in the ELF headers in an ELF binary to find the “NEEDED” libraries, determining Python module imports in a Python script, or looking at ‘require_all’ SMF services in an SMF manifest.
There’s a list of all the things used so far to determine dependencies in the
pkgdepend(1) man page.
pkgdepend has gathered the set of files it thinks should be dependencies for the files you’re delivering, it outputs another copy of your manifest, this time with partially complete ‘depend’ actions.
I say partially complete, because all we know at this stage, is that your package will need a bunch of files in order for it to function properly: we don’t yet know what delivers those files. That’s where the second phase of
pkgdepend comes in: dependency resolution.
During dependency resolution, via the ‘
pkgdepend resolve‘ subcommand, we take that partially complete list of depend actions, and try to determine which package delivers each file the package depends on.
In order to do this,
pkgdepend needs to be pointed at an image populated with all the packages that package could depend on – in most cases, the image is simply the machine you’re building the packages on, (remember, in IPS terms, every package is installed to an “image”: your running copy of Solaris is itself an image) though you could choose to point ‘
pkgdepend resolve‘ to an alternate boot environment containing a different image.
Assuming we’re successful, you are then presented a version of your package with all dependencies converted from just the filenames needed to satisfy each dependency, to the actual packages IPS will install for you in order for your package to function.
Things that can go wrong
I say “assuming we’re successful” because, unfortunately, sometimes we’re not.
There are several things that can go wrong:
- an ELF header entry could be incorrectly specified at build time, or could contain
pkgdependdoesn’t know how to expand
- a file might be delivered by multiple packages on your system, in multiple places
- a python script might modify
sys.path, a shell script might modify
- we could deliver scripts only meant to be read, not run (demo scripts, for example) which could cause either fake dependencies, or dependencies which could never resolve
All of the things above can result in error messages from
pkgdepend, where it’s unable to determine exactly what we should be depending on – this is the part of
pkgdepend I was trying to fix in my putback.
It fixes a few bugs in
pkgdepend when dealing with Python modules and kernel modules, and it introduces two new IPS attributes:
pkg.depend.bypass-generate, is used to specify regular expressions to files on which we should never generate dependencies. This gets us around the cases where multiple packages deliver files in several places, or where $VARIABLES aren’t being expanded. Bypassing dependencies this way is good, though you do need to be careful where and how you apply it — if you bypass a legitimate dependency, then there’s a good chance your package won’t function properly if the packages it depends on aren’t installed.
pkg.depend.runpath, is used to change the standard set of directories that
pkgdepend looks in, per-file-type in order to search for file-dependencies. This gets us around the case where programs are installed in non-standard locations.
Alongside this work, I’ve been doing work on the ON package manifests to greatly reduce the numbers of
pkgdepend errors being reported during the ON build. (sadly, I can’t share the work on the ON manifests, but they will go back once snv_160 is available internally. If you’re an ON engineer there’ll be a Flag Day attached to this, making snv_160 the minimum build on which you can build the gate) Quite soon after that, we’ll be able to enable error-reporting from the
pkgdepend phase of the build, and that will be fabulous.
I’d strongly encourage those working on Illumos and other derivatives of the OpenSolaris codebase to investigate the new
pkgdepend functionality, and put in the time to get their gate pkgdepend-clean too.
Why? Well, in my view, one of the problems with SVR4 packaging was that it lacked any sort of automatic dependency analysis. This meant that packages declared manual, often-bogus dependencies on other packages – and dependencies that aren’t correct make minimisation of systems very difficult.
When we determine dependencies automatically, minimisation becomes a lot easier.
Crucially, so does package refactoring: if we split or merge packages, so long as those new packages are installed on the image being used to resolve dependencies, the packages that have dependencies on those split/merged packages automatically pick up the new package names the next time they’re published.
However, without actually checking the exit status from the
pkgdepend phase of the build, you’re having to insert more manual dependency actions than should be strictly necessary, and that’s a bad thing.
Of course, sometimes we can’t avoid inserting manual dependencies –
pkgdepend isn’t finished yet, and there’s more we could be doing to determine dependencies at package publication time, however the tool does make life a lot easier. So, if you’re ever tempted to insert a manual dependency into your package, please do think carefully about it, and please add a comment to the manifest explaining in detail why that manual dependency is really required.
2010 was a busy year for myself and the family. Here are some of the things I got up to:
- Moved job
I joined the IPS team in January
- Moved company
Sun Ireland disappeared in April, and I became an Oracle employee. Personally, that’s working out very well and I’m not noticing much difference in terms of my day to day work. Yes, the company has a different culture – and if there were a “Top Gear Cool Wall” for IT companies, in my opinion, Oracle would likely be “Seriously uncool“, with Sun as “sub-zero“. I think the Solaris engineering culture has survived the transition so far, at least at my level. We’ll see what the New Year brings.
- Moved country
I moved from Ireland to NZ in May, having gone through the long process of applying for residency here. That this coincided with the economic conditions in Ireland was purely chance. Our decision to move was not related to work or the economy, it was purely based on a willingness to stretch our horizons and see some more of the world. So far, NZ is treating us well – ups and downs.
- Ran another marathon
- Was mentioned in a piece by the Irish Times
(albeit with the errors that they used my wife’s maiden name, and mis-credited my photo to Alan Betson, though I’m happy to have my photographs confused with his, I’m really not in the same league
- Did a total of 24 putbacks to the pkg-gate.
This covered 66 bugs or RFEs – some were minor and didn’t take a lot of work to get right, like 12723, some were major like 13536 and were projects in themselves. All were a lot of fun to work on though, and I feel privileged to continue to be employed on something I enjoy.
Looking forward to 2011, I’m hoping to explore more of NZ. If you follow me on Twitter, subscribe to my Flickr feed or look at some of the stuff I’ve put up on Smugmug, you’ll see some of the photos I’ve taken this year: this country really is incredibly beautiful – indeed, as is Ireland. You should visit both places, then pick which one you want to live in.
There’s lots to do next year – we’re moving house in a few weeks time, and I get to set up a new home office, in a separate building to the house – so I’m hoping my struggle for a healthier work/life balance will start tipping in the right direction.
I’ll also try to keep up the running – possibly entering the Wellington marathon again to see if I can better this year’s time, though being honest, I’m not sure I’ve got a faster pace in me – I’ll aim to improve though. Having recovered from the marathon, I’m doing a lot less road-running and a lot more mountain/off-road running now so perhaps it’s time to look for a different sort of event instead?
Roll on 2011 :-)
There’s been some crowing recently about how wonderful it is, that a scripting language is no longer a dependency for an OS build.
My opinion is that this is a shame on many levels: it’s a shame because the time spent rewriting all of this code could have been better spent elsewhere, it’s a shame because this new code presumably has integrated a heap of new bugs and it’s a shame because the bar was raised for potential contributors to their codebase: if you don’t know C, you can’t write code for them.
Most importantly though, I believe that scripting languages have a better place in /usr/bin and as helper components for core OS functionality than some folks seem to believe.
I’ve been writing in Python for a few years now, first as part of our Xen port, now on IPS, and I think that the sorts of things most OS commands do is far easier to express, code and debug in Python than it is in C.
Perhaps it’s me, but I’m much more comfortable firing up an editor and debugging a script I can see, than I would be having to download and setup a complex build environment, compile sources (if that source is even available :-/ ) and drop binaries in place.
Excising Python from an operating system is like chopping off your arm. Sure, you’ve fewer dependencies now (the original reason cited for removing the code in question) – no need for wrist watches, and you won’t get worn patches on your jumpers at the elbow, etc. but I’m not convinced it was the right move.
Still, each to their own – this is merely my opinion.
changeset: 2135:6eeb55920e13 tag: tip user: Tim Foster date: Wed Nov 10 11:32:32 2010 +1300 description: 3419 history command -- ability to specify dates or date ranges desired 17012 pkg history should include boot environment or snapshot information 17013 pkg history subcommand should display boot environment and snapshot information 17222 pkg history could use a -o option
the pkg ‘history’ subcommand now has some new features.
We now record the name of the boot environment the operation was applied to, any ZFS clones created, and any ZFS snapshots taken during the course of the operation.
In addition, pkg history can also accept a comma-separated list of column names to print different output. The known column names at the moment are:
- The name of the boot environment this operation was started on
- The name of the client
- The version of the client
- The command line used for this operation
- The time that this operation finished
- The user id that started this operation
- The new boot environment created by this operation
- The name of the operation
- A summary of the outcome of this operation
- Additional information on the outcome of this operation
- The snapshot taken during this operation. This is only recorded if the snapshot was not automatically removed after successful operation completion
- The time that this operation started
- The total time taken to perform this operation (for operations that take less than a second, “0:00:00″ will be printed)
- The username that started this operation
The old “result” column has been split into “result” and “reason” to preserve field formatting, and the old “time” column has been renamed to “start”. The “time” column now contains the total operation time (“finish” – “start” times) – I figured, that calculating the total operation time might be useful, rather than expecting users to do it manually.
Finally, pkg history gets a ‘-t’ flag, allowing users to specify a comma separated list of dates, or ranges of dates they’re interested in. Previously users could only choose to see all events or the last ‘n’ events with the -n flag.
I really like the history subcommand – I’ve found that being able to see over time which packages have been installed and removed from the system, and which operations have failed or succeeded is extremely useful. Being able to find detailed information about how packages have been managed over time gives quite an insight into how people use software. It’d be interesting to use this as input on deciding how to craft custom distributions of Solaris that contain the software that people use in the real world.
We didn’t have a history function in SVR4 that I’m aware of – another point in favour of IPS. History Lives on in Historic Historyville!