This is one of the more common FAQs, so I will try to provide an
answer which covers all of the issues involved and questions raised in
this thread, and can then be referred to each time it comes up.
Let's start with some numbers:
- pkgsrc is currently comprised of 14,930 packages, though this number is
somewhat inflated due to multi-version support (e.g. every python
package is built multiple times for each major version of python we
provide). It is around 12,153 unique packages.
- Adding select packages from pkgsrc-wip and pkgsrc-joyent, we are able
to provide almost 13,000 binary packages for SmartOS.
- Over the lifetime of pkgsrc (since August 1997) there have been 291
individual committers, and 218,185 changes committed. Since Jan 1st
this year, 90 individual committers have so far contributed 7,554
changes. Only members of The NetBSD Foundation (TNF) have the ability
to commit changes, however we will often include changes from the
community.
- Most packages do not have a specific maintainer, and are handled by
whichever pkgsrc developer has time and inclination to work on them.
Some packages are maintained by community members, who will forward
updates to a committer or maintain them in pkgsrc-wip.
- Joyent employ two pkgsrc developers, myself and Filip. Our main areas
of focus are on producing quarterly binary package sets, providing
packages requested by customers, and infrastructure work relevant for
our target platform (e.g. multiarch support, Manta images, etc).
- I am not aware of any other commercial entity who employ pkgsrc
developers for the sole focus of working on pkgsrc and related areas.
- pkgsrc supports 22 platforms, some better supported than others, but we
try to ensure that packages work on as many of those platforms as
possible. The same source tree works on SmartOS, Linux, OSX, Minix,
*BSD, Cygwin, and even Haiku which is being actively developed.
- We release every 12 weeks, when a brand new branch cut with the very
latest packages is made available.
As you can see, we are heavily indebted to the pkgsrc developers and wider
community. Without their input, we simply wouldn't be able to offer the
number of packages we currently do, nor would they be anywhere close to
being up-to-date.
Compared to Debian/Ubuntu for example, they:
- have significantly more developers and maintainers, and a much larger
user base from which to accept bug reports.
- have Canonical and other companies employing a large number of
developers to work full-time on packages.
- primarily only concentrate on a single platform - Linux.
However, despite the massive difference in resources, I think we compete
very strongly, and personally I often find I need to use pkgsrc on Linux
machines to give me a version of software I need which isn't horrendously
out of date (nodejs 0.6 anyone?). Plus, I can use the exact same package
manager across any OS I might conceivably want to use.
So, on a pure numbers basis we're already doing very well to compete at
all, and with our aggressive release policy you are much more likely to be
able to provision a node with the versions you need than on other systems
where you either need to wait for a new release or rely on third party
package repositories to supplement your install.
Next, package updates:
- pkgsrc trunk is a rolling unstable release, where we import and update
the very latest software releases. We strive for it to be internally
consistent, so that changes are patched across the tree. For example,
with the recent giflib major update, we patched a large number of
packages ourselves to support the new API. We perform regular bulk
builds to expose such issues.
- Each quarterly branch should be thought of as a major release, akin to
changing Ubuntu LTS, RHEL 6 to 7, Solaris 9 to 10, etc. Even though
there are only 12 weeks in between each release, as shown above each
release will normally include many thousands of changes, and will
almost certainly include a number of major upgrades.
- No effort is made to retain compatibility to older releases or provide
an upgrade path. This is the crux of the issue, but one which has been
made deliberately. We would much rather offer you the most recent
packages each quarter, than be hamstrung to older releases for years.
- We will occasionally increase the default versions of packages, so that
new users get the most appropriate version. For example in
pkgsrc-2013Q4 we changed a number of defaults:
http://pkgsrc.joyent.com/changes.html#default-version-changes
- Many packages are actively hostile to upgrades. One of the more
notable is postgres, which requires a manual dump/restore of databases
for each upgrade. If you try to blindly upgrade from 9.2 or earlier to
9.3, which will happen if you try to upgrade through pkgsrc-2013Q4:
$ pkgin in postgresql92-server
$ svcadm enable svc:/pkgsrc/postgresql:default
$ createdb -U postgres test
$ pkgin rm postgresql92-server postgresql92-client
$ pkgin in postgresql93-server
$ svcs -a | grep postgres
online 9:44:42 svc:/pkgsrc/postgresql:default
$ svcadm restart postgresql
$ svcs -a | grep postgres
maintenance 9:44:56 svc:/pkgsrc/postgresql:default
$ tail -2 /var/log/postgresql.log
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 9.2,
which is not compatible with this version 9.3.4.
With nearly 13,000 packages on offer, the chances of similar issues in
other packages is very high, and there's simply no way we can
accommodate these.
- Occasionally a package will be renamed. Probably the canonical example
of this was the recent scmgit->git change. Whilst we have support in
pkgsrc for this at a source level, pkgin has no notion of renames at
this time, and won't be able to handle this.
- Finally, we cannot guarantee that a cross-branch upgrade won't break
any third party applications you have built yourself, or that all
dependencies will be able to be resolved. Sometimes packages are
removed or their dependencies change, and this may result in an
impossible constraint depending on the packages you already have
installed.
In addition, there are some non-pkgsrc considerations to upgrades:
- Our images aren't just package updates. We also include changes from
illumos to the base /etc, SMF, /var etc., as well as any zoneinit and
mdata changes required to support the latest SDC. An example of how
this may affect you is that a while back we removed a number of users
from the illumos-provided /etc/passwd such as 'mysql', and newer
package sets assume this user does not exist by default.
By only upgrading packages, you may be missing out on other changes
which could affect the operation of your zone in non-obvious ways.
Changes to the 'seed' image and our image creation process can be
tracked in our imagetools repository:
https://github.com/joyent/imagetools
- We believe that re-provisioning is just better practise. It enforces a
separation of configuration and data, and encourages the use of high
level configuration management tools to design and workflow your
deployments. On the contrary, applying blind updates to an existing
install very quickly turns it into a black box, with no way to fully
understand the dependencies involved and making it almost impossible to
be able to replicate that environment from a clean install.
I think that covers the various issues around upgrades, and why we don't
support them. In my personal experience, even on systems where upgrades
are recommended such as Debian/Ubuntu, I've been bitten at times when an
upgrade didn't handle the particular changes I had made, so even there
it's not foolproof.
Having said all that, for those of you who want to upgrade regardless, the
following instructions are probably the most reliable. This is the
process I tend to use on my OSX pkgsrc installs, where I do not have any
daemons with configuration files, and just run a small set of packages
(mutt, nodejs, cdparanoia, etc.)
- Upgrade pkgin and the package tools first.
$ pkg_info | egrep '^(pkgin|pkg_install)-[0-9]'
pkgin-0.6.4nb4 Apt / yum like tool for managing pkgsrc binary packages
pkg_install-20130902 Package management and administration tools for pkgsrc
$ ed /opt/local/etc/pkg_install.conf >/dev/null <<-EOF
s/2013Q4/2014Q1/
w
q
EOF
$ pkg_add -u pkg_install pkgin
$ pkg_info | egrep '^(pkgin|pkg_install)-[0-9]'
pkgin-0.6.4nb5 Apt / yum like tool for managing pkgsrc binary packages
pkg_install-20130902nb1 Package management and administration tools for
pkgsrc
- Save your configuration diffs. Configuration files are handled by
pkgsrc in the following way:
- Each package installs an example configuration file into
$PREFIX/share/examples/<package>/. For example:
/opt/local/share/examples/pkgin/repositories.conf.example
- If there is no existing configuration file in /opt/local/etc for
that package, the example file is copied into place, e.g.:
# On a clean system with no pkgin
$ pkg_add pkgin
pkgin-0.6.4nb5: copying
/opt/local/share/examples/pkgin/repositories.conf.example to
/opt/local/etc/pkgin/repositories.conf
...
- On package removal, if the /opt/local/etc version is identical to
the example, it is removed. If any changes have been made, it is
left in place.
This means that any changes to configuration files are preserved, but
if you haven't made changes then you will always get the latest example
configuration. It also provides you with a way to see what changes you
have made, for example:
$ diff /opt/local/share/examples/postfix/main.cf
/opt/local/etc/postfix/main.cf
657a658,712
>
> #
> # Joyent customisations
> #
> alias_database = hash:/opt/local/etc/postfix/aliases
which shows the changes we make to the default postfix configuration
in our imagetools scripts.
If you have modified configuration files, it is worth taking a diff of
them before upgrading, so you can see exactly what changes you made.
The upgraded package may have significantly changed the configuration
file, making it much harder to see your diffs if you upgrade first.
- Finally, update the pkgin repositories.conf and do a full update.
$ ed /opt/local/etc/pkgin/repositories.conf >/dev/null <<-EOF
/^http/s/2013Q4/2014Q1/
w
q
EOF
$ pkgin update
$ pkgin full-upgrade
It's recommended not to use the '-y' argument to pkgin in this case,
and pay careful attention to the list of packages which will be
upgraded/installed.
At this point you may need to restart SMF services with a /pkgsrc prefix
to ensure they pick up the new binaries.
Some final points I'd like to make.
- There is nothing fundamentally missing in pkgsrc that means we can't
improve the upgrade experience. Our INSTALL scripts are executed at
various points during a package install and removal, so there's no
reason they can't include an attempt at upgrading. Patches welcome!
- As I previously mentioned we are heavily dependent upon our community.
Packaging is one area where people of any ability can get involved, you
don't need to be a kernel hacker, and it's a great way to learn new
skills and contribute back to the community. I've already overseen two
new additions to the TNF pkgsrc developers from this community, and I'd
love to see lots more.
Anyway, hope that's helpful. If I missed anything, please let me know and
I'll try to turn this into something of a canonical document to avoid it
being rehashed each time.
Regards,
--
Jonathan Perkin - Joyent, Inc. - www.joyent.com
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription:
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
