Re: [Toolserver-l] operating systems and web servers

[Aryeh Gregor]

On Tue, Jun 7, 2011 at 5:22 AM, River Tarnell
 wrote:
> ZFS snapshots are nice, but OTOH ZFS can't remove storage from a pool or
> do any kind of online relayout, both of which are standard features in
> VxVM (and I assume in Linux LVM as well, although I have no experience
> of it.)

Linux LVM can do anything you want online, in my experience: grow and
shrink logical volumes, move logical volumes between physical volumes
within the same volume group, and add or remove physical volumes, for
instance.  I've found that moving physical volumes is much slower than
it should be, though, and I've had bad experiences once or twice
moving a PV that's being actively used (system hang, but no data loss
and the operation continued automatically after power-cycling and
completed successfully).

md isn't quite so good about what it can do online, but it's getting
better recently.  It can add or remove devices from RAID1/5/6 online,
and even change the RAID level online sometimes.  If it doesn't do
what you want, and you have enough storage attached, you can always
create the new md device and move the old one's contents to it online
using LVM, possibly leaving one or both RAID arrays degraded while you
do so to get the extra space necessary.

ext4 can grow online if you grow the device it's sitting on, but it
can't shrink online, and (AFAIK) nor can any other high-profile Linux
filesystem except btrfs.

So in practice, on Linux you can do all your I/O-related stuff online
except shrinking filesystems.  This is usually enough, but the
inability to shrink filesystems online can occasionally be a pain.
btrfs is slated to fix this, in addition to solving world hunger and
curing cancer.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] operating systems and web servers

[Marc A. Pelletier]

On 07/06/2011 4:49 AM, River Tarnell wrote:
>
> Why Linux and not FreeBSD, or Solaris, or OpenServer, or MP-RAS, or ...?
> I've asked this many times in the past (usually when someone says "I
> want Linux") and never had a real answer that I can remember.[0]

Availability of expertise.

Just here at Ubisoft, for instance, there are about 25 Linux sysadmins 
(ranging from intermediate to guru-level).  Of all of us, there are only 
five that have Solaris or *BSD experience, and only *two* who have both.

Whatever caused this originally, Linux ended up with a much bigger 
mindshare than the BSD did.  You'll find more users and sysadmins with 
CentOS or Ubuntu experience than all other unices combined once we old 
farts start retiring.

-- Coren / Marc


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] operating systems and web servers

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Jeremy Baron:
> Also, what do you lose with FreeBSD?
 
> so far I can think of:
> * zfs dedupe

ZFS version 28, including dedup, was integrated into FreeBSD head 
(9-CURRENT) in February.  I imagine it will make it back into 8.x 
eventually, but although dedup is nice on paper, I'm not sure it's 
actually very useful. You need so much memory to cache the DDT (since 
putting it on disk kills performance) that it's usually cheaper to just 
buy more storage.

It's also unlikely that we would ever migrate the /home NFS server to 
FreeBSD (at least in the near-term future), since it lacks the HA 
clustering and volume management that Solaris has.  (This is an example 
of an area where Linux *does* have an advantage over FreeBSD.)

It's *also* unlikely that we would ever put /home on ZFS, whatever OS we 
used.  I don't mind ZFS for root, and I use it at home (on both FreeBSD 
and Solaris) without problems, but we've had some unfortunate issues 
with it at the TS that have made me a bit wary.

> * there's no longer a single "goto" company to get support from
> (guessing...) so you lose that annual cost but it may be non trivial to find
> the right hacker to find/fix a problem in an emergency. of course much of
> that support is likely to come for free (a guess)

This is true, and is something that came up the last time we discussed 
this internally.  I'm not too worried about this; the login server 
environment doesn't really tax the OS, and I can't think of many (any?) 
problems we've had there that would require vendor support.  

There are places like iXsystems  
which offer third-party support for FreeBSD.  I don't have any 
experience with that, or any idea how it compares price-wise.

> * you're still a different platform from the rest of wikimedia (basically
> the foundation) and so lose economy of scale

Not sure what economies we could gain here.  The Toolserver is 
completely separate from Wikimedia's infrastructure (except that we use 
the same colo), and both of us prefer it that way.  IOW, even if we 
used Linux, TS systems could not become just another Wikimedia server.

> * you mentioned SGE would still be available but it may be unmaintained
> (last I checked its enwp article, it said oracle was close sourcing it) just
> to keep in mind, I wouldn't stay just for SGE

SGE is dead on every platform, including Solaris.  The replacement is
either Oracle Grid Engine (which costs money), or one of the various 
forks of the open-source SGE (which should work on both FreeBSD and 
Solaris).  We're still using SGE because I want to see which (if any) 
fork gains acceptable before switching.

> things you lose with linux vs. FreeBSD:
> * zfs is gone and (assuming lvm2 + traditional RAID vs. zfs) generally thin
> storage provisioning and snapshotting is much more limited and wasteful and
> fragile/more room for error. also most storage expansions will end up w/
> transition periods that have *no* redundancy.

ZFS snapshots are nice, but OTOH ZFS can't remove storage from a pool or 
do any kind of online relayout, both of which are standard features in 
VxVM (and I assume in Linux LVM as well, although I have no experience 
of it.)

In any case we don't store data on login servers.

> * zones/jails

Linux has lxc and OpenVZ, which are both very similar to zones, as well 
as Xen or KVM for full virtualisation.

> I haven't kept up with the status in the last ~8 months but debian-kfreebsd
> may have matured enough to warrant a look. (See #debian-kbsd on oftc)

Never really saw the point of that.  If you want Linux, use Linux.  (I 
guess at the moment it's the only way for Linux users to get a useful 
ZFS?)

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (SunOS)

iEYEARECAAYFAk3t7cAACgkQIXd7fCuc5vIBUACfcCivOSjgrnbZRSRu3W7lc+Bq
TpgAn0xAsVMmFkCnaUQTx165O6nZKyRs
=d221
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] operating systems and web servers

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Daniel Kinzler:
> why FreeBSD and not Linux?

Why Linux and not FreeBSD, or Solaris, or OpenServer, or MP-RAS, or ...?  
I've asked this many times in the past (usually when someone says "I 
want Linux") and never had a real answer that I can remember.[0]

The way I see it, choice of operating system has very little impact on 
users because users don't, generally, interact with the operating system.  
There are a few specific cases which make a difference (I can think of 
cron, 'ps' and top/prstat), but apart from that, the software users 
actually use is the stuff in /opt/ts, which is independent of operating 
system.[1]

For servers other than login servers (like databases) there are 
additional considerations, since those make a lot more use of OS 
features, like storage management.  I've previously considered and 
rejected FreeBSD for use as a database server (in favour of staying with 
Solaris + VxVM).  

To me, FreeBSD seems to be an ideal candidate for a platform to layer 
/opt/ts on top of: it provides a base operating system which is 
reliable, has a good feature set in areas which only the OS can provide 
(e.g. DTrace, ZFS, auditing), and it's simple to install and maintain.

- river.

[0] I know some people are concerned about use of proprietary software 
on the Toolserver, but that's a separate issue, and in any case FreeBSD 
and Linux are equally open source.

[1] This assumes that all the software we provide works on FreeBSD, of 
course.  I haven't verified this, because we provide ~650 packages and 
it would take many hours, so there's no point doing it unless we 
actually decide to change OS.  However, I don't foresee any problems.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (SunOS)

iEYEARECAAYFAk3t5ikACgkQIXd7fCuc5vLokACfUbRk0fIc6yU6tdXaS+oTr26/
0G8AoMGuJ149lizDh9KMO426taRJVDr8
=Hom2
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] operating systems and web servers

[Jeremy Baron]

(Take with some grains of salt: never used FreeBSD or Solaris shells myself
and am not a ts-admin. Also, this is a cell phone and I shouldn't even be
awake now!)

Also, what do you lose with FreeBSD?

so far I can think of:
* zfs dedupe
* there's no longer a single "goto" company to get support from
(guessing...) so you lose that annual cost but it may be non trivial to find
the right hacker to find/fix a problem in an emergency. of course much of
that support is likely to come for free (a guess)
* you're still a different platform from the rest of wikimedia (basically
the foundation) and so lose economy of scale
* you mentioned SGE would still be available but it may be unmaintained
(last I checked its enwp article, it said oracle was close sourcing it) just
to keep in mind, I wouldn't stay just for SGE

things you lose with linux vs. FreeBSD:
* zfs is gone and (assuming lvm2 + traditional RAID vs. zfs) generally thin
storage provisioning and snapshotting is much more limited and wasteful and
fragile/more room for error. also most storage expansions will end up w/
transition periods that have *no* redundancy.
* zones/jails

I haven't kept up with the status in the last ~8 months but debian-kfreebsd
may have matured enough to warrant a look. (See #debian-kbsd on oftc)

-Jeremy
On Jun 7, 2011 2:58 AM, "Daniel Kinzler"  wrote:
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] operating systems and web servers

[Daniel Kinzler]

hi river

i know we have talkedabout this before, but for the benefit of other users, let
me ask the obvious question: why FreeBSD and not Linux?

-- daniel

On 07.06.2011 04:04, River Tarnell wrote:
> Hi,
> 
> So, this is not an announcement of any intent to change anything, I just
> want to get an idea of how people feel about two things we could,
> perhaps, change in the future:
> 
> 1. ZWS to Apache on the web server
> 2. Solaris to FreeBSD on login servers
> 
> #2 depends on #1, so it seems sensible to discuss both together.
> 
> I don't have any strong opinion about either of these myself, but I
> would like to hear what users think.
> 
> ZWS to Apache:
> 
> I know it annoyed people when we moved from Apache to ZWS initially,
> because rewrite rules had to be redone, some .htaccess stuff stopped
> working, etc.  At the time we were using mod_suphp for per-user (suexec)
> PHP scripts, and it was extremely inefficient; the system spent most of
> its time doing nothing.  ZWS allowed us to fix the problem cheaply (no
> new hardware required), and I think for most users it worked just the
> same.
> 
> Since then, two things have changed: firstly, Apache with mod_fcgid now
> has decent FastCGI support, and with only a little work could be made to
> support suexec PHP as well.  Secondly, ZWS is now in maintenance, and
> won't see any further development (so it might be better to switch now,
> rather than wait until one month before ZWS support ends entirely and be
> forced to switch).
> 
> With that in mind, it makes sense to consider moving back to Apache.
> The main downside is that rewrite rules would have to be converted back
> to Apache format (mod_rewrite).  OTOH, .htaccess features missing from
> ZWS would be available again (I don't know if anyone actually needs
> this, but I believe at least a few users have complained about missing
> features.)
> 
> Solaris to FreeBSD:
> 
> Of the two changes, I think this one would actually be the less
> disruptive.  For users, nearly everything would stay the same: we
> already provide the GNU userland ('ls', etc) by default (and would
> continue to do so) and the third-party software in /opt/ts would be
> identical, as would cronie, SGE, Perl/Python/..., etc.
> 
> Software-wise, since nothing would really change, I don't see any
> particular advantages for users.  Disadvantages: 'ps -eaf' would stop
> working ;-) and anyone with locally-compiled software (C/C++, or XS Perl
> modules, etc.) would need to recompile them.
> 
> For us (admins), the main advantage is reduced maintenance overhead:
> FreeBSD releases a new minor version about once a year, and supports
> each for 2 years; each release branch only gets very infrequent updates
> for security or errata.  In comparison, there is a new Solaris update
> every 6 months, and during yesterday's maintenance we installed 358 (!)
> separate patches.  Oracle doesn't provide a security-updates-only
> release, and it's difficult to mix-and-match patches (e.g. to only get
> security patches).
> 
> This doesn't directly affect users, but fewer OS changes should lead to
> less lengthy / disruptive maintenance and less frequent reboots.  OTOH,
> I don't know if this has a noticeable impact on users at the moment, and
> the previous maintenance was the first for ~170 days.
> 
>   - river.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] operating systems and web servers

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

So, this is not an announcement of any intent to change anything, I just 
want to get an idea of how people feel about two things we could, 
perhaps, change in the future:

1. ZWS to Apache on the web server
2. Solaris to FreeBSD on login servers

#2 depends on #1, so it seems sensible to discuss both together.

I don't have any strong opinion about either of these myself, but I 
would like to hear what users think.

ZWS to Apache:

I know it annoyed people when we moved from Apache to ZWS initially, 
because rewrite rules had to be redone, some .htaccess stuff stopped 
working, etc.  At the time we were using mod_suphp for per-user (suexec) 
PHP scripts, and it was extremely inefficient; the system spent most of 
its time doing nothing.  ZWS allowed us to fix the problem cheaply (no 
new hardware required), and I think for most users it worked just the 
same.

Since then, two things have changed: firstly, Apache with mod_fcgid now 
has decent FastCGI support, and with only a little work could be made to 
support suexec PHP as well.  Secondly, ZWS is now in maintenance, and 
won't see any further development (so it might be better to switch now, 
rather than wait until one month before ZWS support ends entirely and be 
forced to switch).

With that in mind, it makes sense to consider moving back to Apache.  
The main downside is that rewrite rules would have to be converted back 
to Apache format (mod_rewrite).  OTOH, .htaccess features missing from 
ZWS would be available again (I don't know if anyone actually needs 
this, but I believe at least a few users have complained about missing 
features.)

Solaris to FreeBSD:

Of the two changes, I think this one would actually be the less 
disruptive.  For users, nearly everything would stay the same: we 
already provide the GNU userland ('ls', etc) by default (and would 
continue to do so) and the third-party software in /opt/ts would be 
identical, as would cronie, SGE, Perl/Python/..., etc.  

Software-wise, since nothing would really change, I don't see any 
particular advantages for users.  Disadvantages: 'ps -eaf' would stop 
working ;-) and anyone with locally-compiled software (C/C++, or XS Perl 
modules, etc.) would need to recompile them.

For us (admins), the main advantage is reduced maintenance overhead: 
FreeBSD releases a new minor version about once a year, and supports 
each for 2 years; each release branch only gets very infrequent updates 
for security or errata.  In comparison, there is a new Solaris update 
every 6 months, and during yesterday's maintenance we installed 358 (!) 
separate patches.  Oracle doesn't provide a security-updates-only 
release, and it's difficult to mix-and-match patches (e.g. to only get 
security patches).  

This doesn't directly affect users, but fewer OS changes should lead to 
less lengthy / disruptive maintenance and less frequent reboots.  OTOH, 
I don't know if this has a noticeable impact on users at the moment, and 
the previous maintenance was the first for ~170 days.

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (SunOS)

iEYEARECAAYFAk3thyIACgkQIXd7fCuc5vJjsgCfYfbW28r+eW3Lr6L1T+BrzGV/
v4YAn3lkZ7+YfG8ajKBQRx7DBQRKhfnS
=lcFN
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette