Re: [Gluster-users] Gluster 3.4.2 on Ubuntu 12.04 LTS Server - Upstart No Go
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 I get around the problem on our gluster clusters by putting "nobootwait" and/or "noauto" in my fstab and then having rc.local run the mount command for the gluster mount points. Do not know if that is more or less elegant then your workaround. It at least doesn't require one to fight with upstart. On 3/4/14 1:09 PM, Brock Nanson wrote: > I thought I would share my experience in case it helps someone else > avoid a bruised forehead. > > I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes, > replicated) as server and client on both. Going through all the > typical configuration work, all seemed good. The problem came when > I tried to get the boxes to start Gluster and mount the volume at > boot... without user intervention. Thinking that the most > difficult part of the installation should be configuring Gluster > (newbie to distributed file systems in general, never used Gluster > before), I foolishly wasted a pile of time trying to fix the > *simple* boot problem. Well, I did fix it, but only by finally > disabling the upstart files bundled in the install > (glusterfs-server.conf and mounting-glusterfs.conf). > > This was a clean install of Ubuntu 12.04. Actually, two clean > installs for testing. And then another clean install on one box to > rule out the original install! Same behaviour on all installs. > > I spent some time experimenting with different fstab entries and > found I could change the mount behaviour, but only to determine > *when* the mount would fail... it always failed! > > My solution was to disable the two Gluster upstart conf files with > .override files containing 'manual'. Then a simple script to a) > start glusterfs-server; b) sleep for a few seconds; c) mount the > Gluster volume. The script is called from rc.local. > > Hardly an elegant solution I know, but at some point *'works'* > beats *'pretty but non-functional'*. > > It's possible there is something quirky with my install that could > be responsible. But given that other than installing and > configuring Gluster, nothing on the servers was tweaked, I'm > thinking there is a problem with the Gluster upstart conf files. > Or something in 12.04 has rendered them inoperable. Or maybe it's > as a friend in Poland says... (direct translation) "it's the Malice > of Things..." Truly a better description than anything we have in > English! ;-) > > I'm not looking for a solution unless someone has an obvious fix > that Google couldn't locate. I have a quick and dirty work around > that works. I just hope to save someone else the pain. > > > > ___ Gluster-users > mailing list Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > - -- Raymond Powell Systems Administrator Laboratory for Advanced Computing University of Chicago, KCBD 10160 900 East 57th Street Chicago, IL 60637 -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.20 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJTGN1dAAoJEBIjAczqLCW8rU8QAKk86q7zUVZ5r/iJVM5wteGb apfYEeiNlTGrNNZxTPTI9I8nHVVHtCit4PmOjUTWAutLtc8DHWxbLTIE0tsmSRZB Iu6U41DMkl+wBiOFFgNsiaGQPc0gdoe3r8f8kulruA+UNQj5dGSZfkT7+wCeA8mM xZcHCFVKMSRkJ7uLD/TWsRkRjXBYfPw3vmFC4ya1k6Uh9QSRBew8J/nh6oR1pQj4 w3ix6UtbwC7rxW2d2qHDiuNMdNrmqJE0UROWHSlDyMjKsexcuUhGMVJdNFnsR4uS ox2d1IdX2D3bz4wY4zCn/dv9GZkuogZJGwKRnuzYUwqtL3bTWjDHhLAwm9ZpZxr5 h04NiNuxcKQiz4WY716JESK/C1/F7GebymxX1fzoejHinACb8lL5ql7BsVA7vyj4 ANgvu35HPUzHk6sIEpslqqQas+hPokLmWCvkGTmhayWNdp6hG4gUboIsVnvEUpKy Yllnbx5Y1E1f1Fp90FZNHTM+15UDU4hZdSsMBZGLdPkVv0r5zmMQKKCadBuI8w3u yDYBcz9c6k9XPTUsm2viaV6qkgDsS55CJc4QMX5Ic0jZN6ijiHc+FiZS8ZZ/WfdM BzQZEkcmcN39vmEhRx4R/qlFqykhUPhhdRslgb3HxacIEotyFF3s8f8UmM5ty8lX J37OwYbND3kM4IHpKxwb =z/nW -END PGP SIGNATURE- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster 3.4.2 on Ubuntu 12.04 LTS Server - Upstart No Go
Thanks for the thought Matt! I went through a few different fstab lines that I found in various gluster configuration howto's and information pages. I never did use the noatime option though and played more with -netdev. And the lines I used actually all worked if I issued a mount -a command *after* the boot completed. The problem comes down to what I think is two things: First, the gluster daemon loads but doesn't actually start - at least, querying it after the boot always resulted in a 'stop/waiting' result, which to me is 'loaded and ready but not running'. 'Why' could be just about anything and is beyond my abilities but might be related to waiting for the network interface to come up... which I think is what mounting-glusterfs.conf is trying to solve. So maybe that second conf file is part of the problem... dunno! The second issue is that the mount seems to fail if you issue the command right after a 'start' command to gluster. I had to include a few seconds of sleep between the commands to get the result I needed. So it may be that the upstart conf files manage to delay the start of glusterfs-server until the interface is up, but the mount is attempted too soon after, resulting in a fail. But that doesn't explain why gluster is in the stop/wait mode... It's a circular problem one would think, which is why I jumped out of the loop and came up with my own solution... I'm trying to smoke-test gluster, not re-invent upstart (which I have grown to loathe by the way... I hadn't realized init.d was broken! ;-) ) Brock On Tue, Mar 4, 2014 at 11:14 AM, Matt Edwards wrote: > I could be completely off-base, but were you trying to mount with noatime? > > I ran into a similar problem where, after upgrading to 3.4.2, all my > client mounts failed. I discovered (by accident) that it was only due to > incompatible mount options (namely noatime). In previous versions, the > client just gave a warning message about unsupported options, but now it > fails without mentioning the cause. I believe there is a bug report for > this issue (that I discovered afterwards). > > Anyways, I thought this had a chance of being the issue, but maybe it's > something different in your case. > > Matt > > > On Tue, Mar 4, 2014 at 2:09 PM, Brock Nanson wrote: > >> I thought I would share my experience in case it helps someone else >> avoid a bruised forehead. >> >> I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes, >> replicated) as server and client on both. Going through all the typical >> configuration work, all seemed good. The problem came when I tried to get >> the boxes to start Gluster and mount the volume at boot... without user >> intervention. Thinking that the most difficult part of the installation >> should be configuring Gluster (newbie to distributed file systems in >> general, never used Gluster before), I foolishly wasted a pile of time >> trying to fix the *simple* boot problem. Well, I did fix it, but only by >> finally disabling the upstart files bundled in the install >> (glusterfs-server.conf and mounting-glusterfs.conf). >> >> This was a clean install of Ubuntu 12.04. Actually, two clean installs >> for testing. And then another clean install on one box to rule out the >> original install! Same behaviour on all installs. >> >> I spent some time experimenting with different fstab entries and found I >> could change the mount behaviour, but only to determine *when* the mount >> would fail... it always failed! >> >> My solution was to disable the two Gluster upstart conf files with >> .override files containing 'manual'. Then a simple script to a) start >> glusterfs-server; b) sleep for a few seconds; c) mount the Gluster volume. >> The script is called from rc.local. >> >> Hardly an elegant solution I know, but at some point *'works'* beats *'pretty >> but non-functional'*. >> >> It's possible there is something quirky with my install that could be >> responsible. But given that other than installing and configuring Gluster, >> nothing on the servers was tweaked, I'm thinking there is a problem with >> the Gluster upstart conf files. Or something in 12.04 has rendered them >> inoperable. Or maybe it's as a friend in Poland says... (direct >> translation) "it's the Malice of Things..." Truly a better description >> than anything we have in English! ;-) >> >> I'm not looking for a solution unless someone has an obvious fix that >> Google couldn't locate. I have a quick and dirty work around that works. >> I just hope to save someone else the pain. >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> > > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster 3.4.2 on Ubuntu 12.04 LTS Server - Upstart No Go
I could be completely off-base, but were you trying to mount with noatime? I ran into a similar problem where, after upgrading to 3.4.2, all my client mounts failed. I discovered (by accident) that it was only due to incompatible mount options (namely noatime). In previous versions, the client just gave a warning message about unsupported options, but now it fails without mentioning the cause. I believe there is a bug report for this issue (that I discovered afterwards). Anyways, I thought this had a chance of being the issue, but maybe it's something different in your case. Matt On Tue, Mar 4, 2014 at 2:09 PM, Brock Nanson wrote: > I thought I would share my experience in case it helps someone else avoid > a bruised forehead. > > I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes, > replicated) as server and client on both. Going through all the typical > configuration work, all seemed good. The problem came when I tried to get > the boxes to start Gluster and mount the volume at boot... without user > intervention. Thinking that the most difficult part of the installation > should be configuring Gluster (newbie to distributed file systems in > general, never used Gluster before), I foolishly wasted a pile of time > trying to fix the *simple* boot problem. Well, I did fix it, but only by > finally disabling the upstart files bundled in the install > (glusterfs-server.conf and mounting-glusterfs.conf). > > This was a clean install of Ubuntu 12.04. Actually, two clean installs > for testing. And then another clean install on one box to rule out the > original install! Same behaviour on all installs. > > I spent some time experimenting with different fstab entries and found I > could change the mount behaviour, but only to determine *when* the mount > would fail... it always failed! > > My solution was to disable the two Gluster upstart conf files with > .override files containing 'manual'. Then a simple script to a) start > glusterfs-server; b) sleep for a few seconds; c) mount the Gluster volume. > The script is called from rc.local. > > Hardly an elegant solution I know, but at some point *'works'* beats *'pretty > but non-functional'*. > > It's possible there is something quirky with my install that could be > responsible. But given that other than installing and configuring Gluster, > nothing on the servers was tweaked, I'm thinking there is a problem with > the Gluster upstart conf files. Or something in 12.04 has rendered them > inoperable. Or maybe it's as a friend in Poland says... (direct > translation) "it's the Malice of Things..." Truly a better description > than anything we have in English! ;-) > > I'm not looking for a solution unless someone has an obvious fix that > Google couldn't locate. I have a quick and dirty work around that works. > I just hope to save someone else the pain. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster 3.4.2 on Ubuntu 12.04 LTS Server - Upstart No Go
I thought I would share my experience in case it helps someone else avoid a bruised forehead. I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes, replicated) as server and client on both. Going through all the typical configuration work, all seemed good. The problem came when I tried to get the boxes to start Gluster and mount the volume at boot... without user intervention. Thinking that the most difficult part of the installation should be configuring Gluster (newbie to distributed file systems in general, never used Gluster before), I foolishly wasted a pile of time trying to fix the *simple* boot problem. Well, I did fix it, but only by finally disabling the upstart files bundled in the install (glusterfs-server.conf and mounting-glusterfs.conf). This was a clean install of Ubuntu 12.04. Actually, two clean installs for testing. And then another clean install on one box to rule out the original install! Same behaviour on all installs. I spent some time experimenting with different fstab entries and found I could change the mount behaviour, but only to determine *when* the mount would fail... it always failed! My solution was to disable the two Gluster upstart conf files with .override files containing 'manual'. Then a simple script to a) start glusterfs-server; b) sleep for a few seconds; c) mount the Gluster volume. The script is called from rc.local. Hardly an elegant solution I know, but at some point *'works'* beats *'pretty but non-functional'*. It's possible there is something quirky with my install that could be responsible. But given that other than installing and configuring Gluster, nothing on the servers was tweaked, I'm thinking there is a problem with the Gluster upstart conf files. Or something in 12.04 has rendered them inoperable. Or maybe it's as a friend in Poland says... (direct translation) "it's the Malice of Things..." Truly a better description than anything we have in English! ;-) I'm not looking for a solution unless someone has an obvious fix that Google couldn't locate. I have a quick and dirty work around that works. I just hope to save someone else the pain. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
