Re: [Gluster-users] upgrading from gluster-3.2.6 to gluster-3.4.2

[Paul Simpson]

Thanks Tamas for the suggestion.  According to the documentation I've read
the upgrade procedure from 3.2.x to 3.[34] is the same. So there's no point
going from 3.2 -> 3.3, then 3.3 -> 3.4.

Then again, I might be wrong - but trying to get any straight answers
around here seems to be quite difficult.  Hence, why I'm looking to replace
Gluster ASAP...




On 21 February 2014 23:33, Tamas Papp  wrote:

>
> On 02/21/2014 05:32 PM, Paul Simpson wrote:
> > I too would like to know about this.
> >
> > I also tried this process on my 3.2.7 cluster and reported my findings
> > here:
> > http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3/
> >
>
>
> Why don't you try to upgrade to 3.3 first?
>
> tamas
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] upgrading from gluster-3.2.6 to gluster-3.4.2

[Paul Simpson]

I too would like to know about this.

I also tried this process on my 3.2.7 cluster and reported my findings here:
http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3/

No reply as of yet...

Not wanting to sound negative, but I do find there's little support from
the Gluster sites/community on issues such as this. Especially compared to
other projects I've seen. It's not very reassuring for a film-system in
which we're supposed to trust with our precious data.

For this reason - we are looking to migrate away from Gluster after using
it for 3+ years.  It's a shame - and I was a believer (still think it's a
great idea) - but we can't afford to carry on groping around in the dark
(with sparse documentation, obscure error messages & a low-bandwidth
community) any more. :/

Still interested to hear if / how we can upgrade by hand (if need be).
Would certainly help me in the interim - might even change my mind (not
having seen 3.4 in action)..  :)








On 21 February 2014 13:22, Dmitry Kopelevich wrote:

>  I would like to follow up on my question regarding an upgrade from 3.2.6
> to 3.4.2.
> Can anybody tell me whether I'm doing something completely wrong? Am I
> trying to skip too many versions of gluster in my upgrade? Is CentOS 5 too
> old for this?
>
> Thanks,
>
> Dmitry
>
> On 2/18/2014 2:51 PM, Dmitry Kopelevich wrote:
>
> I am attempting to upgrade my GlusterFS from 3.2.6 to 3.4.2 using the
> instructions posted at
> http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3. These
> guidelines are for an upgrade to 3.3 but it is stated at
> http://vbellur.wordpress.com/2013/07/15/upgrading-to-glusterfs-3-4 that
> they can also be used to upgrade to 3.4.0. So I was hoping that they would
> also work with an upgrade to 3.4.2.
>
> I'm running CentOS 5 and installed the following rpms on the gluster
> servers:
>
> glusterfs-libs-3.4.2-1.el5.x86_64.rpm
>  glusterfs-3.4.2-1.el5.x86_64.rpm
>  glusterfs-fuse-3.4.2-1.el5.x86_64.rpm
>  glusterfs-cli-3.4.2-1.el5.x86_64.rpm
>  glusterfs-server-3.4.2-1.el5.x86_64.rpm
>  glusterfs-rdma-3.4.2-1.el5.x86_64.rpm
>  glusterfs-geo-replication-3.4.2-1.el5.x86_64.rpm
>
> According to the installation guidelines, installation from rpms should
> automatically copy the files from /etc/glusterd to /var/lib/glusterd. This
> didn't happen for me -- the directory /var/lib/glusterd contained only
> empty subdirectories. But the content of /etc/glusterd directory has moved
> to /etc/glusterd/glusterd.
>
> So, I decided to manually copy files from /etc/glusterd/glusterd to
> /var/lib/glusterd and follow step 5 of the installation guidelines (which
> was supposed to be skipped when installing from rpms):
>
> glusterd --xlator-option *.upgrade=on -N
>
> This didn't work (error message: glusterd: No match)
>
> Then I tried specifying explicitly the name of my volume:
>
> glusterd --xlator-option .upgrade=on -N
>
> This lead to the following messages in file etc-glusterfs-glusterd.vol.log:
>
> [2014-02-18 17:22:27.146449] I [glusterd.c:961:init] 0-management: Using
> /var/lib/glusterd as working directory
> [2014-02-18 17:22:27.149097] I [socket.c:3480:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2014-02-18 17:22:27.149126] I [socket.c:3495:socket_init]
> 0-socket.management: using system polling thread
> [2014-02-18 17:22:29.282665] I
> [glusterd-store.c:1339:glusterd_restore_op_version] 0-glusterd: retrieved
> op-version: 1
> [2014-02-18 17:22:29.283478] E
> [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-0
> [2014-02-18 17:22:29.283513] E
> [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-1
> [2014-02-18 17:22:29.283534] E
> [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-2
> ...
> and so on for all other bricks.
>
> After that, files nfs.log, glustershd.log, and
> etc-glusterfs-glusterd.vol.log get filled with a large number of warning
> messages and nothing else seems to happen. The following messages appear to
> be relevant:
>
> - Files nfs.log, glustershd.log:
>
> 2014-02-18 15:58:01.889847] W [rdma.c:1079:gf_rdma_cm_event_handler]
> 0-data-volume-client-2: cma event RDMA_CM_EVENT_ADDR_ERROR, error -2 (me:
> peer:)
>
> (the name of my volume is data-volume and its transport type is RDMA)
>
> - File etc-glusterfs-glusterd.vol.log
>
> [2014-02-18 17:22:33.322565] W [socket.c:514:__socket_rwv] 0-management:
> readv failed (No data available)
>
> Also, for some reason the time stamps in the log files are incorrect.
>
> Any suggestions for fixing this would be greatly appreciated.
>
> Thanks,
>
> Dmitry
>
> --
> Dmitry Kopelevich
> Associate Professor
> Chemical Engineering Department
> University of Florida
> Gainesville, FL 32611
>
> Phone:   (352)-392-4422
> Fax: (352)-392-9513
> E-mail:  dkopelev...@che.ufl.edu
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://superc

Re: [Gluster-users] GlusterFS was removed from Fedora EPEL

[Paul Simpson]

Hi Nguyen,

We also have fixed using 3.2.7 for production - it works well for us.

I was thinking to upgrade to 3.4.x in the next month or so. I'd like to
understand more - how many people are actually using 3.4.1? Can anyone
(users) or from gluster/redhat chime in here - would be reassuring to hear
that the latest version is also the greatest..

Regards,

Paul





On 2 December 2013 15:52, Nguyen Viet Cuong  wrote:

> Hi,
>
> Actually, I have very bad experience with GlusterFS 3.3.x and 3.4.x under
> very high pressure (> 64 processes write in parallel in more than 10
> minutes, for example). GlusterFS 3.2.7 from EPEL is really stable and we
> use it for production.
>
> Unfortunately, there is no official built of GlusterFS 3.2.x on Gluster's
> repo.
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] changing volume from Distributed-Replicate to Distributed

[Paul Simpson]

thanks lukas,

that is what i'm planning to test. i guess my underlying question is - will
gluster automatically "know" that a volume of replica count 2 is not
replicated any more? and if so, how will it react to the fact that the
replicated bricks are now missing? ie, should i / or is there a way of
changing the replica count of a volume first? would that automatically
remove the bricks for you?





On 4 November 2013 12:57, lukas.bezdicka wrote:

> Hi, I would say it's quite OK unless you don't shrink the distribute.
>
> For example volume with replica 2 distribute 2 would look like:
> _/test1:/test
>  \test2:/test
> _/test3:/test
>  \test4:/test
>
> if your gluster volume heal test info is empty (eg theres nothing to
> heal) it should be fairly safe to remove test2:/test and test4:/test
> with:
> gluster volume remove-brick test replica 1 test4:/test test2:/test
>
> but I would test it first on your gluster version!
>
> On Mon, 2013-11-04 at 12:45 +, Paul Simpson wrote:
> > ping!
> >
> >
> >
> >
> > On 31 October 2013 11:47, Paul Simpson  wrote:
> > hi all,
> >
> >
> > as the title says - i'm looking to change a volume from
> > dist/repl -> dist.
> >
> >
> > we're currently running 3.2.7.  a few of questions for you
> > gurus out there:
> > - is this possible to do on 3.2.7?
> > - is this possible to do with 3.4.1? (would involve upgrade)
> > - are there any pitfalls i should be aware of?
> >
> >
> > many thanks in advance,
> >
> >
> > regards,
> >
> >
> > paul
> >
> >
> >
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] changing volume from Distributed-Replicate to Distributed

[Paul Simpson]

ping!



On 31 October 2013 11:47, Paul Simpson  wrote:

> hi all,
>
> as the title says - i'm looking to change a volume from dist/repl -> dist.
>
> we're currently running 3.2.7.  a few of questions for you gurus out there:
> - is this possible to do on 3.2.7?
> - is this possible to do with 3.4.1? (would involve upgrade)
> - are there any pitfalls i should be aware of?
>
> many thanks in advance,
>
> regards,
>
> paul
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] changing volume from Distributed-Replicate to Distributed

[Paul Simpson]

hi all,

as the title says - i'm looking to change a volume from dist/repl -> dist.

we're currently running 3.2.7.  a few of questions for you gurus out there:
- is this possible to do on 3.2.7?
- is this possible to do with 3.4.1? (would involve upgrade)
- are there any pitfalls i should be aware of?

many thanks in advance,

regards,

paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] removing brick on 3.2.7

[Paul Simpson]

hi gluster gurus,

is it possible to safely remove a brick using 3.2.7?

if not - any recommendations to what to upgrade to?  should we go for
latest version 3.4.x or a more stable 3.3.x version?

regards,

paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Can I use FSCache/Cachefiles with Gluster Native Client mode?

[Paul Simpson]

i'd also be interested in hearing if this is possible.  i tried -
unsuccessfully to do this six months ago..

-p



On 3 December 2012 03:29, 吴巍  wrote:

> Hey,
>
> I Download the GlusterFS 3.2.7 Release and it works well. And I want to
> improve the reading performance with FSCache/Cachefiles model.
> I use the command line:
> mount -t glusterfs server1:/test-volume /mnt/glusterfs -o fsc
> and no error reported. But the local cache function of FSCache is not
> active. How can I use the local cache function of FSCache with Gluster
> Native Client  mount mode?
>
> Thanks.
>
>
> --
> Wei Wu
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] worth upgrading from 3.2.7 to 3.3.1?

[Paul Simpson]

gently bumping my own post here.

maybe i should put this another way - have many of you upgraded to 3.3.1?
 ie, anyone out there using 3.3.1 successfully?




On 26 November 2012 19:31, Paul Simpson  wrote:

> hi gluster gurus,
>
> we've been using 3.2.7 for a year now - and it's been quite stable.  so
> far so good.  however, gluster has moved on quite a bit and maybe it's time
> to upgrade?  so, a couple of questions if you don't mind:
> 1/ what are the main benefits to upgrading?  i'm particularly interested
> in hearing about stability & speed..
> 2/ do people keep the automatic self-heal switched on?  (i ask as i
> remember someone here turning it off for some reason)
> 3/ anything else i should ask?  ;)
>
> would most appreciate any responses both positive or negative (as long
> as constructive).
>
> regards to all,
>
> paul
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] worth upgrading from 3.2.7 to 3.3.1?

[Paul Simpson]

hi gluster gurus,

we've been using 3.2.7 for a year now - and it's been quite stable.  so far
so good.  however, gluster has moved on quite a bit and maybe it's time to
upgrade?  so, a couple of questions if you don't mind:
1/ what are the main benefits to upgrading?  i'm particularly interested in
hearing about stability & speed..
2/ do people keep the automatic self-heal switched on?  (i ask as i
remember someone here turning it off for some reason)
3/ anything else i should ask?  ;)

would most appreciate any responses both positive or negative (as long
as constructive).

regards to all,

paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] NASA uses gluster..

[Paul Simpson]

thought this might be of interest to you all out there:

http://opensource.com/life/12/10/NASA-achieves-data-goals-Mars-rover-open-source-software
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Frequent glusterd restarts needed to avoid NFS performance degradation

[Paul Simpson]

just like to add that we sometimes need to restart glusterd on servers too.
 again - on a renderfarm that hammers our 4 server dist/repl servers
heavily.

-p


On 23 April 2012 15:38, Brian Cipriano  wrote:

> Hi Dan - I've seen this problem too. I agree with everything you've
> described - seems to happen more quickly on more heavily used volumes, and
> a restart fixes it right away. I've also been considering writing a cronjob
> to fix this - have you made any progress on this, anything to report?
>
> I'm running a fairly simple distributed, non-replicated volume across two
> servers. What sort of tasks are you using your gluster for? Ours is for a
> render farm, so we see a very large number of mounts/unmounts as render
> nodes mount various parts of the filesystem. I wonder if this has anything
> to do with it; is your use case anything similar?
>
> - brian
>
>
> On 4/17/12 7:30 PM, Dan Bretherton wrote:
>
>> Dear All-
>> I find that I have to restart glusterd every few days on my servers to
>> stop NFS performance from becoming unbearably slow.  When the problem
>> occurs, volumes can take several minutes to mount and there are long delays
>> responding to "ls".   Mounting from a different server, i.e. one not
>> normally used for NFS export, results in normal NFS access speeds.  This
>> doesn't seem to have anything to do with load because it happens whether or
>> not there is anything running on the compute servers.  Even when the system
>> is mostly idle there are often a lot of glusterfsd processes running, and
>> on several of the servers I looked at this evening there is a process
>> called glusterfs using 100% of one CPU.  I can't find anything unusual in
>> nfs.log or etc-glusterfs-glusterd.vol.log on the servers affected.
>>  Restarting glusterd seems to stop this strange behaviour and make NFS
>> access run smoothly again, but this usually only lasts for a day or two.
>>
>> This behaviour is not necessarily related to the length of time since
>> glusterd was started, but has more to do with the amount of work the
>> GlusterFS processes on each server have to do.  I use a different server to
>> export each of my 8 different volumes, and the NFS performance degradation
>> seems to affect the most heavily used volumes more than the others.  I
>> really need to find a solution to this problem; all I can think of doing is
>> setting up a cron job on each server to restart glusterd every day, but I
>> am worried about what side effects that might have.  I am using GlusterFS
>> version 3.2.5.  All suggestions would be much appreciated.
>>
>> Regards,
>> Dan.
>> __**_
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://gluster.org/cgi-bin/**mailman/listinfo/gluster-users
>>
>
> __**_
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/**mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Estimated date for release of Gluster 3.3

[Paul Simpson]

>
> Also, I'll add this - Red Hat's QA efforts in the process has been
> invaluable. Yes, it's made the release process slower, but the benefits of
> a more stable 3.3 tree will be very noticeable.


music to my ears!

-paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Usage Case: just not getting the performance I was hoping for

[Paul Simpson]

same - XFS works v well for us too.  maybe this is just a stripe issue?


On 15 March 2012 12:49, Fabricio  wrote:

> Em 15-03-2012 09:24, Sabuj Pattanayek escreveu:
>
>  Striped volumes are unfortunately broken on top of XFS at the moment:
>>> http://oss.sgi.com/archives/**xfs/2012-03/msg00161.html
>>>
>>
>> Yea, I found this out after copying several million files into a
>> stripe that XFS doesn't report the size of sparse files correctly.
>> Lost a week of time on the copy after which I switched all my bricks
>> to ext4. XFS, never again!
>>
>
> XFS saved my ass more than once when loss was certain, Sabuj. I would not
> write it off so fast.
>
> --
> Certified Shoelace Untier - Level 3
>
> __**_
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/**mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS and cachefilesd

[Paul Simpson]

i tried this - but it didn't work compared to a "normal" NFS mount.  keen
to hear any other thoughts...


On 10 February 2012 12:04, rickytato rickytato wrote:

> It's possible and make sense use nfs mount and cachefilesd for GlusterFS?
> Actually I use native fuse client to mount my simple cluster, but I'd like
> to know if
> this setup can work well.
>
>
> rr
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] brick logs filling up with errors (3.2.5)

[Paul Simpson]

just pinging again - this issue hasn't gone away - and i'm keen to get this
resolved...



On 18 January 2012 14:55, Paul Simpson  wrote:

> hi all,
>
> i'm using 2x2 servers g1,g2,g3,g4 all running (opensuse 11.3) running
> 3.2.5.  ie, dist-replicated.
>
> my logs are filling up with errors such as:
>
> [2012-01-18 14:49:00.317604] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11016950: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
> [2012-01-18 14:49:00.420118] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11016964: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
> [2012-01-18 14:49:00.561886] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11016973: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
> [2012-01-18 14:49:00.583313] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11016982: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
> [2012-01-18 14:49:00.926677] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11017010: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
> [2012-01-18 14:49:00.939979] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11017019: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
> [2012-01-18 14:49:01.399885] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11017070: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
> [2012-01-18 14:49:01.492153] I [server3_1-fops.c:1417:server_create_cbk]
> 0-glustervol1-server: 11017101: CREATE
> /production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
> -1 (Permission denied)
>
>
> and:
>
> [2012-01-18 14:49:49.224118] W [inode.c:1044:inode_path]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
> [0x7ffe19019bd8]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
> [0x7ffe19019ad0]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
> [0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
> -2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
> [2012-01-18 14:49:49.264120] W [inode.c:1044:inode_path]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
> [0x7ffe19019bd8]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
> [0x7ffe19019ad0]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
> [0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
> -2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
> [2012-01-18 14:49:49.283681] W [inode.c:1044:inode_path]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
> [0x7ffe19019bd8]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
> [0x7ffe19019ad0]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
> [0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
> -2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
> [2012-01-18 14:49:49.303780] W [inode.c:1044:inode_path]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
> [0x7ffe19019bd8]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
> [0x7ffe19019ad0]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
> [0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
> -2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
> [2012-01-18 14:49:49.323806] W [inode.c:1044:inode_path]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
> [0x7ffe19019bd8]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
> [0x7ffe19019ad0]
> (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
> [0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
> -2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
> [2012

[Gluster-users] brick logs filling up with errors (3.2.5)

[Paul Simpson]

hi all,

i'm using 2x2 servers g1,g2,g3,g4 all running (opensuse 11.3) running
3.2.5.  ie, dist-replicated.

my logs are filling up with errors such as:

[2012-01-18 14:49:00.317604] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11016950: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)
[2012-01-18 14:49:00.420118] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11016964: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)
[2012-01-18 14:49:00.561886] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11016973: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)
[2012-01-18 14:49:00.583313] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11016982: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)
[2012-01-18 14:49:00.926677] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11017010: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)
[2012-01-18 14:49:00.939979] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11017019: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)
[2012-01-18 14:49:01.399885] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11017070: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)
[2012-01-18 14:49:01.492153] I [server3_1-fops.c:1417:server_create_cbk]
0-glustervol1-server: 11017101: CREATE
/production/hyundai/shop/houdini/scripts/python/gridSplitSeed.pyc (0) ==>
-1 (Permission denied)


and:

[2012-01-18 14:49:49.224118] W [inode.c:1044:inode_path]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
[0x7ffe19019bd8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
[0x7ffe19019ad0]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
[0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
-2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
[2012-01-18 14:49:49.264120] W [inode.c:1044:inode_path]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
[0x7ffe19019bd8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
[0x7ffe19019ad0]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
[0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
-2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
[2012-01-18 14:49:49.283681] W [inode.c:1044:inode_path]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
[0x7ffe19019bd8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
[0x7ffe19019ad0]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
[0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
-2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
[2012-01-18 14:49:49.303780] W [inode.c:1044:inode_path]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
[0x7ffe19019bd8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
[0x7ffe19019ad0]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
[0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
-2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
[2012-01-18 14:49:49.323806] W [inode.c:1044:inode_path]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8)
[0x7ffe19019bd8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70)
[0x7ffe19019ad0]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105)
[0x7ffe19019645]))) 0-/mnt/glus1/inode: no dentry for non-root inode
-2547827547: b185c544-77b8-4db3-babf-0cd1ad642fd4
[2012-01-18 14:51:28.398639] I [server3_1-fops.c:964:server_unlink_cbk]
0-glustervol1-server: 4905904: UNLINK
/production/lib/python2.6/site-packages/simplejson/__init__.pyc
(-746354762) ==> -1 (Permission denied)


...this doesn't look good.  i have ~35 linux clients (opensuse 11.3) which
access the servers using both FUSE and NFS.

i'm concerned about the data integrity on the system.  also, we get i/o
errors every now and then plus statting errors such as:  ls -l produces
 ??? ??? a lot.

i would be most appreciative with any help.

regards,

paul
___
Gluster-users mailing list
Gluster-u

Re: [Gluster-users] fuse mount disconnecting...

[paul simpson]

hi jan,


>...should i use the patched one - or have the patches been rolled into the
> >upstream/main version?  i notice that the latest fuse
> >on http://download.gluster.com/pub/gluster/glusterfs/fuse/ is 2.7 - while
> my
> >distro is now upto 2.8.5.
>
> The glfs patches have not made it into the kernel (determined by looking
> at the source), but that may have other reasons:
>
> http://www.spinics.net/lists/linux-nfs/msg22731.html
> http://www.spinics.net/lists/linux-nfs/msg22867.html


thats interesting.  is there an official response to this from gluster?  it
would be great to have this issue resolved - as patching only complicates
installs.



> OBS also has glusterfs-3.2.4 in /filesystems.
>

i'm not sure what this sentence means - good or bad! :)

-p

ps - i was reticent to upgrade to 3.2 and am currently on the 3.1.6 branch.
 i'd be keen to hear if 3.2.x is considered more stable than 3.1.6.  if so,
i'll be happy to upgrade.  also, be good to hear the official word on this.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] fuse mount disconnecting...

[paul simpson]

hi,

i am getting regular crashes which result in the mount being dropped:

n1:~ # ls /n/auto/gv1/
ls: cannot access /n/auto/gv1/: Transport endpoint is not connected

client side error log:  http://pastebin.com/UgMaLq42

..i am also finding that the gluster severs also sometimes just drop out -
and i need to kill all the server side gluster processes and restart
glusterd.  i'm not sure if this is related.  btw: n1 is a virtual machine
with 4G RAM - so i'm not sure what the memory alloc errs relate to.

server config file:  http://pastebin.com/fHVECyxV

running on opensuse 11.3 both client and server.

i am running default opensuse fuse:
n1:~ # rpm -qa | grep fuse
glusterfs-fuse-3.1.6-1.x86_64
gvfs-fuse-1.6.1-2.7.x86_64
libfuse2-2.8.5-2.7.1.x86_64
fuse-2.8.5-2.7.1.x86_64

...should i use the patched one - or have the patches been rolled into the
upstream/main version?  i notice that the latest fuse on
http://download.gluster.com/pub/gluster/glusterfs/fuse/ is 2.7 - while my
distro is now upto 2.8.5.

these issues have become increasingly common and are now stopping my
overnight rsnapshot/rsync backups - so getting a little unsettling.

many thanks in advance,

-paul

ps - congratulations for the redhat deal.  amazing!
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ZFS + Linux + Glusterfs for a production ready 100+ TB NAS on cloud

[paul simpson]

hi joe,

many thanks for your insights here - much food for thought.

regards,

paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ZFS + Linux + Glusterfs for a production ready 100+ TB NAS on cloud

[paul simpson]

hi david,

thanks for the pointer.  already reading the readme on the repo.  it looks
interesting - and i'd be keen to hear from any gluster guru's what their
thoughts are on such a setup...

regards,

-paul


On 29 September 2011 18:44, David Miller  wrote:

> On Thu, Sep 29, 2011 at 1:32 PM, David Miller  wrote:
>
>> Couldn't you  accomplish the same thing with flashcache?
>> https://github.com/facebook/flashcache/
>>
>
> I should expand on that a little bit.  Flashcache is a kernel module
> created by Facebook that uses the device mapper interface in Linux to
> provide a ssd cache layer to any block device.
>
> What I think would be interesting is using flashcache with a pcie ssd as
> the caching device.  That would add about $500-$600 to the cost of each
> brick node but should be able to buffer the active IO from the spinning
> media pretty well.
>
> Somthing like this.
> http://www.amazon.com/OCZ-Technology-Drive-240GB-Express/dp/B0058RECUE or
> something from FusionIO if you want something that's aimed more at the
> enterprise.
>  --
> David
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ZFS + Linux + Glusterfs for a production ready 100+ TB NAS on cloud

[paul simpson]

hey joe,

recived the legal issues loud and clear - all very good points.  hope these
issues will become clarified in due course.

putting the legal issues aside - still v keen to hear your and others
thoughts about ZFS&L2ARC being a good platform for glusterfs.  that fast SSD
tier sounds like a perfect compliment to glusters slow small file
performance.

regards,

paul



On 29 September 2011 17:48, Joe Landman wrote:

> On 09/29/2011 12:38 PM, paul simpson wrote:
>
>> been reading this thread - quite fascinating.
>>
>> zfsonlinux + gluster looks like an intriguing combination.  i'm
>> interested in your findings to date; specifically would the zfs L2ARC
>> (with SSDs) speed up underlying gluster operations?  it sounds like it
>> could be a potent mix.
>>
>
> Just don't minimize the legal risk issue.  Its very hard for a vendor to
> ship/support this due to the potential risk.  Its arguably hard for a user
> to deploy zfs on linux due to the risk, unless they had a way to argue that
> they are not violating licensing (can't intermix GPL and CDDL and
> ship/support it) for commercial purposes.
>
> Lots of folks can't claim the type of cover that a national lab can claim
> (researching storage models).  You have to decide if the risk is worth it.
>
> If you were to do this, I'd suggest going the Illumos/OpenIndiana or BSD
> route.  Yeah, work still needs to be done to get Gluster to build there, but
> the licensing is on firmer ground (hard to claim that an "open source"
> license such as CDDL does not mean what it says).
>
> Understand where you stand first.  Speak to a lawyer type first.  Make sure
> you won't have issues.
>
> And do remember, that while Oracle and Netapp have (for the moment)
> de-escalated hostilities, Oracle did not provide indemnity to non-Oracle
> customers.  So Netapp (and others) *can* resume their actions.  A question
> was asked why not go after Nexenta versus others.  Simple. There are many
> others (e.g. more potential licensing/legal fees) as compared to a single
> Nexenta.  Its arguably less about rights as it is revenue from legal action.
>  But that stuff does happen ...
>
> Oracle is probably the only one whom can ship ZFS anything safely.  And,
> I'd guess that they are perfectly happy with that situation.
>
>
>> regards,
>>
>> -paul
>>
>
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman@scalableinformatics.**com 
> web  : http://scalableinformatics.com
>   
> http://scalableinformatics.**com/sicluster<http://scalableinformatics.com/sicluster>
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ZFS + Linux + Glusterfs for a production ready 100+ TB NAS on cloud

[paul simpson]

been reading this thread - quite fascinating.

zfsonlinux + gluster looks like an intriguing combination.  i'm interested
in your findings to date; specifically would the zfs L2ARC (with SSDs) speed
up underlying gluster operations?  it sounds like it could be a potent mix.

regards,

-paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] help! - cant bring 1 of 4 servers up..

[paul simpson]

pranith - a huge and heartfelt thanks for your super prompt attention.  a
very scary event turned into a non-event.  :)

regards,

-p


On 8 August 2011 18:35, Pranith Kumar K  wrote:

> **
> After debugging the problem with paul on IRC, we found that because his
> disk had no free space, the subsequent writes on one of the peer files (used
> for recovering run-time information) failed so the file became empty.
> Because of this glusterd could not restore that peer so it is not
> re-starting successfully. We copied the contents of that file from other
> peer in the cluster to the problematic one. Then glusterd started
> successfully.
>
> Pranith.
>
>
> On 08/08/2011 10:32 PM, paul simpson wrote:
>
> hi pranith,
>
>  many thanks for the super quick reply!  i've attached the files asked for
> - be keen to hear your thoughts.  i'm stumped - and scared!
>
>  regards,
>
>  paul
>
>
>
>
> On 8 August 2011 17:59, Pranith Kumar K  wrote:
>
>>  zip /etc/glusterd and send across
>>
>> Pranith
>>
>> On 08/08/2011 10:15 PM, paul simpson wrote:
>>
>>  hi gluster gurus,
>>
>>  i have 4 servers g1,g2,g3 & g4 with 24T each running gluster 3.1.5 on
>> opensuse 11.3.  they have been running well for the last few months in a
>> distributed+replicated setup.
>>
>>  i just found that the nfs log had filled up my root disk of g4 (my bad).
>>  so, i removed the log file - and a couple of other large ones and restored
>> a load of disk space.  however, gluster 3.1.5 will not restart on this
>> machine!!  it err's out with http://pastebin.com/646W8zjg
>>
>>  i've searched this forum, and searched the documentation.  however, i
>> cant see anything that mentions this situation.  please can anyone help -
>> i'm quite concerned about my system.  this is a live server with live data.
>>  i need to get g4 up and running and back into sync ASAP.
>>
>>  many thanks in advance,
>>
>>  -paul
>>
>>  ps - the following command just hangs:
>>
>> g4:~ # gluster peer status
>>
>>
>>  ..however, on g3 it works:
>>
>>  g3:/etc/glusterd/logs # gluster peer status
>>  Number of Peers: 3
>>
>>   Hostname: 10.0.0.12
>>  Uuid: 8061196e-a075-42f6-89f5-1f60281485f5
>>  State: Peer in Cluster (Connected)
>>
>>   Hostname: g2
>>  Uuid: 154d5c46-f62f-4e9c-a328-443e30cadf4e
>>  State: Peer in Cluster (Connected)
>>
>>   Hostname: g4
>>  Uuid: 62365589-61f8-479f-bb50-11519beba045
>>  State: Peer in Cluster (Disconnected)
>>
>> ..i've also tried rebooting the machine - and nothing changes.
>>
>>
>> ___
>> Gluster-users mailing 
>> listGluster-users@gluster.orghttp://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] help! - cant bring 1 of 4 servers up..

[paul simpson]

hi gluster gurus,

i have 4 servers g1,g2,g3 & g4 with 24T each running gluster 3.1.5 on
opensuse 11.3.  they have been running well for the last few months in a
distributed+replicated setup.

i just found that the nfs log had filled up my root disk of g4 (my bad).
 so, i removed the log file - and a couple of other large ones and restored
a load of disk space.  however, gluster 3.1.5 will not restart on this
machine!!  it err's out with http://pastebin.com/646W8zjg

i've searched this forum, and searched the documentation.  however, i cant
see anything that mentions this situation.  please can anyone help - i'm
quite concerned about my system.  this is a live server with live data.  i
need to get g4 up and running and back into sync ASAP.

many thanks in advance,

-paul

ps - the following command just hangs:

g4:~ # gluster peer status


..however, on g3 it works:

g3:/etc/glusterd/logs # gluster peer status
Number of Peers: 3

Hostname: 10.0.0.12
Uuid: 8061196e-a075-42f6-89f5-1f60281485f5
State: Peer in Cluster (Connected)

Hostname: g2
Uuid: 154d5c46-f62f-4e9c-a328-443e30cadf4e
State: Peer in Cluster (Connected)

Hostname: g4
Uuid: 62365589-61f8-479f-bb50-11519beba045
State: Peer in Cluster (Disconnected)

..i've also tried rebooting the machine - and nothing changes.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Somebody build glusterfs 3.2 with macosx sucessfully

[paul simpson]

Also very keen to hear the answer to this...
On Aug 8, 2011 7:43 AM, "Uwe Kastens"  wrote:
> Hi John,
>
> So, ist macos X supported as native glusterfs client?
> Yes or No?
>
> Kind Regards
>
> Uwe
>
>
> 2011/8/5 John Mark Walker 
>
>> Hi Luis,
>>
>> Thanks for the clarification. I wasn't referring to the client, but I
>> should have been more specific - I misunderstood the context of the
thread.
>>
>> Yes, on the glusterfs client side, we have multiple users and customers
>> happily using Mac OS X.
>>
>> -JM
>>
>>
>> --
>> *From:* Luis Cerezo [l...@luiscerezo.org]
>> *Sent:* Friday, August 05, 2011 1:09 PM
>> *To:* John Mark Walker
>> *Cc:* Uwe Kastens; gluster-users@gluster.org
>>
>> *Subject:* Re: [Gluster-users] Somebody build glusterfs 3.2 with macosx
>> sucessfully
>>
>> paying customer here and our users will use a mac gluster client.
>>
>>
>> -luis
>> On Aug 5, 2011, at 3:05 PM, John Mark Walker wrote:
>>
>> The "official" word is that we don't support OSX. However, we would
>> certainly welcome a community effort to help out, if enough people are
>> interested.
>>
>> -JM
>>
>>
>> 
>> Portal: http://www.gluster.org/
>> Get help: http://community.gluster.org/
>> Report bugs: http://bugs.gluster.com/
>>
>>
>>
>> --
>> *From:* gluster-users-boun...@gluster.org [
>> gluster-users-boun...@gluster.org] on behalf of Uwe Kastens [
>> kiste...@googlemail.com]
>> *Sent:* Friday, August 05, 2011 12:55 PM
>> *To:* gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Somebody build glusterfs 3.2 with macosx
>> sucessfully
>>
>> Looks like there is an fix on github available
>>
>>
https://github.com/gluster/glusterfs/commit/e719518e561033af0ffd07d0c8ad7d15fe89#xlators/features/marker/src/marker.h
>>
>>
>> 2011/8/5 paul simpson 
>>
>>> any official word here? native gluster on the mac would be great...
>>>
>>>
>>>
>>> On 5 August 2011 16:55, Uwe Kastens  wrote:
>>>
>>>> Hi,
>>>>
>>>> there is a open bug which seems to prevent one from compiling on macosx
>>>> Darwin und Lion.
>>>>
>>>> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=3292
>>>>
>>>>
>>>>
>>>> 2011/8/5 Amar Tumballi 
>>>>
>>>>> xcode components. I had worked on a Portfile during 2.0.x time to
create
>>>>> a .dmg package. At present we are not working on Mac, and hence not
tested
>>>>> in long time.
>>>>>
>>>>> On Fri, Aug 5, 2011 at 3:32 PM, Uwe Kastens 
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Are you using the xcode components (gcc, make) or ports?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Uwe
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ___
>>>> Gluster-users mailing list
>>>> Gluster-users@gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>
>>>>
>>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>>
>> Luis E. Cerezo
>>
>> http://www.luiscerezo.org
>> http://twitter.com/luiscerezo
>> http://flickr.com/photos/luiscerezo
>> photos for sale:
>> http://photos.luiscerezo.org
>> Voice: 412 223 7396
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] GlusterFS 3.1.5 now available

[paul simpson]

hi john,

many thanks for the heads up on this new version.

just for everyone's edification (i'm always curious if/when people
upgrade) i've just upgraded our 4 servers from 3.1.3 to 3.1.5.  so, if you
dont hear from me - then you can assume everything went well.

regards,

paul


On 21 June 2011 23:31, John Mark Walker  wrote:

> If you haven't seen it already, GlusterFS 3.1.5 is now available at
> http://www.gluster.org/download/
>
> For those of you currently on the 3.1.x series, we recommend that you
> upgrade to this latest release.
>
> Here are some issues fixed in this release:
>
> Bug 2294: Fixed the issue occurred during creating and sharing of volumes
> with both RDMA and TCP/IP transport type.
> Bug 2522: Fixed the issue of invalid argument while removing a file. This
> was caused by gfid mismatch, which happens during distribute rename.
> Bug 2553: Fixed the issue occurred during NFS file creation with Mac
> client.
> Bug 2904: Fixed the issues occurred during upgrading from 3.1.x version to
> 3.2.x.
> Bug 2949: Fixed the issue of glusterfs hanging when self-heal command is
> issued on two glusterfs clients.
> Bug 2941: Fixed the issue of glusterd breaking down when starting hundreds
> of volumes.
> Bug 2870: Fixed the issue of inconsistent xattr values when creating
> bricks.
> Bug 2994: Fixed the issue of glusterfs hanging when untar and rm commands
> are run in parallel.
>
> The yum repository has been updated to reflect the 3.1.5 release.
>
> -JM
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: Re: Inputs requested on 3.2.0

[paul simpson]

hi vidya,

thanks very much for this very comprehensive summary.  it's most reassuring
to read - this is exactly what the community needs.  looking forward to
further reports and releases...

regards,

paul


On 15 June 2011 14:57, Vidya Sakar  wrote:

>  Hi all,
>
> A couple of weeks back I had sought inputs from you all on 3.2.0 and I had
> committed to revert back with an action plan to address all
> concerns/suggestions.
>
> As you would have noticed, 3.2.1 was released a few days ago - already
> addressing some of your concerns. Please keep the feedback coming, inputs
> from the community are very important to us. Please find below responses to
> each of your concerns / suggestions :
>  *
> **1) Stability : *(Paul simpson, James Burnash, Mohit Anchlia)
>
> This is of utmost importance to us. We are investing more in QA engineers,
> QA lead and also in engineers who will dedicatedly work only on bug fixes.
> We are refining our development processes focusing more on quality and
> stability. We are increasing the test coverage and automating the tests
> along with more investments in lab infrastructure.
>
> *2) Better way to handle errors and failures, better troubleshooting tools
> : *(James Burnash, Whit Blauvelt)
>
> The scale of the number of components and systems involved increases the
> need for better troubleshooting tools and better handling of errors and
> failures. The recently introduced top/profile is a step in this direction to
> build diagnosability in to the product. There was also an effort on
> improving log messages in all modules – to make them more meaningful,
> prioritizing them appropriately, introducing new logs where necessary and
> things like that. The recently completed enhancements around logging in
> 3.2.0 is only the first of a multi-phase effort to improve logging and make
> the user experience much better.
>
> *3) **Better documentation :* (Udo Waechter, Christian Kauhaus)
>
>  We are revamping the documentation and we should soon have more content
> in existing sections and we will also have new content – things like
> troubleshooting section, performance tuning section, best practices section
> are all in the pipeline. The format of documentation is also due for a
> change shortly.
>
> *4) **P**osix compatibility :* (Paul simpson, Udo Waechter)
>
> We are fully committed to POSIX compatibility. Scenarios where it is not
> compatible if any would be high priority bugs for us.
>
> *5) **Better self-heal / reporting on which files are not replicated :*(Paul 
> simpson, Whit Blauvelt)
>
> This is already in our roadmap for 3.3. It is actively being worked on and
> we can soon expect improvements in this area.
>
> *6) **NFS nlm locking :* (Paul simpson)
>
> We do realize the need for NLM, I will get back to you all shortly on what
> our plan on NLM is going to be.
>
> *7) **Ability to list config settings **(including **default**)** /
> Performance settings and benchmarks : *(Mike Hanby, Udo Waechter)
>
> The list of defaults is given in this blog post :
> http://sacurs.blogspot.com/2011/05/glusterfs-defaults.html
>
>  And we also have a bug for this :
> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2041
> *
> 8) Bugs : *(Mohit Anchlia, Udo Waechter)
>
> 1) gfid mismatch issue : Is this bug 2522? Please let me know if you are
> referring to something else
>
> 2) 0 byte file size : Bug 2922, we are working on it
>
> 3) Multiple files of same name showing up on the clent side : This problem
> too will get addressed as part of fixing 2522.
>
> 4) client hangs and crash : There are some related bugs which we fixed in
> 3.2.1, if this issue still persists, we are committed to fix the remaining
> issues.
>
> 5) xattr mismatch on gluster mount points causing split brain errors : This
> issue too is fixed in 3.2.1.
>
> 6) non-root owned files not shown : Bug 3030 has been filed to track this.
>
> Thanks,
> VS
>
> --
> Vidya Sakar N
> Senior Engineering Manager
> Gluster Software India (P) Ltd
> Bangalore
>
>
>  Original Message   Subject: Re: [Gluster-users] Inputs
> requested on 3.2.0  Date: Thu, 26 May 2011 20:38:30 +0530  From: Vidya
> SakarTo: gluster-users@gluster.org
>
> Thanks to everyone for all the inputs. I will get back with responses to
> each of your suggestions shortly.
>
> Thanks,
> VS
>
> --
> Vidya Sakar N
> Senior Engineering Manager
> Gluster Software India (P) Ltd
> Bangalore
>
>
> On Wednesday 25 May 2011 10:54 PM, Mohit Anchlia wrote:
> > Hi Vidya,
> >
> > Thanks for asking. I think it will be worthwhile to look at the
> > archives and mailing list to see what other peo

Re: [Gluster-users] warning: pure path resolution

[paul simpson]

many thanks all!  it's reassuring to know.  looking forward to 3.2.2 - as
right now i'm scared to upgrade from 3.1.3 which seems to be working pretty
smoothly within a defined range.

out of interest - is full posix compatibility (ie, proper nlm locking) on
the immediate todo list?


On 15 June 2011 15:29, Joe Landman  wrote:

> On 06/15/2011 10:23 AM, Whit Blauvelt wrote:
>
>> On Wed, Jun 15, 2011 at 10:16:02AM -0400, Joe Landman wrote:
>>
>>  As a general rule, the W simply tells you its a warning.
>>>
>>
>> In practice, is it safe to simply ignore all warnings from Gluster?
>> Different projects have different thresholds between no warning, warning,
>> and critical messages. Is Gluster's, in your experience, such that
>> warnings
>> may as well be discarded?
>>
>> Whit
>>
>
>
> Ok ... Warnings shouldn't be ignored, just logged. In most cases, they will
> turn out to be nothing.  In some cases, they may turn out to be something.
>  I'll defer to the devs, but our experience suggests that warnings that
> don't develop into E states are things you don't have to worry about.
>
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: land...@scalableinformatics.com
> web  : http://scalableinformatics.com
>   http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] warning: pure path resolution

[paul simpson]

hi,

can anyone please answer this?  it's hard to contribute back to the
community when there's a deathly silence..

regards,

paul



On 10 June 2011 17:38, paul simpson  wrote:

> hi,
>
> i'm seeing this warning a *lot* in my logs.  this is on 3.1.3 running
> dist-repl on 4 servers.  ie,
>
> [2011-06-10 17:06:08.326245] W [server-resolve.c:565:server_resolve]
> 0-glustervol1-server: pure path resolution for /production/seed/env/boot
> (OPEN)
> [2011-06-10 17:06:08.327092] W [server-resolve.c:565:server_resolve]
> 0-glustervol1-server: pure path resolution for /production/seed/env/boot
> (INODELK)
>
> more can be found here:  http://pastebin.com/hWRWU6cv
>
> ..i cant seem to find any mention of this message in the docs.  can anyone
> please tell me if or how much i should be concerned?  and is there anything
> i need to do to fix things??
>
>  many thanks in advance,
>
> paul
>
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] warning: pure path resolution

[paul simpson]

hi,

i'm seeing this warning a *lot* in my logs.  this is on 3.1.3 running
dist-repl on 4 servers.  ie,

[2011-06-10 17:06:08.326245] W [server-resolve.c:565:server_resolve]
0-glustervol1-server: pure path resolution for /production/seed/env/boot
(OPEN)
[2011-06-10 17:06:08.327092] W [server-resolve.c:565:server_resolve]
0-glustervol1-server: pure path resolution for /production/seed/env/boot
(INODELK)

more can be found here:  http://pastebin.com/hWRWU6cv

..i cant seem to find any mention of this message in the docs.  can anyone
please tell me if or how much i should be concerned?  and is there anything
i need to do to fix things??

many thanks in advance,

paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ESXi & Gluster setup options

[paul simpson]

Thanks for this. Is their any other advantage our disadvantage in using the
fuse client? Speed, reliability? In fact, has the anyone run vm's off fuse?
On May 26, 2011 11:56 PM, "Chris Haumesser"  wrote:
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] ESXi & Gluster setup options

[paul simpson]

i'm also interested in this.  is there any pro/con to using native gluster
FUSE client for xen images?  i would have thought that would mitigate the
use of ucarp (apart from initial connect).

-p



On 26 May 2011 21:39, Whit Blauvelt  wrote:

> On Thu, May 26, 2011 at 08:28:29PM +, Matt Temple wrote:
>
> > We've been looking for a way to have HA for our VMWare datastore, too.
> (Our
> > single server had a kernel panic last night and took down the VMs.)
> > We're very much interested in a similar setup, using Gluster, but I have
> a
> > question ... with Gluster NFS, don't you have to choose a specific
> address of a
> > server to connect to?   And if yes, if that node goes down, how does
> VMWare
> > respond?
>
> Pretty much the standard thing to do is use a virtual IP address with ucarp
> handling reassignment if the primary node goes down. I've yet to run that
> through thorough tests to see if how transparent it is in a case of
> failure.
> It's simple to set up though.
>
> There are, of course, plenty of more complicated alternatives to ucarp in
> the HA world - heartbeat, pacemaker, corosync
>
> Whit
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Inputs requested on 3.2.0

[paul simpson]

hi vidya,

the main general points for me are:
1/ documentation.  if gluster is not 100% posix compatible then please
*dont* say that it is as it only undermines the credibility of gluster as a
file-system.  if we know what does and doesn't work then we can make
informed decisions and work around any issues.
2/ better reporting.  ie, we should be able to know which files are not
replicated.  right now, there's no way knowing how safe any of your data is.
 i'd be interested if this is even possible with the very decentralised way
gluster works?
3/ NFS nlm locking & other posix compatibility issues solved.
4/ stability stability and more stability.  please, no more "features" till
the foundations are rock solid.  i want gluster to work - but it's
credibility as a "safe" file system is at issue here.  what could be more
important than that?

..i hope this is of use.

regards,

paul



On 24 May 2011 11:52, Vidya Sakar  wrote:

> Hello all,
>
> We are planing a 3.2.1 release and we would like to hear from you all on
> the specific concerns you may have about 3.2.0 stability and any bugs that
> you are hitting in 3.2.0 (if they already have not been filed yet). Inputs
> on 3.1.x is welcome too, please be specific on the information you provide
> us. Community feedback is of paramount importance to us, please file bugs to
> help us address them soon.
>
> Thanks,
> VS
>
> --
> Vidya Sakar N
> Senior Engineering Manager
> Gluster Software India (P) Ltd
> Bangalore
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster 3.2.0 - totally broken?

[paul simpson]

great to know - this is very reassuring to hear!  i know it's early days for
a file-system - and that fact so many people are using it so quickly (say,
as compared to BTRFS) is amazing.  i think there's lots of goodwill here -
which can/will translate into a even more vibrant community.  i look forward
to seeing these new developments roll out.

-paul


On 18 May 2011 21:16, Anand Babu Periasamy  wrote:

> GlusterFS is completely free. Same versions released to the community are
> used for commercial deployments too. Their issues gets higher priority
> though. Code related to other proprietary software such as VMWare, AWS,
> RightScale are kept proprietary.
>
> We acknowledge that we have done a poor job when it comes to  managing
> community, documentation and bug tracking. While we improved a lot since 2.x
> versions, I agree we are not there yet. We hired a lot of engineers to
> specifically focus on testing and bug fixes recently.  QA team is
> growing steadily. Lab size has been doubled. New QA lead is joining us next
> month. QA team will have closer interaction with the community moving
> forward. We also appointed Dave Garnett from HP as VP product manager and
> Vidya Sakar from Sun/Oracle as Engineering manager.
>
> We fully understand the importance of community. Paid vs Non-paid should
> not matter when it comes to quality of software. Intangible contributions
> from the community are equally valuable to the success of GlusterFS project.
>  We have appointed John Mark Walker as community manager. We launched
> community.gluster.org site recently. Starting next month, we will have
> regular community sessions. Problems raised by the community will also get
> prioritized.
>
> We are redoing the documentation completely. New system will be based on
> Red Hat's Publican. Documentation team too will closely work with the
> community.
>
> *Criticisms are taken positively. So please don't hesitate.*
> Thanks!
> -ab
>
> On Wed, May 18, 2011 at 7:05 AM, paul simpson wrote:
>
>> hi guys,
>>
>> we're using 3.1.3 and i'm not moving off it.  i totally agree with
>> stephans comments: the gluster devs *need* to concentrate on stability
>> before adding any new features.  it seems gluster dev is sales driven - not
>> tech focused.  we need less new buzz words - and more solid foundations.
>>
>> gluster is a great idea - but is in danger of falling short and failing if
>> the current trajectory is now altered.  greater posix compatibility
>> (permissions, NLM locking) should be a perquisite for an NFS server. hell,
>> the documentation is terrible; it's hard for us users to contribute to the
>> community when we are groping around in the dark too.
>>
>> question : is anyone using 3.2 in a real world production situation?
>>
>> regards to all,
>>
>> -paul
>>
>>
>> On 18 May 2011 14:54, Whit Blauvelt  wrote:
>>
>>> From reading this list, I wonder if this would be an accurate summary of
>>> the
>>> current state of Gluster:
>>>
>>> 3.1.3 - most dependable current version
>>>
>>> 3.1.4 - gained a few bugs
>>>
>>> 3.2.0 - not stable
>>>
>>> So 3.1.3 would be suitable for production systems, as long as the known
>>> bug
>>> in mishandling Posix group permissions is worked around (by loosening
>>> permissions).
>>>
>>> There has been a suggestion that stat-prefetch be turned off, and perhaps
>>> that other, non-default options are better not used.
>>>
>>> Now, I'm not personally knowledgeable on any of this aside from the Posix
>>> group problem. Just asking for confirmation or not of the basic sense I'm
>>> getting from those with extensive experience that 3.1.3 is essentially
>>> dependable, while 3.1.4 is problematic, and 3.2.0 should perhaps only be
>>> used if you want to gain familiarity with the new geo-replication
>>> feature,
>>> but avoided for current production use.
>>>
>>> Whit
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>
>
> --
> Anand Babu Periasamy
> Blog [http://www.unlocksmith.org]
>
> Imagination is more important than knowledge --Albert Einstein
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster 3.2.0 - totally broken?

[paul simpson]

hi guys,

we're using 3.1.3 and i'm not moving off it.  i totally agree with stephans
comments: the gluster devs *need* to concentrate on stability before adding
any new features.  it seems gluster dev is sales driven - not tech focused.
 we need less new buzz words - and more solid foundations.

gluster is a great idea - but is in danger of falling short and failing if
the current trajectory is now altered.  greater posix compatibility
(permissions, NLM locking) should be a perquisite for an NFS server. hell,
the documentation is terrible; it's hard for us users to contribute to the
community when we are groping around in the dark too.

question : is anyone using 3.2 in a real world production situation?

regards to all,

-paul


On 18 May 2011 14:54, Whit Blauvelt  wrote:

> From reading this list, I wonder if this would be an accurate summary of
> the
> current state of Gluster:
>
> 3.1.3 - most dependable current version
>
> 3.1.4 - gained a few bugs
>
> 3.2.0 - not stable
>
> So 3.1.3 would be suitable for production systems, as long as the known bug
> in mishandling Posix group permissions is worked around (by loosening
> permissions).
>
> There has been a suggestion that stat-prefetch be turned off, and perhaps
> that other, non-default options are better not used.
>
> Now, I'm not personally knowledgeable on any of this aside from the Posix
> group problem. Just asking for confirmation or not of the basic sense I'm
> getting from those with extensive experience that 3.1.3 is essentially
> dependable, while 3.1.4 is problematic, and 3.2.0 should perhaps only be
> used if you want to gain familiarity with the new geo-replication feature,
> but avoided for current production use.
>
> Whit
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] version 3.2

[paul simpson]

yes, +1 to all your points.

On 26 April 2011 17:06, Whit Blauvelt  wrote:

> Hi Renee,
>
> Perhaps the release notes are incomplete. There is no discussion at all
> about upgrading from 3.1.x. Only the 3.0 upgrade path is mentioned.
>
> Perhaps something could be added here?
>
>
> http://gluster.com/community/documentation/index.php/Gluster_3.2_Release_Notes:_Upgrading_to_Gluster_3.2
>
> Also, the release notes in the web doc make no mention of specific bugs
> fixed since 3.1. The new features mentioned are promising, exciting even,
> but for those of us who don't need those immediately, it would be quite
> useful to have a concise list of issues which have been addressed, to judge
> where to rush to upgrade, or to wait until others have discovered what new
> bugs might be in 3.2 - always a balancing act. See:
>
>
> http://gluster.com/community/documentation/index.php/Gluster_3.2_Release_Notes:_What%27s_New_in_Gluster_3.2%3F
>
> Thanks,
> Whit
>
> On Tue, Apr 26, 2011 at 10:45:36AM -0500, Renee Beckloff wrote:
> > Thanks for pointing this out!
> >
> > 3.2 has been GA'd to the community and qualified on CentOS 5.4.  There
> are
> > release notes available in the 3.2 documentation:
> > http://gluster.com/community/documentation/index.php/Main_Page
> >
> >
> > I will get the website updated today.
> >
> >
> >
> > Renee
> > Gluster Support
> >
> >
> > -Original Message-
> > From: gluster-users-boun...@gluster.org
> > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Whit Blauvelt
> > Sent: Tuesday, April 26, 2011 8:39 AM
> > To: Giovanni Toraldo
> > Cc: Gluster General Discussion List
> > Subject: Re: [Gluster-users] version 3.2
> >
> > Also:
> >
> > The docs as of today say "3.1.4 is the current stable version" - which
> > could
> > imply that 3.2.0 is still in a testing state. Is that accurate?
> >
> > Whit
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance

[paul simpson]

many thanks for sharing guys.  an informative read indeed!

i've 4x dells - each running 12 drives on PERC 600.  was dissapointed to
hear they're so bad!  we never got round to doing intensive tests this in
depth.  12x2T WD RE4 (sata) is giving me ~600Mb/s write on the bare
filesystem.  joe, does that tally with your expectations for 12 SATA drives
running RAID6?  (i'd put more faith in your gut reaction than our last
tests...)  ;)

-p


On 20 April 2011 21:02, Mohit Anchlia  wrote:

> Thanks a lot for taking time and effort. I will try raw performance
> first but that will only be going to one disk instead of 4. But I
> think it definitely makes sense as the first step.
>
> On Wed, Apr 20, 2011 at 12:53 PM, Joe Landman
>  wrote:
> > On 04/20/2011 03:43 PM, Mohit Anchlia wrote:
> >>
> >> Thanks! Is there any recommended configuration you want me to use when
> >> using mdadm?
> >>
> >> I got this link:
> >>
> >> http://tldp.org/HOWTO/Software-RAID-HOWTO-5.html#ss5.1
> >
> > First things first, break the RAID0, and then lets measure performance
> per
> > disk, to make sure nothing else bad is going on.
> >
> >dd if=/dev/zero of=/dev/DISK bs=128k count=80k oflag=direct
> >dd of=/dev/null if=/dev/DISK bs=128k count=80k iflag=direct
> >
> > for /dev/DISK being one of the drives in your existing RAID0.  Once we
> know
> > the raw performance, I'd suggest something like this
> >
> >mdadm --create /dev/md0 --metadata=1.2 --chunk=512 \
> >--raid-devices=4 /dev/DISK1 /dev/DISK2 \
> > /dev/DISK3 /dev/DISK4
> >mdadm --examine --scan | grep "md\/0" >> /etc/mdadm.conf
> >
> > then
> >
> >dd if=/dev/zero of=/dev/md0 bs=128k count=80k oflag=direct
> >dd of=/dev/null if=/dev/md0 bs=128k count=80k iflag=direct
> >
> > and lets see how it behaves.  If these are good, then
> >
> >mkfs.xfs -l version=2 -d su=512k,sw=4,agcount=32 /dev/md0
> >
> > (yeah, I know, gluster folk have a preference for ext* ... we generally
> > don't recommend ext* for anything other than OS drives ... you might need
> to
> > install xfsprogs and the xfs kernel module ... which kernel are you using
> > BTW?)
> >
> > then
> >
> >mount -o logbufs=4,logbsize=64k /dev/md0 /data
> >mkdir stress
> >
> >
> >dd if=/dev/zero of=/data/big.file bs=128k count=80k oflag=direct
> >dd of=/dev/null if=/data/big.file bs=128k count=80k iflag=direct
> >
> > and see how it handles things.
> >
> > When btrfs finally stabilizes enough to be used, it should be a
> reasonable
> > replacement for xfs, but this is likely to be a few years.
> >
> > --
> > Joseph Landman, Ph.D
> > Founder and CEO
> > Scalable Informatics Inc.
> > email: land...@scalableinformatics.com
> > web  : http://scalableinformatics.com
> >   http://scalableinformatics.com/sicluster
> > phone: +1 734 786 8423 x121
> > fax  : +1 866 888 3112
> > cell : +1 734 612 4615
> >
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Editing text files on NFS mounted volumes

[paul simpson]

I found this too. All permissions are ok. Root can write to file. File owner
can't! Any ideas how this can be further debugged...?
On Apr 6, 2011 11:53 AM, "Dan Bretherton" 
wrote:
> Sorry there are no errors with "cat" or "less".
> -Dan.
>
> --
> Mr. D.A. Bretherton
> Computer System Manager
> Environmental Systems Science Centre
> Harry Pitt Building
> 3 Earley Gate
> University of Reading
> Reading, RG6 6AL
> UK
>
> Tel. +44 118 378 5205
> Fax: +44 118 378 6413
>
>
> On 05/04/11 06:21, Shehjar Tikoo wrote:
>> Try catting a file from the shell. I'd like to see the error message
>> printed by a shell tool to see whats going on. Thanks.
>>
>>
>> Dan Bretherton wrote:
>>> Hello list-
>>> I have a strange problem opening text files on NFS mounted volumes
>>> with certain text editors. With nedit and gedit I get the error
>>> "Could not open file" or similar, and kwrite just show an empty
>>> file. Other editors such as emacs, vi, mousepad (XFCE4 desktop) and
>>> kedit (simple KDE editor) work fine. The problem is that nedit,
>>> gedit and kwrite are the ones most people want to use. None of the
>>> editors have any problems opening files on native GlusterFS mounts,
>>> but I only use the GlusterFS client on our compute servers and I
>>> don't want to have to install it on all the office PCs. Does anybody
>>> know why this problem occurs with NFS or how to stop it?
>>>
>>> -Dan.
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Editing text files on NFS mounted volumes

[paul simpson]

i have also seen this too.  even with vi!  and nothing in the log.  i'm in a
crunch period right now - so have not had a change to investigate further.
 anyone else out there seeing this?  it's scary...

i'm running:  3.1.3 glusterfs on 4x opensuse bricks.  dist-repl.  nfs &
glusterfs fuse clients.

-p

On 4 April 2011 19:00, Dan Bretherton  wrote:

> Hello list-
> I have a strange problem opening text files on NFS mounted volumes with
> certain text editors.  With nedit and gedit I get the error "Could not open
> file" or similar, and kwrite just show an empty file.  Other editors such as
> emacs, vi, mousepad (XFCE4 desktop) and kedit (simple KDE editor) work fine.
>  The problem is that nedit, gedit and kwrite are the ones most people want
> to use.  None of the editors have any problems opening files on native
> GlusterFS mounts, but I only use the GlusterFS client on our compute servers
> and I don't want to have to install it on all the office PCs.  Does anybody
> know why this problem occurs with NFS or how to stop it?
>
> -Dan.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] 3.1.3 NFS - cant overwrite certain files..

[paul simpson]

hi,

i'm running 3.1.3.  i'm finding that certain machines cant overwrite certain
files - getting a operation not permitted.  the files are owned by the same
user.  nothing appears on the gluster nfs log.  config:

g1:/var/log/glusterfs # gluster volume info

Volume Name: glustervol1
Type: Distributed-Replicate
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: g1:/mnt/glus1
Brick2: g2:/mnt/glus1
Brick3: g3:/mnt/glus1
Brick4: g4:/mnt/glus1
Brick5: g1:/mnt/glus2
Brick6: g2:/mnt/glus2
Brick7: g3:/mnt/glus2
Brick8: g4:/mnt/glus2
Options Reconfigured:
performance.write-behind-window-size: 1mb
performance.cache-size: 1gb
performance.stat-prefetch: 1
network.ping-timeout: 20
diagnostics.latency-measurement: off
diagnostics.dump-fd-stats: on


when mounted with FUSE - files can be overwritten.  this is very disruptive
- as it's stalling/breaking complex grid jobs.  has anyone seen
this behaviour at all?   ...any ideas???

regards to all,

paul

ps - this also happened with 3.1.2.  i just upgraded hoping that it would be
fixed.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster geo location setup?

[paul simpson]

+1 for greater robustness too.
On Mar 20, 2011 11:27 PM, "Count Zero"  wrote:
> Agreed 100% - For Gluster, this makes the most sense for now. It has to
establish itself as reliable and consistent, before it adds bells and
whistles.
>
>
>
> On Mar 21, 2011, at 2:05 AM, R.C. wrote:
>
>> I hope that, before moving development resources to the WAN extension,
some
>> more urgent features in the mainline will be implemented; one for all:
>> automatic self-heal startup.
>>
>> IMMO (In My Modest Opinion)
>>
>> Raf
>>
>>
>> - Original Message - From: "Count Zero" 
>> To: "Brent Clark" 
>> Cc: 
>> Sent: Sunday, March 20, 2011 9:49 PM
>> Subject: Re: [Gluster-users] Gluster geo location setup?
>>
>>
>>> As others responded, Gluster is not ready for WAN scenarios, and it is
>>> unclear if/when it will be...
>>> I am following the Gluster project closely, and have been using it for a
>>> year now on various internal test clusters.
>>>
>>> What can work for you right now is a project called 'sector-sphere'. It
is
>>> actually built with WAN in mind, is more secure than Gluster (uses
>>> encryption), and even supports a topology map, something Gluster does
not
>>> do (yet).
>>>
>>> (And by the way, even when it IS finally announced in Gluster, then
based
>>> on my experience with Gluster, I would be very wary to use the first
>>> version until a second or third version is released with bug
>>> fixes/improvements for the WAN mechanisms. Not trying to diss the
>>> technology, just being responsible and think you should be cautious and
>>> test things out thoroughly before pushing anything to production!).
>>>
>>>
>>> On Mar 19, 2011, at 8:42 PM, Brent Clark wrote:
>>>
 Hiya

 I would like to know if you can or if anyone is using Gluster in a
 geolocation setup for a Distributed filesystem sense.

 Reason I ask is. I got this two machines, and they are continents apart
 (Datacentre South Africa (Johannesburg) and Datacentre Germany).
 Currently Im using rsync to pull all updated and modified files.
 I was hoping I could use Gluster as a Active / Active Filesystem.

 If someone could share any thoughts or suggestions, it would be
 appreciated.
 Kind Regards
 Brent Clark
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Best practices after a peer failure?

[paul simpson]

i'm very keen to hear the definitive answer to this question.  there seems
an absolute dearth of documentation / best practices right now.  regarding
roadmap - this surely must be the priority...
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Mac / NFS problems

[paul simpson]

opps - please disregard that post.  we had a workaround - mounting a remount
on plain nfs.  this wasn't working initially - but now is due to that answer
(nolock).  the original gluster NFS export still doesn't work correctly.  we
can see the files system via NFS - but not write to it!

so, i repeat the original question: has anyone else got mac NFS working with
gluster?  or, anyone got mac FUSE to compile correctly?

-p


On 15 March 2011 10:46, paul simpson  wrote:

> well, answering our own question;  it seems that NFS on the mac (10.6.6)
> has become problematic due to the increased amount of NFS locking used.  you
> just mount with nolocks and things start working.  i hope this help someone
> else out there...
>
> regards,
>
> paul
>
>
> quoting from http://www.facebook.com/note.php?note_id=125738946623
>
> I don't know what it is about Apple and NFS, but they keep moving things
> around. The new UI to NFS mounting is much nicer than it was before, but
> it's now in a totally different place: the Disk Utility. But if you use a
> lot of NFS file systems, it's a pain to have to mount them one by one:
> ignoring the UI and using the /net automount filesystem is far more
> convenient. Just use the file name /net/hostname/path and you don't have to
> mess with any mounting, it just happens by automagic. I wrote a blog entry
> about this a long time ago.
> However, there is a huge problem with this: OS X does a phenominal amount
> of file locking (some would say, needlessly so) and has always been really
> sensitive to the configuration of locking on the NFS servers. So much so
> that if you randomly pick an NFS server in a large enterprise, true success
> is pretty unlikely. It'll succeed, but you'll keep getting messages
> indicating that the lock server is down, followed quickly by another message
> that the lock server is back up again. Even if you do get the NFS server
> tuned precisely the way that OS X wants it, performance sucks because of all
> the lock/unlock protocol requests that fly across the network. They clearly
> did something in Snow Leopard to aggravate this problem: it's now nasty
> enough to make NFS almost useless for me.
>
> Fortunately, there is a fix: just turn off network locking. You can do it
> by adding the "nolocks,locallocks" options in the advanced options field of
> the Disk Utility NFS mounting UI, but this is painful if you do a lot of
> them, and doesn't help at all with /net. You can edit /etc/auto_master to
> add these options to the /net entry, but it doesn't affect other mounts -
> however I do recommend deleting the hidefromfinder option in auto_master. If
> you want to fix every automount, edit /etc/autofs.conf and search for the
> line that starts with AUTOMOUNTD_MNTOPTS=. These options get applied on
> every mount. Add nolocks,locallocks and your world will be faster and
> happier after you reboot.
>
>
>
> On 11 March 2011 09:52, Shehjar Tikoo  wrote:
>
>> David Lloyd wrote:
>>
>>> Hello,
>>>
>>> Were having issues with macs writing to our gluster system.
>>> Gluster vol info at end.
>>>
>>> On a mac, if I make a file in the shell I get the following message:
>>>
>>> smoke:hunter david$ echo hello > test
>>> -bash: test: Operation not permitted
>>>
>>>
>> I can help if you can send the nfs.log file from the /etc/glusterd
>> directory on the nfs server. Before your mount command, set the log-level to
>> trace for nfs server and then run the echo command above. Unmount as soon as
>> you see the error above and email me the nfs.log.
>>
>> -Shehjar
>>
>>
>>
>>
>>> And the file is made but is zero size.
>>>
>>> smoke:hunter david$ ls -l test
>>> -rw-r--r--  1 david  realise  0 Mar  3 08:44 test
>>>
>>>
>>> glusterfs/nfslog logs thus:
>>>
>>> [2011-03-03 08:44:10.379188] I [io-stats.c:333:io_stats_dump_fd]
>>> glustervol1: --- fd stats ---
>>>
>>> [2011-03-03 08:44:10.379222] I [io-stats.c:338:io_stats_dump_fd]
>>> glustervol1:   Filename : /production/hunter/test
>>>
>>> Then try to open the file:
>>>
>>> smoke:hunter david$ cat test
>>>
>>> and get the following messages in the log:
>>>
>>> [2011-03-03 08:51:13.957319] I [afr-common.c:716:afr_lookup_done]
>>> glustervol1-replicate-0: background  meta-data self-heal triggered. path:
>>> /production/hunter/test
>>> [2011-03-03 08:51:13.959466] I
>>> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
>>&

Re: [Gluster-users] Mac / NFS problems

[paul simpson]

well, answering our own question;  it seems that NFS on the mac (10.6.6) has
become problematic due to the increased amount of NFS locking used.  you
just mount with nolocks and things start working.  i hope this help someone
else out there...

regards,

paul


quoting from http://www.facebook.com/note.php?note_id=125738946623

I don't know what it is about Apple and NFS, but they keep moving things
around. The new UI to NFS mounting is much nicer than it was before, but
it's now in a totally different place: the Disk Utility. But if you use a
lot of NFS file systems, it's a pain to have to mount them one by one:
ignoring the UI and using the /net automount filesystem is far more
convenient. Just use the file name /net/hostname/path and you don't have to
mess with any mounting, it just happens by automagic. I wrote a blog entry
about this a long time ago.
However, there is a huge problem with this: OS X does a phenominal amount of
file locking (some would say, needlessly so) and has always been really
sensitive to the configuration of locking on the NFS servers. So much so
that if you randomly pick an NFS server in a large enterprise, true success
is pretty unlikely. It'll succeed, but you'll keep getting messages
indicating that the lock server is down, followed quickly by another message
that the lock server is back up again. Even if you do get the NFS server
tuned precisely the way that OS X wants it, performance sucks because of all
the lock/unlock protocol requests that fly across the network. They clearly
did something in Snow Leopard to aggravate this problem: it's now nasty
enough to make NFS almost useless for me.

Fortunately, there is a fix: just turn off network locking. You can do it by
adding the "nolocks,locallocks" options in the advanced options field of the
Disk Utility NFS mounting UI, but this is painful if you do a lot of them,
and doesn't help at all with /net. You can edit /etc/auto_master to add
these options to the /net entry, but it doesn't affect other mounts -
however I do recommend deleting the hidefromfinder option in auto_master. If
you want to fix every automount, edit /etc/autofs.conf and search for the
line that starts with AUTOMOUNTD_MNTOPTS=. These options get applied on
every mount. Add nolocks,locallocks and your world will be faster and
happier after you reboot.



On 11 March 2011 09:52, Shehjar Tikoo  wrote:

> David Lloyd wrote:
>
>> Hello,
>>
>> Were having issues with macs writing to our gluster system.
>> Gluster vol info at end.
>>
>> On a mac, if I make a file in the shell I get the following message:
>>
>> smoke:hunter david$ echo hello > test
>> -bash: test: Operation not permitted
>>
>>
> I can help if you can send the nfs.log file from the /etc/glusterd
> directory on the nfs server. Before your mount command, set the log-level to
> trace for nfs server and then run the echo command above. Unmount as soon as
> you see the error above and email me the nfs.log.
>
> -Shehjar
>
>
>
>
>> And the file is made but is zero size.
>>
>> smoke:hunter david$ ls -l test
>> -rw-r--r--  1 david  realise  0 Mar  3 08:44 test
>>
>>
>> glusterfs/nfslog logs thus:
>>
>> [2011-03-03 08:44:10.379188] I [io-stats.c:333:io_stats_dump_fd]
>> glustervol1: --- fd stats ---
>>
>> [2011-03-03 08:44:10.379222] I [io-stats.c:338:io_stats_dump_fd]
>> glustervol1:   Filename : /production/hunter/test
>>
>> Then try to open the file:
>>
>> smoke:hunter david$ cat test
>>
>> and get the following messages in the log:
>>
>> [2011-03-03 08:51:13.957319] I [afr-common.c:716:afr_lookup_done]
>> glustervol1-replicate-0: background  meta-data self-heal triggered. path:
>> /production/hunter/test
>> [2011-03-03 08:51:13.959466] I
>> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
>> glustervol1-replicate-0: background  meta-data self-heal completed on
>> /production/hunter/test
>>
>> If I do the same test on a linux machine (nfs) it's fine.
>>
>> We get the same issue on all the macs. They are 10.6.6.
>>
>> Gluster volume is mounted:
>> /n/auto/gv1 -rw,hard,tcp,rsize=32768,wsize=32768,intr
>> gus:/glustervol1
>> Other nfs mounts on mac (from linux servers) are OK
>>
>> We're using LDAP to authenticate on the macs, the gluster servers aren't
>> bound into the LDAP domain.
>>
>> Any ideas?
>>
>> Thanks
>> David
>>
>>
>> g3:/var/log/glusterfs # gluster volume info
>> Volume Name: glustervol1
>> Type: Distributed-Replicate
>> Status: Started
>> Number of Bricks: 4 x 2 = 8
>> Transport-type: tcp
>> Bricks:
>> Brick1: g1:/mnt/glus1
>> Brick2: g2:/mnt/glus1
>> Brick3: g3:/mnt/glus1
>> Brick4: g4:/mnt/glus1
>> Brick5: g1:/mnt/glus2
>> Brick6: g2:/mnt/glus2
>> Brick7: g3:/mnt/glus2
>> Brick8: g4:/mnt/glus2
>> Options Reconfigured:
>> performance.stat-prefetch: 1
>> performance.cache-size: 1gb
>> performance.write-behind-window-size: 1mb
>> network.ping-timeout: 20
>> diagnostics.latency-measurement: off
>> diagnostics.dump-fd-stats: on
>>
>>
>>
>>
>>
>>
>>
>> ---

Re: [Gluster-users] Gluster-users Digest, Vol 35, Issue 25

[paul simpson]

i totally agree.  it is concerning to never really know if that
advertised redundancy is really there...


On 10 March 2011 20:25, Hareem Haque  wrote:

> There should be a better way to setup self healing.  As data file increase
> in numbers this triggered self healing would not be the most efficient
> method of healing cluster data.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Seeking Feedback on Gluster Development Priorities/Roadmap

[paul simpson]

hi ben,

it's good to be consulted!  my take on priorities would be:
0/ as per stephens email - basic syncing tools.  right now, i'm unsure (see
1) how often to do a "find sync" to "heal" the fs.  the baseline fs needs to
be more robust.  it feels very delicate and i often find myself needing to
restart a client/glusterd.
1/ documentation.  gluster looks great - but there's a lack of good solid
docs.  and lots of old legacy / conflicting documentation out there too.
 this is a barrier for new users (ie, me) to try it out.
2/ improved UI. by this, do you mean reporting/migration tools?  if so, yes.
3/ gNFS NLM locking

i'd concentrate on getting whats already there working better & faster than
adding new features.

looking forward to irc conference.

regards,

paul


Following that, our internal priorities are:
>
> -Continuous Data Replication (over WAN)
> -Improved User Interface
> -CIFS/Active Directory Support
> -Object storage  (unified file and object)
> -Geo-replication to Amazon S3 (unify public and private cloud)
> -Continuous Data Protection
> -REST management API's
> -Enhanced support for ISCSi SANs
>
> Are these the right priorities? How would you prioritize?
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2

[paul simpson]

hi shehjar,

many thanks for your reply.

that is 4 servers - serving ~30 clients - 95% linux, 5% mac.  all NFS.
>>
> Mac OS as a nfs client remains untested against Gluster NFS. Do you see
> these errors on Mac or Linux clients?


only tested locking on linux so far.


 - sometimes root can stat/read the file in question while the user cannot!
>>  i can remount the same NFS share to another mount point - and i can then
>> see that with the same user.
>>
> I think that may be occurring because NFS+LDAP requires a slightly
> different authentication scheme as compared to a NFS only setup. Please try
> the same test without LDAP in the middle.


will do tomorrow - been crazy busy fire fighting today.  (btw, root always
seemed to work - which is obviously not an ldap authenticated user - but
will try another one...)


> - sample output of g1 nfs.log file:
>>
>> [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd]
>> glustervol1:   Filename :
>> glustervol1:   Filename :
>> /production/conan/hda/published/shot/backup/.svn/tmp/entries
>>
>> ...so, the files not working don't have lifetime, read/written lines after
>> their log entry.
>>
>> I'll need the log for the NFS server in TRACE log level when you run a
> command that results in any of the errors above. i.e. stale file handle,
> remote IO error and input/output error.
>

will do.

> Locking is part of the core GlusterFS protocol but the NFS server module
does not have NLM support yet(NLM is the locking protocol associated with
NFSv3).
> On linux, the workaround is generally to mount with the -o nolock option
although I dont see why excluding this option results in stale file handle
and other
> errors. let me go through the complete thread, I'll reply elsewhere.

oh, thats a biggie - and not mentioned in any docs i've read so far.  i
think that a features table comparing NFS to gluster would be most useful.
 until that table appears - are there any other nfs v3 features that gluster
NFS doesn't support?  would allow us to manage our own expectations and use
gluster in an appropriate fashion...

> Today, if locking among multiple client machines is a must-have, you'll
have to use FUSE.

i have tried svn with multiple clients using FUSE - and have experienced the
same input/output errors.  should NLM locking in the fuse client work as per
NFS v3?

again thanks, logs will follow asap.

-paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2

[paul simpson]

hi joe,

it's very reassuring to read your post.  answers in-line.


> I do believe it is a serious and stable system.  We are running into
> specific bugs.  I'm ok with workarounds, but I really want to find a way to
> cause the problems.
>

same with both.  right now i'm putting more of my production tree back onto
an NFS server.  i think this is quite a reproducible bug: just check out a
subversion repo onto a distributed replicated volume - alter some files,
check them back in and repeat a few times.  within <20 mins we've been
getting input/output errors and stale NFS handles.  this is with both
NFS/fuse clients on multiple clients.


No one really questions other parallel FS, and from some of our experiences
> with them, they have far worse issues.  Even with the issues, Gluster is one
> of the best on the market.
>
> Gluster is evolving, but the issue is, without the replicator, all we can
> do is say "we are experiencing an issue" and hope it can be tracked.


i hear you - and very much understand the value of reproducible bug reports
:)  i hope above example helps.  i've already sent a detailed report to the
gluster people last week.  i'd be happy to give them a ssh account to see it
first hand.  i want to help/contribute...


> i'd really like to hear from any official gluster people out there -
>> right now the silence is deafening.  is this issue know? is it viewed a
>>
>
> I don't know if you have a support contract with them.  If you do, you
> should be speaking with them directly.  If not, their paying customers come
> first.


i have chatted with them - and certainly would expect that to service paying
customers (which we are not yet).  i've had couple of chats - and they seem
like a very cool helpful switched on bunch...

again, thanks for responding.  good to know i'm not alone..

-paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2

[paul simpson]

thanks for all responding - at least i know i'm not all alone.  i am shocked
to think that so many on this list are having serious fundamental issues
with glusterfs - and seemingly for a long time.  so, without wanting to
troll - my  question is: "is gluster a serious stable general purpose file
system"?  or, is it more a good "caching system" for a specific narrow
domain?

i'd really like to hear from any official gluster people out there - right
now the silence is deafening.  is this issue know? is it viewed a serious?
it is being worked on? i'm all up for volunteering to help by sending in a
test case, sending logs - whatever is asked of me.  i want to believe
gluster is going to work - as do many other sys-admins i know of in the
post/film industry.  however, i'm rapidly loosing confidence in gluster with
each passing day of silence...

in hope - paul


On Mon, Feb 21, 2011 at 6:47 PM, Joe Landman <
land...@scalableinformatics.com> wrote:

> On 02/21/2011 01:39 PM, Kon Wilms wrote:
>
>> On Mon, Feb 21, 2011 at 9:45 AM, Steve Wilson  wrote:
>>
>>> We had trouble with reliability for small, actively-accessed files on a
>>> distribute-replicate volume in both GlusterFS 3.11 and 3.12.  It seems
>>> that
>>> the replicated servers would eventually get out of sync with each other
>>> on
>>> these kinds of files.  For a while, we dropped replication and only ran
>>> the
>>> volume as distributed.  This has worked reliably for the past week or so
>>> without any errors that we were seeing before: no such file, invalid
>>> argument, etc.
>>>
>>
>> I'm running thousands of small files over NFSv3 through NGINX with
>> distribute and have had the opposite experience. Unfortunately when
>> NGINX can't access a file over NFS it means a customer calling us, so
>> right now gluster is basically sitting idle (posted my output to the
>> list a while back with no response).
>>
>
> We've had lots of issues with files disappearing or being inaccessible
> prior to 3.1.2 with the NFS client and server translator.  After 3.1.2, many
> of these problems *seem* to have been resolved, though all this means in
> this instance is that the customer hasn't submitted a ticket yet.
>
> I had thought it was originally a timebase issue ... as we had a minute or
> two drift on some of the nodes (since fixed).  But we had a pretty
> consistent error in this regard.
>
> We did open problem reports.  Unfortunately, no action so far (they just
> closed them this morning, though nothing has been solved per se, the issue
> simply has not yet resurfaced).  I'll leave those reports closed for now.
>
> This said, this error, or one with a very similar signature, has been in
> the code since the 2.x series.  I really ... really want to track it down,
> but I can't create a simple replicator for it to present to the team.  If
> you have what you think is a simple replicator, please, email me offline.
>  We'll try it here, and if we can get it down to a very simple replication
> case and test, we'll re-open the bugs.
>
> I'd hate to think its a heisenbug, but that is where I am leaning now.
>
>
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: land...@scalableinformatics.com
> web  : http://scalableinformatics.com
>   http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2

[paul simpson]

>
>
>>  Thanks, Joe.  Both servers use NTP against the same subnet router so it's
> unlikely that they had a time discrepancy.  I just checked the two servers
> and their times are consistent with each other at the moment.


...same with our machines here.  all running ntpd.

it's good to hear i'm not alone with this issue - it's been making me tear
my hair out.  it's a show-stopper for a file system...

joe - i really hope your correct with 3.1.3 fixing this.  i'd dearly love to
hear from any gluster people - anonymous or otherwise... ;)  these kind of
errors really undermine that cast iron feeling you need when thinking about
using a filesystem in a live production environment.

regards to all,

paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2

[paul simpson]

so, while your all about - my big question is can/does gluster (with
NFS/fuse client) properly lock files?

ie, a simple test is to checkout a svn tree to a gluster, modify, checkin,
list, alter, revert.  everytime i do this with 3.1.2 i get input/output
errors from my client machine within a few minutes.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2

[paul simpson]

hi fabricio,

many thanks for your input.  indeed i am using xfs - but that seems to be
mentioned in the gluster docs without any mention of problems.  we
benchmarked xfs vs ext4 - and found that xfs to be much better at dealing
with the bulk of our data - hi-def frames ~3-10M each - and large
geometry/particle/volume files.  10M-200M.  so, i'm keen to hear from anyone
abotu xfs's suitability for gluster storage...

as for file size; my understanding is that a distributed file system
performance only really kicks in when your dealing with large >1M files.
 however, is dealing with small files meant to be unreliable with
locking/access errors?

again thanks - and i look forward to hearing if gluster is able
to reliably serve svn working directories and cope with locks...

regards,

paul


On Mon, Feb 21, 2011 at 2:42 PM, Fabricio Cannini wrote:

> Em Sexta-feira 18 Fevereiro 2011, às 23:24:10, paul simpson escreveu:
> > hello all,
> >
> > i have been testing gluster as a central file server for a small
> animation
> > studio/post production company.  my initial experiments were using the
> fuse
> > glusterfs protocol - but that ran extremely slowly for home dirs and
> > general file sharing.  we have since switched to using NFS over
> glusterfs.
> >  NFS has certainly seemed more responsive re. stat and dir traversal.
> > however, i'm now being plagued with three different types of errors:
> >
> > 1/ Stale NFS file handle
> > 2/ input/output errors
> > 3/ and a new one:
> > $ l -l /n/auto/gv1/production/conan/hda/published/OLD/
> > ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot:
> > Remote I/O error
> > total 0
> > d? ? ? ? ?? shot
> >
> > ...so it's a bit all over the place.  i've tried rebooting both servers
> and
> > clients.  these issues are very erratic - they come and go.
> >
> > some information on my setup: glusterfs 3.1.2
> >
> > g1:~ # gluster volume info
> >
> > Volume Name: glustervol1
> > Type: Distributed-Replicate
> > Status: Started
> > Number of Bricks: 4 x 2 = 8
> > Transport-type: tcp
> > Bricks:
> > Brick1: g1:/mnt/glus1
> > Brick2: g2:/mnt/glus1
> > Brick3: g3:/mnt/glus1
> > Brick4: g4:/mnt/glus1
> > Brick5: g1:/mnt/glus2
> > Brick6: g2:/mnt/glus2
> > Brick7: g3:/mnt/glus2
> > Brick8: g4:/mnt/glus2
> > Options Reconfigured:
> >
> >
> > performance.write-behind-window-size: 1mb
> >
> >
> > performance.cache-size: 1gb
> >
> >
> > performance.stat-prefetch: 1
> >
> >
> > network.ping-timeout: 20
> >
> >
> > diagnostics.latency-measurement: off
> >
> >
> > diagnostics.dump-fd-stats: on
> >
> >
> > that is 4 servers - serving ~30 clients - 95% linux, 5% mac.  all NFS.
> >  other points:
> > - i'm automounting using NFS via autofs (with ldap).  ie:
> >   gus:/glustervol1 on /n/auto/gv1 type nfs
> > (rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13)
> > gus is pointing to rr dns machines (g1,g2,g3,g4).  that all seems to be
> > working.
> >
> > - backend files system on g[1-4] is xfs.  ie,
> >
> > g1:/var/log/glusterfs # xfs_info /mnt/glus1
> > meta-data=/dev/sdb1  isize=256agcount=7, agsize=268435200
> > blks
> >  =   sectsz=512   attr=2
> > data =   bsize=4096   blocks=1627196928,
> imaxpct=5
> >  =   sunit=256swidth=2560 blks
> > naming   =version 2  bsize=4096   ascii-ci=0
> > log  =internal   bsize=4096   blocks=32768, version=2
> >  =   sectsz=512   sunit=8 blks, lazy-count=0
> > realtime =none   extsz=4096   blocks=0, rtextents=0
> >
> >
> > - sometimes root can stat/read the file in question while the user
> cannot!
> >  i can remount the same NFS share to another mount point - and i can then
> > see that with the same user.
> >
> > - sample output of g1 nfs.log file:
> >
> > [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd]
> > glustervol1:   Filename :
> > /production/conan/hda/published/shot/backup/.svn/tmp/entries
> > [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd]
> > glustervol1:   BytesWritten : 1414 bytes
> > [2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd]
> > glustervol1: Write 001024b+ : 1
> > [2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd]
> > gluste

[Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2

[paul simpson]

hello all,

i have been testing gluster as a central file server for a small animation
studio/post production company.  my initial experiments were using the fuse
glusterfs protocol - but that ran extremely slowly for home dirs and general
file sharing.  we have since switched to using NFS over glusterfs.  NFS
has certainly seemed more responsive re. stat and dir traversal.  however,
i'm now being plagued with three different types of errors:

1/ Stale NFS file handle
2/ input/output errors
3/ and a new one:
$ l -l /n/auto/gv1/production/conan/hda/published/OLD/
ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot:
Remote I/O error
total 0
d? ? ? ? ?? shot

...so it's a bit all over the place.  i've tried rebooting both servers and
clients.  these issues are very erratic - they come and go.

some information on my setup: glusterfs 3.1.2

g1:~ # gluster volume info

Volume Name: glustervol1
Type: Distributed-Replicate
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: g1:/mnt/glus1
Brick2: g2:/mnt/glus1
Brick3: g3:/mnt/glus1
Brick4: g4:/mnt/glus1
Brick5: g1:/mnt/glus2
Brick6: g2:/mnt/glus2
Brick7: g3:/mnt/glus2
Brick8: g4:/mnt/glus2
Options Reconfigured:


performance.write-behind-window-size: 1mb


performance.cache-size: 1gb


performance.stat-prefetch: 1


network.ping-timeout: 20


diagnostics.latency-measurement: off


diagnostics.dump-fd-stats: on


that is 4 servers - serving ~30 clients - 95% linux, 5% mac.  all NFS.
 other points:
- i'm automounting using NFS via autofs (with ldap).  ie:
  gus:/glustervol1 on /n/auto/gv1 type nfs
(rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13)
gus is pointing to rr dns machines (g1,g2,g3,g4).  that all seems to be
working.

- backend files system on g[1-4] is xfs.  ie,

g1:/var/log/glusterfs # xfs_info /mnt/glus1
meta-data=/dev/sdb1  isize=256agcount=7, agsize=268435200
blks
 =   sectsz=512   attr=2
data =   bsize=4096   blocks=1627196928, imaxpct=5
 =   sunit=256swidth=2560 blks
naming   =version 2  bsize=4096   ascii-ci=0
log  =internal   bsize=4096   blocks=32768, version=2
 =   sectsz=512   sunit=8 blks, lazy-count=0
realtime =none   extsz=4096   blocks=0, rtextents=0


- sometimes root can stat/read the file in question while the user cannot!
 i can remount the same NFS share to another mount point - and i can then
see that with the same user.

- sample output of g1 nfs.log file:

[2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd]
glustervol1:   Filename :
/production/conan/hda/published/shot/backup/.svn/tmp/entries
[2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd]
glustervol1:   BytesWritten : 1414 bytes
[2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd]
glustervol1: Write 001024b+ : 1
[2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd]
glustervol1: --- fd stats ---
[2011-02-18 15:27:07.206032] I [io-stats.c:338:io_stats_dump_fd]
glustervol1:   Filename :
/production/conan/hda/published/shot/backup/.svn/props/tempfile.tmp
[2011-02-18 15:27:07.210799] I [io-stats.c:333:io_stats_dump_fd]
glustervol1: --- fd stats ---
[2011-02-18 15:27:07.210824] I [io-stats.c:338:io_stats_dump_fd]
glustervol1:   Filename :
/production/conan/hda/published/shot/backup/.svn/tmp/log
[2011-02-18 15:27:07.211904] I [io-stats.c:333:io_stats_dump_fd]
glustervol1: --- fd stats ---
[2011-02-18 15:27:07.211928] I [io-stats.c:338:io_stats_dump_fd]
glustervol1:   Filename :
/prod_data/xmas/lgl/pic/mr_all_PBR_HIGHNO_DF/035/1920x1080/mr_all_PBR_HIGHNO_DF.6084.exr
[2011-02-18 15:27:07.211940] I [io-stats.c:343:io_stats_dump_fd]
glustervol1:   Lifetime : 8731secs, 610796usecs
[2011-02-18 15:27:07.211951] I [io-stats.c:353:io_stats_dump_fd]
glustervol1:   BytesWritten : 2321370 bytes
[2011-02-18 15:27:07.211962] I [io-stats.c:365:io_stats_dump_fd]
glustervol1: Write 000512b+ : 1
[2011-02-18 15:27:07.211972] I [io-stats.c:365:io_stats_dump_fd]
glustervol1: Write 002048b+ : 1
[2011-02-18 15:27:07.211983] I [io-stats.c:365:io_stats_dump_fd]
glustervol1: Write 004096b+ : 4
[2011-02-18 15:27:07.212009] I [io-stats.c:365:io_stats_dump_fd]
glustervol1: Write 008192b+ : 4
[2011-02-18 15:27:07.212019] I [io-stats.c:365:io_stats_dump_fd]
glustervol1: Write 016384b+ : 20
[2011-02-18 15:27:07.212030] I [io-stats.c:365:io_stats_dump_fd]
glustervol1: Write 032768b+ : 54
[2011-02-18 15:27:07.228051] I [io-stats.c:333:io_stats_dump_fd]
glustervol1: --- fd stats ---
[2011-02-18 15:27:07.228078] I [io-stats.c:338:io_stats_dump_fd]
glustervol1:   Filename :
/production/conan/hda/published/shot/backup/.svn/tmp/entries

...so, the files not working don't have lifetime, read/written lines after
their log entry.

all very perplexing - and scary.  one thing that reliably fails is using sv

[Gluster-users] files not syncing up with glusterfs 3.1.1

[paul simpson]

hi all,

we're a post production company - and have our 30 machine renderfarm now
using this volume as a image and data store.  this has been running ok for a
week or so - until now.

i've just rendered a bunch of frames to the gluster volume - and while i'm
rendering i've been viewing the images.  however, now that all the frames
have finished rendering - i'm still seeing the older partially rendered
frames on my primary workstation.  if i copy the frames (by ssh'ing into
another machine to a shared location - i see them all correctly rendered).
so, my own machine seems intent on seeing them as they were over 20 mins
ago.  i'm not sure if this is a server or client issue.

i'm using/testing out gluster 3.1.1 on 4 machines using the following setup:

g1:~ # gluster volume info
Volume Name: glustervol1
Type: Distributed-Replicate
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: g1:/mnt/glus1
Brick2: g2:/mnt/glus1
Brick3: g3:/mnt/glus1
Brick4: g4:/mnt/glus1
Brick5: g1:/mnt/glus2
Brick6: g2:/mnt/glus2
Brick7: g3:/mnt/glus2
Brick8: g4:/mnt/glus2
Options Reconfigured:
performance.stat-prefetch: 1
performance.cache-size: 1gb
performance.write-behind-window-size: 512mb

...x4 dells with 10ge cards running opensuse 11.3 using glusterfs/fuse.  so
far it's been pretty good.

the volume is mounted via ldap/autofs (not that it should affect anything i
think):
g1:/glustervol1 on /n/auto/gv1 type fuse.glusterfs
(rw,allow_other,default_permissions,max_read=131072)

..i'm at a loss - and would appreciate any ideas/suggestions as to why this
is and can it be fixed.  it's a pretty fundamental issue - i'm hoping that
i've overlooked something obvious.

regards,

paul

ps - i've also "stat"ed all the files - that didn't seem to make any
difference.
pps - access permissions are ok too.  ie, not a 2nd group issue.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users