Re: [Gluster-users] tar_ssh.pem?

[Venky Shankar]

On 04/28/2014 07:00 PM, Joe Julian wrote:

Where is this process documented?

On April 28, 2014 6:03:16 AM PDT, Venky Shankar  wrote:

On 04/27/2014 11:55 PM, James Le Cuirot wrote:

Hello all,

I'm new to Gluster but have successfully tried geo-rep with 3.5.0.

I've

read about the new tar+ssh feature and it sounds good but nothing has
been said about the tar_ssh.pem file that gsyncd.conf references. Why
is a separate key needed? Does it not use gsyncd on the other end? If
not, what command should I lock it down to in authorized_keys, bug
#1091079 notwithstanding?

geo-replication "create push-pem" command should add the keys on the
slave for tar+ssh to work. That is done as part of geo-rep setup.

Thanks,
-venky
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

https://github.com/gluster/glusterfs/blob/release-3.5/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md

Thanks,
-venky
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] volume start causes glusterd to core dump in 3.5.0

[Carlos Capriotti]

Matthew, just out of curiosity, what is the underlying file system on your
gluster bricks ?

Reason for my asking is that there is (or there was) an issue with EXT4. If
you used it, you might want to reformat them using XFS -i 512.




On Tue, Apr 29, 2014 at 1:59 AM, Matthew Rinella wrote:

>
>
> I just built a pair of AWS Red Hat 6.5 instances to create a gluster
> replicated pair file system.  I can install everything, peer probe, and
> create the volume, but as soon as I try to start the volume, glusterd dumps
> core.
>
>
>
> The tail of the log after the crash:
>
>
>
>
> +--+
>
> [2014-04-28 21:49:18.102981] I
> [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
> from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f, host: fed-gfs4, port: 0
>
> [2014-04-28 21:49:18.138936] I
> [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
> Received friend update from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f
>
> [2014-04-28 21:49:18.138982] I
> [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
> uuid: c7a11029-12ab-4e3a-b898-7c62e98fa4d1, hostname:fed-gfs3
>
> [2014-04-28 21:49:18.138995] I
> [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
> uuid as Friend
>
> [2014-04-28 21:49:18.179134] I
> [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
> using the op-version 2
>
> [2014-04-28 21:49:18.199020] I
> [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
> Received probe from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f
>
> [2014-04-28 21:49:18.199111] I
> [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to fed-gfs4 (0), ret: 0
>
> [2014-04-28 21:49:18.48] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> 84c0bb8e-bf48-4386-ada1-3e4db68b980f, host: fed-gfs4
>
> [2014-04-28 21:49:18.262901] I
> [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f
>
> [2014-04-28 21:49:20.401429] I
> [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
>
> [2014-04-28 21:49:20.402072] I
> [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
>
> pending frames:
>
> frame : type(0) op(0)
>
>
>
> patchset: git://git.gluster.com/glusterfs.git
>
> signal received: 6
>
> time of crash: 2014-04-28 21:53:12configuration details:
>
> argp 1
>
> backtrace 1
>
> dlfcn 1
>
> fdatasync 1
>
> libpthread 1
>
> llistxattr 1
>
> setfsid 1
>
> spinlock 1
>
> epoll.h 1
>
> xattr.h 1
>
> st_atim.tv_nsec 1
>
> package-string: glusterfs 3.5.0
>
> /lib64/libc.so.6(+0x329a0)[0x7eff4946e9a0]
>
> /lib64/libc.so.6(gsignal+0x35)[0x7eff4946e925]
>
> /lib64/libc.so.6(abort+0x175)[0x7eff49470105]
>
> /usr/lib64/libcrypto.so.10(+0x67ebf)[0x7eff49837ebf]
>
> /usr/lib64/libcrypto.so.10(MD5_Init+0x49)[0x7eff4983e619]
>
> /usr/lib64/libcrypto.so.10(MD5+0x3a)[0x7eff4983e9ea]
>
> /usr/lib64/libglusterfs.so.0(md5_wrapper+0x3c)[0x7eff4ae6091c]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_set_socket_filepath+0x72)[0x7eff45e02b72]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_set_brick_socket_filepath+0x158)[0x7eff45e02df8]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0x4c9)[0x7eff45e094a9]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_brick_start+0x119)[0x7eff45e0af29]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_start_volume+0xfd)[0x7eff45e45a8d]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x53b)[0x7eff45df471b]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(gd_commit_op_phase+0xbe)[0x7eff45e5193e]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x2c2)[0x7eff45e53632]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_begin_synctask+0x3b)[0x7eff45e5376b]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(__glusterd_handle_cli_start_volume+0x1b6)[0x7eff45e46cc6]
>
>
> /usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7eff45ddaf7f]
>
> /usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7eff4ae818e2]
>
> /lib64/libc.so.6(+0x43bf0)[0x7eff4947fbf0]
>
>
>
> Interestingly enough I downloaded and installed  3.5.0 because this same
> thing happened with  3.4.2 on Red Hat 6.5 instances in AWS as well.  I tore
> down and rebuilt them with all the same results.Is there a library its
> conflicting with?  I am still new to gluster, so I don’t know too many of
> the details of what to look for when it crashes.
>
>
>
>
>
> Thanks for any help.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo

[Gluster-users] volume start causes glusterd to core dump in 3.5.0

[Matthew Rinella]

I just built a pair of AWS Red Hat 6.5 instances to create a gluster replicated 
pair file system.  I can install everything, peer probe, and create the volume, 
but as soon as I try to start the volume, glusterd dumps core.

The tail of the log after the crash:

+--+
[2014-04-28 21:49:18.102981] I 
[glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC 
from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f, host: fed-gfs4, port: 0
[2014-04-28 21:49:18.138936] I 
[glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received 
friend update from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f
[2014-04-28 21:49:18.138982] I 
[glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 
c7a11029-12ab-4e3a-b898-7c62e98fa4d1, hostname:fed-gfs3
[2014-04-28 21:49:18.138995] I 
[glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid 
as Friend
[2014-04-28 21:49:18.179134] I 
[glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: 
using the op-version 2
[2014-04-28 21:49:18.199020] I 
[glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: 
Received probe from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f
[2014-04-28 21:49:18.199111] I 
[glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded 
to fed-gfs4 (0), ret: 0
[2014-04-28 21:49:18.48] I 
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 
84c0bb8e-bf48-4386-ada1-3e4db68b980f, host: fed-gfs4
[2014-04-28 21:49:18.262901] I 
[glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received 
ACC from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f
[2014-04-28 21:49:20.401429] I 
[glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received 
get vol req
[2014-04-28 21:49:20.402072] I 
[glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received 
get vol req
pending frames:
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 2014-04-28 21:53:12configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.5.0
/lib64/libc.so.6(+0x329a0)[0x7eff4946e9a0]
/lib64/libc.so.6(gsignal+0x35)[0x7eff4946e925]
/lib64/libc.so.6(abort+0x175)[0x7eff49470105]
/usr/lib64/libcrypto.so.10(+0x67ebf)[0x7eff49837ebf]
/usr/lib64/libcrypto.so.10(MD5_Init+0x49)[0x7eff4983e619]
/usr/lib64/libcrypto.so.10(MD5+0x3a)[0x7eff4983e9ea]
/usr/lib64/libglusterfs.so.0(md5_wrapper+0x3c)[0x7eff4ae6091c]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_set_socket_filepath+0x72)[0x7eff45e02b72]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_set_brick_socket_filepath+0x158)[0x7eff45e02df8]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0x4c9)[0x7eff45e094a9]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_brick_start+0x119)[0x7eff45e0af29]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_start_volume+0xfd)[0x7eff45e45a8d]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x53b)[0x7eff45df471b]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(gd_commit_op_phase+0xbe)[0x7eff45e5193e]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x2c2)[0x7eff45e53632]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_begin_synctask+0x3b)[0x7eff45e5376b]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(__glusterd_handle_cli_start_volume+0x1b6)[0x7eff45e46cc6]
/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7eff45ddaf7f]
/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7eff4ae818e2]
/lib64/libc.so.6(+0x43bf0)[0x7eff4947fbf0]

Interestingly enough I downloaded and installed  3.5.0 because this same thing 
happened with  3.4.2 on Red Hat 6.5 instances in AWS as well.  I tore down and 
rebuilt them with all the same results.Is there a library its conflicting 
with?  I am still new to gluster, so I don't know too many of the details of 
what to look for when it crashes.


Thanks for any help.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Need help in understanding volume heal-info behavior

[Chalcogen]

Thank you very much!

On Monday 28 April 2014 07:41 AM, Ravishankar N wrote:

On 04/28/2014 01:30 AM, Chalcogen wrote:

Hi everyone,

I have trouble understanding the following behavior:

Suppose I have a replica 2 volume 'testvol' on two servers, server1 
and server2, composed of server1:/bricks/testvol/brick and 
server2:/bricks/testvol/brick. Also, suppose it contains a good 
number of files.


Now, assume I remove one of the two bricks, as:

root@server1~# gluster volume remove-brick testvol replica 1 
server1:/bricks/testvol/brick


Now, I unmount and delete the logical volume supporting the brick and 
then recreate it (with a different size), and mount it the same way 
as it was mounted before (at /brick/testvol/). Then, I re-add it as:


root@server1~# gluster volume add-brick testvol replica 2 
server1:/bricks/testvol/brick


I observe that the brick on server1 does not contain any of the data 
that was in the volume.


root@server1~# ls /bricks/testvol/brick
root@server1~#

This is all right by me, since glusterfs needs some time to discover 
and sync files that are absent on the brick of server1. In fact, if I 
leave the setup undisturbed for 15 minutes to half an hour, I find 
that all data appears within the brick of server1, just as you would 
expect. Also, if I wish to speed up the process, I simply do a ls -Ra 
on the directory where the volume is mounted, and all files sync onto 
server1's brick. This is also very much as expected.


However, during the period where data on server1's brick is not 
available, if you query the heal info for the volume, gluster cli 
reports that 'Number of entries' is '0', and that too all of 'info', 
'heal-failed', and 'split-brain'. This is what becomes a bit of a 
trouble for me. Fact is, we are attempting to automate the monitoring 
of our glusterfs volumes, and we depend upon heal info alone to 
decide whether data on server1 and server2 are in sync.


Could somebody, therefore, help me with the following questions?
a) Which files exactly show up in heal info?
The files which are healed either by the self-heal daemon or by the 
gluster heal commands.
b) What exactly should I look to monitor if we are to ascertain that 
data on our servers are in sync?


After adding a new replica brick, you need to run a full heal (gluster 
volume heal  full). Then the results will show up in the 
heal info output.

Thanks a lot for your responses!

Anirban

P.s. I am using glusterfs 3.4.2 over linux kernel version 2.6.34.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] New upstream glusterfs-hadoop release 2.1.10

[Jay Vyas]

Hi folks.  We've now pushed a couple new releases to glusterfs-hadoop.

1) We now support hadoop-2.3.0 using standard instructions on the forge for
Linux container setup, with the caveat that you use GlusterLinuxContainer.
This is a workaround for YARN-1235.

2) We also have fixed the unit tests so that they test 2.0 specs, avoiding
conflicts where 1.0 and 2.0 FS semantics differ (testListStatus).

Happy hadooping !

The plugin jars can be found here :
http://rhbd.s3.amazonaws.com/maven/indexV2.html

-- 
Jay Vyas
http://jayunit100.blogspot.com
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Can i shut down unneeded Bricks?

[Dragon]

Hi,

i have a 3 Brick Distributed Volume. 2 out of 3 Bricks filled up and only on Brick2 are enough free space left. The question is to safe power, could i shutdown the other bricks if i copy data on the volume? The data will copy to brick2. Is this a problem for the logic of glusterfs, or is this a way to safe power at home ;).

 

thx
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Data placement algorithm

[Joe Julian]

GlustetFS distribute translator (file based, not block based) uses a 
distributed hash translation. See 
http://joejulian.name/blog/dht-misses-are-expensive/ for a detailed explanation.

On April 28, 2014 2:28:42 AM PDT, "Séguin Cyril"  
wrote:
>Hy all,
>
>I'm currently interesting in comparing Gluster's block replica
>placement 
>policy with other placements algorithm.
>
>Is it possible to access to the source code of Gluster's placement 
>policy and where can I find it?
>
>Thanks a lot.
>
>Best regards.
>
>CS
>
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://supercolony.gluster.org/mailman/listinfo/gluster-users

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Data placement algorithm

[Séguin Cyril]

Hy all,

I'm currently interesting in comparing Gluster's block replica placement 
policy with other placements algorithm.


Is it possible to access to the source code of Gluster's placement 
policy and where can I find it?


Thanks a lot.

Best regards.

CS

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] OS X porting merged

[Andrew Hatfield]

Hey Dan,

Did you ever test SMB with the streams xattr vfs object?

Did it work?  Was performance ok?


On Mon, Apr 28, 2014 at 1:57 PM, Dan Mons  wrote:

> Preliminary success - compiled, installed and mounting on MacOSX
> 10.8.5 (our production rollout here).
>
> Finder in preview mode loads up a folder with 406 images quite
> quickly, and scrubs back and forth at reasonable speed (much better
> than SMB, comparable to NFS).  My test machine is quite old however,
> so I need to push it to a better machine to get some good performance
> and usage stats.
>
> Now to find some willing volunteers to put it through it's paces.
> Shouldn't be too hard, because folks here hate SMB. :)
>
> -Dan
>
>
> 
> Dan Mons
> Unbreaker of broken things
> Cutting Edge
> http://cuttingedge.com.au
>
>
> On 27 April 2014 12:09, Justin Clift  wrote:
> > This is EXCELLENT! :D
> >
> > Once we're happy with the state of GlusterFS on OSX, I'll see about
> > creating a Homebrew formula for it.  That'll enable a much wider
> > MacOS userbase (with appropriate promotion). :)
> >
> > + Justin
> >
> > On 26/04/2014, at 10:22 AM, Dan Mons wrote:
> >> I'm very exited about this. I'll be testing very soon.
> >>
> >> -Dan
> >>
> >> On 26 Apr 2014 18:53, "Dennis Schafroth"  wrote:
> >> Hi
> >>
> >> I just wanted to inform that the OS X porting has been merged to
> masterb. There are no longer any need to use the alternative repo. Thanks
> to Harsha for effort on porting and for putting it through the review
> process.
> >>
> >> It can now build and run GlusterFS clients  on OS X, FUSE mounting
> Linux servers. This is still very new, so please test it well before using
> it on production system.
> >>
> >> On the server part we are not there yet. I have been running OS X
> bricks with OS X clients, but only replicated ones seems to work.
> Distributed ends up with duplicate file entries  in the fuse mount.
> >>
> >> It requires OSX FUSE (http://osxfuse.github.io/) installed.
> >>
> >> cheers,
> >> :-Dennis Schafroth
> >>
> >>
> >> ___
> >> Gluster-devel mailing list
> >> gluster-de...@gluster.org
> >> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
> >>
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >
> > --
> > Open Source and Standards @ Red Hat
> >
> > twitter.com/realjustinclift
> >
> ___
> Gluster-devel mailing list
> gluster-de...@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] tar_ssh.pem?

[Joe Julian]

Where is this process documented?

On April 28, 2014 6:03:16 AM PDT, Venky Shankar  wrote:
>On 04/27/2014 11:55 PM, James Le Cuirot wrote:
>> Hello all,
>>
>> I'm new to Gluster but have successfully tried geo-rep with 3.5.0.
>I've
>> read about the new tar+ssh feature and it sounds good but nothing has
>> been said about the tar_ssh.pem file that gsyncd.conf references. Why
>> is a separate key needed? Does it not use gsyncd on the other end? If
>> not, what command should I lock it down to in authorized_keys, bug
>> #1091079 notwithstanding?
>
>geo-replication "create push-pem" command should add the keys on the 
>slave for tar+ssh to work. That is done as part of geo-rep setup.
>
>Thanks,
>-venky
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://supercolony.gluster.org/mailman/listinfo/gluster-users

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] tar_ssh.pem?

[Venky Shankar]

On 04/27/2014 11:55 PM, James Le Cuirot wrote:

Hello all,

I'm new to Gluster but have successfully tried geo-rep with 3.5.0. I've
read about the new tar+ssh feature and it sounds good but nothing has
been said about the tar_ssh.pem file that gsyncd.conf references. Why
is a separate key needed? Does it not use gsyncd on the other end? If
not, what command should I lock it down to in authorized_keys, bug
#1091079 notwithstanding?


geo-replication "create push-pem" command should add the keys on the 
slave for tar+ssh to work. That is done as part of geo-rep setup.


Thanks,
-venky
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] OS X porting merged

[Dan Mons]

I should add to all of this, currently we are doing:

* On MacOSX workstations, we have a small shell script that initiates
SMB mounts of our Gluster setup, so that artists can mount up into the
correct absolute path (no, not /Volumes with a capital V, thank you
Apple).  Artists need to run this once per logon on workstations.
Painful, but less painful than Finder NFS dramas.

* On MacOSX render nodes, we're using NFS3 currently, mounted via
autofs. It appears that most apps (the good ones at least - i.e.:
anything not Adobe, which is a whole other rant) are OK with CLI batch
processing and read/write operations to GlusterFS that way.  The
difference appears to be whatever framework the developers use - most
of our developers use the BSD subsystem of OSX which works to NFS.
Ones that use the Carbon/Cocoa/Finder framework break.

* All Linux workstations and render nodes (all Kubuntu 12.04LTS) use
FUSE+GlusterFS mounted via autofs, because it works better.
Exceptions are legacy "black box" vendor machines like Autodesk Flame
(RHEL5.3 base with a custom kernel and no supported way to add new
stuff) that don't offer FUSE natively, so they're stuck on NFS.  Add
Auodesk to the vendor rant list directly behind Adobe and Apple.
Protip: don't start a software company that begins with the letter A.
It's bad mojo.

* Windows - we dumped it at the end of last year.  Useless for us for
so many reasons.  (Microsoft are the exception that proves the "A*"
rule).

The goal is to give this recent MacOSX porting effort a good solid
bash, and get it into production if it appears relatively stable.
Because testing code in production is how we roll. :)

-Dan



Dan Mons
Unbreaker of broken things
Cutting Edge
http://cuttingedge.com.au


On 28 April 2014 19:44, Dan Mons  wrote:
> On 28 April 2014 16:11, Andrew Hatfield
>  wrote:
>> Hey Dan,
>>
>> Did you ever test SMB with the streams xattr vfs object?
>
> Yes. :)
>
>> Did it work?  Was performance ok?
>
> It functions, but forces us to compromise.  It was the only way to get
> GlusterFS working via SMB on MacOSX.
>
> For anyone who hasn't discovered this: MacOSX's Finder looks for a
> "._filename" resource fork file for every "filename" data file it
> finds.  GlusterFS (and any clustered file system, to be fair) is quite
> poor at negative lookups (i.e.: when it can't find a file on a
> particular brick, it has to run around asking every brick in the
> cluster if they have the file), and so a folder with ~1000 files in it
> (quite common for a VFX studio, across dozens of shots inside dozens
> of projects) can take several minutes to populate when trying to view
> it in OSX's Finder.
>
> streams xattr means that the resource fork information can be held in
> the xattr component of the underlying file system (for us that's XFS
> on our CentOS6 bricks), and completely removes the need for the
> negative lookup, as that resource fork data lives with the file.  (I
> for one find the whole resource fork concept totally outdated and
> silly, but Apple seem keen to keep it around).
>
> Downsides:
>
> 1) SMB is slower than either NFS or FUSE+GlusterFS.  vfs_glusterfs
> helps, but still doesn't compete in real world usage.
>
> 2) MacOSX does not allow system-wide mounting of SMB shares, ala NFS3.
>  Our studio relies heavily on this as machines need network file
> systems mounted up so that multiple users can hit them at once.  We
> run a large number of machines in what's called a "render farm" which
> are batch-processing multiple jobs each in parallel. Our standard spec
> render node currently is a dual Xeon (8 cores per proc, 2 procs, per
> node, with HT = 32 threads / logical CPUs) and 128GB RAM.  These
> sometimes do one very large job, and sometimes many smaller jobs in
> parallel.  Jobs run as the UID of the person who submitted them, so
> NFS3 style network mounting is mandatory for this to work.  There is
> no such thing as "one user on one machine" in our world.
>
> MacOSX can't do this via SMB.  In 10.8 and 10.9, whichever user
> initiates the SMB connection "owns" the mount (regardless of the
> credentials they use at mount time), and any other UID is locked out
> of that share.
>
> Upsides:
>
> 1) At least it works at all, unlike NFS which Apple broke in Finder
> back in 10.5 and have refused to even acknowledge let alone fix.
> Honestly, how does an OS based on UNIX not do NFS?  10.9 now
> represents 5 complete production versions of their OS in a row with
> broken NFS.  Ludicrous.
>
> 2) AFP has the same UID-clobbering mount issues as SMB, but SMB is
> faster than AFP (thanks to streams xattr) and we can cluster SMB more
> easily.
> Long story short, SMB is a stop-gap.  Ideally either Apple fix NFS
> (I'm not holding my breath, as it's clear Apple's priority is selling
> iPhones and media, and not actually offering a usable operating
> system), or the Gluster community add the ever moving target of MacOSX
> support to their FUSE+GlusterFS client, which i

Re: [Gluster-users] [Gluster-devel] OS X porting merged

[Dan Mons]

On 28 April 2014 16:11, Andrew Hatfield
 wrote:
> Hey Dan,
>
> Did you ever test SMB with the streams xattr vfs object?

Yes. :)

> Did it work?  Was performance ok?

It functions, but forces us to compromise.  It was the only way to get
GlusterFS working via SMB on MacOSX.

For anyone who hasn't discovered this: MacOSX's Finder looks for a
"._filename" resource fork file for every "filename" data file it
finds.  GlusterFS (and any clustered file system, to be fair) is quite
poor at negative lookups (i.e.: when it can't find a file on a
particular brick, it has to run around asking every brick in the
cluster if they have the file), and so a folder with ~1000 files in it
(quite common for a VFX studio, across dozens of shots inside dozens
of projects) can take several minutes to populate when trying to view
it in OSX's Finder.

streams xattr means that the resource fork information can be held in
the xattr component of the underlying file system (for us that's XFS
on our CentOS6 bricks), and completely removes the need for the
negative lookup, as that resource fork data lives with the file.  (I
for one find the whole resource fork concept totally outdated and
silly, but Apple seem keen to keep it around).

Downsides:

1) SMB is slower than either NFS or FUSE+GlusterFS.  vfs_glusterfs
helps, but still doesn't compete in real world usage.

2) MacOSX does not allow system-wide mounting of SMB shares, ala NFS3.
 Our studio relies heavily on this as machines need network file
systems mounted up so that multiple users can hit them at once.  We
run a large number of machines in what's called a "render farm" which
are batch-processing multiple jobs each in parallel. Our standard spec
render node currently is a dual Xeon (8 cores per proc, 2 procs, per
node, with HT = 32 threads / logical CPUs) and 128GB RAM.  These
sometimes do one very large job, and sometimes many smaller jobs in
parallel.  Jobs run as the UID of the person who submitted them, so
NFS3 style network mounting is mandatory for this to work.  There is
no such thing as "one user on one machine" in our world.

MacOSX can't do this via SMB.  In 10.8 and 10.9, whichever user
initiates the SMB connection "owns" the mount (regardless of the
credentials they use at mount time), and any other UID is locked out
of that share.

Upsides:

1) At least it works at all, unlike NFS which Apple broke in Finder
back in 10.5 and have refused to even acknowledge let alone fix.
Honestly, how does an OS based on UNIX not do NFS?  10.9 now
represents 5 complete production versions of their OS in a row with
broken NFS.  Ludicrous.

2) AFP has the same UID-clobbering mount issues as SMB, but SMB is
faster than AFP (thanks to streams xattr) and we can cluster SMB more
easily.
Long story short, SMB is a stop-gap.  Ideally either Apple fix NFS
(I'm not holding my breath, as it's clear Apple's priority is selling
iPhones and media, and not actually offering a usable operating
system), or the Gluster community add the ever moving target of MacOSX
support to their FUSE+GlusterFS client, which is now moving along
nicely as per this thread.

So my eternal thanks to everyone contributing to this OSX porting
effort (and of course everyone who's ever contributed a line of code
to GlusterFS in general).  You are all wonderful human beings.  :)

-Dan
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to manually/force remove a bad geo-rep agreement?

[Vijaykumar Koppad]

On 04/24/2014 07:40 PM, Steve Dainard wrote:

# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 stop
geo-replication command failed'

Try out

# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 stop force
# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 delete



cli.log:
2014-04-24 14:08:03.916112] W [rpc-transport.c:175:rpc_transport_load] 
0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2014-04-24 14:08:03.918050] I [socket.c:3480:socket_init] 
0-glusterfs: SSL support is NOT enabled
[2014-04-24 14:08:03.918068] I [socket.c:3495:socket_init] 
0-glusterfs: using system polling thread

[2014-04-24 14:08:04.146710] I [input.c:36:cli_batch] 0-: Exiting with: -1



*Steve Dainard *
IT Infrastructure Manager
Miovision  | /Rethink Traffic/

*Blog  | **LinkedIn 
  | Twitter 
  | Facebook 
*


Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, Kitchener, 
ON, Canada | N2C 1L3
This e-mail may contain information that is privileged or 
confidential. If you are not the intended recipient, please delete the 
e-mail and any attachments and notify us immediately.



On Thu, Apr 24, 2014 at 9:47 AM, Steve Dainard > wrote:


Version: glusterfs-server-3.4.2-1.el6.x86_64

I have an issue where I'm not getting the correct status for
geo-replication, this is shown below. Also I've had issues where
I've not been able to stop geo-replication without using a
firewall rule on the slave. I would get back a cryptic error and
nothing useful in the logs.

# gluster volume geo-replication status
NODE MASTER   SLAVE  
   STATUS


---
ovirt001.miovision.corp rep1 gluster://10.0.11.4:/rep1  faulty
ovirt001.miovision.corp miofiles gluster://10.0.11.4:/miofiles
 faulty

# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 start
geo-replication session between rep1 & gluster://10.0.11.4:/rep1
already started
geo-replication command failed

[root@ovirt001 ~]# gluster volume geo-replication status
NODE MASTER   SLAVE  
   STATUS


---
ovirt001.miovision.corp rep1 gluster://10.0.11.4:/rep1  faulty
ovirt001.miovision.corp miofiles gluster://10.0.11.4:/miofiles
 faulty


How can I manually remove a geo-rep agreement?

Thanks,


*Steve
*




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to manually/force remove a bad geo-rep agreement?

[Vijaykumar Koppad]

On 04/24/2014 07:40 PM, Steve Dainard wrote:

# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 stop
geo-replication command failed'

Try out

# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 stop force
# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 delete

-Vijaykumar


cli.log:
2014-04-24 14:08:03.916112] W [rpc-transport.c:175:rpc_transport_load] 
0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2014-04-24 14:08:03.918050] I [socket.c:3480:socket_init] 
0-glusterfs: SSL support is NOT enabled
[2014-04-24 14:08:03.918068] I [socket.c:3495:socket_init] 
0-glusterfs: using system polling thread

[2014-04-24 14:08:04.146710] I [input.c:36:cli_batch] 0-: Exiting with: -1



*Steve Dainard *
IT Infrastructure Manager
Miovision  | /Rethink Traffic/

*Blog  | **LinkedIn 
  | Twitter 
  | Facebook 
*


Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, Kitchener, 
ON, Canada | N2C 1L3
This e-mail may contain information that is privileged or 
confidential. If you are not the intended recipient, please delete the 
e-mail and any attachments and notify us immediately.



On Thu, Apr 24, 2014 at 9:47 AM, Steve Dainard > wrote:


Version: glusterfs-server-3.4.2-1.el6.x86_64

I have an issue where I'm not getting the correct status for
geo-replication, this is shown below. Also I've had issues where
I've not been able to stop geo-replication without using a
firewall rule on the slave. I would get back a cryptic error and
nothing useful in the logs.

# gluster volume geo-replication status
NODE MASTER   SLAVE  
   STATUS


---
ovirt001.miovision.corp rep1 gluster://10.0.11.4:/rep1  faulty
ovirt001.miovision.corp miofiles gluster://10.0.11.4:/miofiles
 faulty

# gluster volume geo-replication rep1 gluster://10.0.11.4:/rep1 start
geo-replication session between rep1 & gluster://10.0.11.4:/rep1
already started
geo-replication command failed

[root@ovirt001 ~]# gluster volume geo-replication status
NODE MASTER   SLAVE  
   STATUS


---
ovirt001.miovision.corp rep1 gluster://10.0.11.4:/rep1  faulty
ovirt001.miovision.corp miofiles gluster://10.0.11.4:/miofiles
 faulty


How can I manually remove a geo-rep agreement?

Thanks,


*Steve
*




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users