Re: [Gluster-users] Typical setup questions
Guys, Thanks for the responses it is appreciated. On 8/28/12 5:28 PM, Bryan Whitehead wrote: I'f found pricing for Infiniband switches / cards to be cheaper than 10G cards/switches with the addition of being 4X fast. I will look into this but putting all of our compute on Infiniband may be cost prohibitive. On Tue, Aug 28, 2012 at 11:44 AM, Joe Topjian wrote: Hi Matt, On Tue, Aug 28, 2012 at 9:29 AM, Matt Weil wrote: Since we are on the subject of hardware what would be the perfect fit for a gluster brick. We where looking at a PowerEdge C2100 Rack Server. Just a note: the c2100 has been superseded by the Dell r720xd. Although the r720 is not part of the c-series, it's their official replacement. I looked at these but they only hold 8 3.5" drives verses 12 and two on the inside on the 2100. I will ask our rep about this. Do you typically run hot spares or just keep cold spares handy? During testing I found it pretty easy to saturate 1 Gig network links. This was also the case when multiple links where bonded together. Are there any cheap 10 gig switch alternatives that anyone would suggest? While not necessarily cheap, I've had great luck with Arista 7050 switches. also looking at dell's new force 10 switches. Wonder how they compare price wise. We implement them in sets of two, linked together. We then use dual-port 10gb NICs and connect each NIC to each switch. It gives multiple layers of redundancy + a theoretical 20gb throughput per server. Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Typical setup questions
Brian thanks for this response. Since we are on the subject of hardware what would be the perfect fit for a gluster brick. We where looking at a PowerEdge C2100 Rack Server. During testing I found it pretty easy to saturate 1 Gig network links. This was also the case when multiple links where bonded together. Are there any cheap 10 gig switch alternatives that anyone would suggest? Matt On 8/24/12 4:28 PM, Brian Candler wrote: On Fri, Aug 24, 2012 at 10:51:24AM -0500, Matt Weil wrote: I am curious what is used typically for the file system replication and how do you make sure that it is consistent. So for example when using large 3TB+ sata/NL-sas drives. Is is typical to replicate three times to get similar protection to raid 6? Gluster sits on top of existing filesystems on the storage bricks, so it's fine to continue to use RAID10 (for performance) or RAID6 (for capacity) on those nodes. Gluster replicated volumes, and/or gluster geo-replication, then give you an additional layer of replication on top of that, and the ability to handle entire servers going out of service. If I were you, I would not want to have a non-resilient array like a RAID0 on my storage bricks. Whilst in principle you could have lots of separate 3TB filesystems and put them into a large distributed/replicated set, I think this is likely to be difficult to manage. In particular, the process of replacing a failed disk requires more skill than a simple RAID drive swap. One word of warning: when choosing 3TB SATA drives, ensure they support error recovery control (a.k.a. time-limited error recovery). Enterprise drives do, but many consumer ones don't. The Hitachi consumer ones do, for now anyway; Seagate ones do not. To attempt to enable it on a particular drive: # smartctl -l scterc,70,70 /dev/sda If the drive supports it, you'll see: SCT Error Recovery Control set to: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) There's plenty of discussion on the linux-raid mailing list if you want to go through the archives. Also what is typically done to ensure that all replicas are in place and consistent? A cron that stats of ls's the file system from a single client? I don't have a good answer to that. Stat'ing all files recursively used to be required for gluster <3.3 to force healing. As of gluster 3.3, there is a self-healing daemon which handles this automatically. So basically, you trust gluster to do its job. I guess there could be value in running a recursive md5sum on each replica locally and comparing the results (but you'd have to allow for files which were in the process of changing during the scan) Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Typical setup questions
All, I am curious what is used typically for the file system replication and how do you make sure that it is consistent. So for example when using large 3TB+ sata/NL-sas drives. Is is typical to replicate three times to get similar protection to raid 6? Also what is typically done to ensure that all replicas are in place and consistent? A cron that stats of ls's the file system from a single client? Thanks in advance for your responses. Matt ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] syslog options for gluster
are there any options to have glusterd use syslog? Would like the logs to go to a central server. Thanks Matt ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] folders with no permissions.
On 12/7/11 12:53 PM, Matt Weil wrote: Just some simply iozone testing failed due to folder permissions. The tmp folder is created by iozone. six nodes with stripe 6 and underlying EXT4 file system. The ext4 filesystems where not mounted with out the -o acl option. I meant with out the -o acl option. Any Ideas? in both cases it created a folder with no permissions. 188K d- 2 root root 24K 2011-12-06 11:28 tmp/ iozone$ rm -rf tmp rm: cannot remove directory `tmp': Permission denied test$ cd iozone.broke/ iozone.broke$ ls ./ ../ tmp/ iozone.broke$ ls -lash total 580K>> 288K drwxrwxrwx 6 root root 24K 2011-12-04 14:28 ../ 4.0K d- 2 root root 4.0K 2011-12-06 11:32 tmp/ each node has error trying to set permissions on that folder. [2011-12-06 12:23:36.593604] E [marker.c:2018:marker_setattr_cbk] 0-gluster1-marker: Operation not permitted occured during setattr of [2011-12-06 12:23:36.593669] I [server3_1-fops.c:1526:server_setattr_cbk] 0-gluster1-server: 433: SETATTR /test/iozone/tmp (-734804259) ==> -1 (Operation not permitted) [2011-12-06 12:23:36.593669] I [server3_1-fops.c:1526:server_setattr_cbk] 0-gluster1-server: 433: SETATTR /test/iozone/tmp (-734804259) ==> -1 (Operation not permitted) 188K d- 2 root root 24K 2011-12-06 11:28 tmp/ iozone$ chmod +rw tmp/ chmod: changing permissions of `tmp/': Operation not permitted iozone$ ls -lash ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] folders with no permissions.
Just some simply iozone testing failed due to folder permissions. The tmp folder is created by iozone. six nodes with stripe 6 and underlying EXT4 file system. The ext4 filesystems where not mounted with the -o acl option. Any Ideas? in both cases it created a folder with no permissions. 188K d- 2 rootroot 24K 2011-12-06 11:28 tmp/ iozone$ rm -rf tmp rm: cannot remove directory `tmp': Permission denied test$ cd iozone.broke/ iozone.broke$ ls ./ ../ tmp/ iozone.broke$ ls -lash total 580K>> 288K drwxrwxrwx 6 rootroot 24K 2011-12-04 14:28 ../ 4.0K d- 2 rootroot 4.0K 2011-12-06 11:32 tmp/ each node has error trying to set permissions on that folder. [2011-12-06 12:23:36.593604] E [marker.c:2018:marker_setattr_cbk] 0-gluster1-marker: Operation not permitted occured during setattr of [2011-12-06 12:23:36.593669] I [server3_1-fops.c:1526:server_setattr_cbk] 0-gluster1-server: 433: SETATTR /test/iozone/tmp (-734804259) ==> -1 (Operation not permitted) [2011-12-06 12:23:36.593669] I [server3_1-fops.c:1526:server_setattr_cbk] 0-gluster1-server: 433: SETATTR /test/iozone/tmp (-734804259) ==> -1 (Operation not permitted) 188K d- 2 rootroot 24K 2011-12-06 11:28 tmp/ iozone$ chmod +rw tmp/ chmod: changing permissions of `tmp/': Operation not permitted iozone$ ls -lash ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Transport endpoint is not connected
All, Is this normal? Can this be corrected? Thanks in advance for your responses. [2011-12-06 17:56:59.48153] I [glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC from uuid: fc5e6659-a90a-4e25-a3a7-11de9a7de81d [2011-12-06 17:56:59.48811] I [glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC from uuid: d1216f43-2ae6-42bd-a597-c0ab6a101d6b [2011-12-06 17:56:59.49073] I [glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC from uuid: 4bf94e6e-69ca-4d51-9a85-c1d98a95325d [2011-12-06 17:56:59.49137] I [glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC from uuid: 154cdbb2-6a53-449d-b6e3-bfd84091d90c [2011-12-06 17:56:59.49567] I [glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC from uuid: 4c9d68d6-d573-43d0-aec5-07173c1699d0 [2011-12-06 17:56:59.49803] I [glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: fc5e6659-a90a-4e25-a3a7-11de9a7de81d [2011-12-06 17:56:59.49850] I [glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: d1216f43-2ae6-42bd-a597-c0ab6a101d6b [2011-12-06 17:56:59.50228] I [glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 4bf94e6e-69ca-4d51-9a85-c1d98a95325d [2011-12-06 17:56:59.50285] I [glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 154cdbb2-6a53-449d-b6e3-bfd84091d90c [2011-12-06 17:56:59.50346] I [glusterd-rpc-ops.c:818:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 4c9d68d6-d573-43d0-aec5-07173c1699d0 [2011-12-06 17:56:59.50375] I [glusterd-op-sm.c:7250:glusterd_op_txn_complete] 0-glusterd: Cleared local lock [2011-12-06 17:56:59.52105] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:694) [2011-12-06 17:56:59.168257] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.11:730) [2011-12-06 17:56:59.168357] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.11:728) [2011-12-06 17:56:59.168441] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.11:726) [2011-12-06 17:56:59.168503] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.11:724) [2011-12-06 17:56:59.168591] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.11:722) [2011-12-06 17:56:59.168672] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.11:720) [2011-12-06 17:56:59.169287] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.15:730) [2011-12-06 17:56:59.169359] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.15:728) [2011-12-06 17:56:59.169398] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.13:730) [2011-12-06 17:56:59.169438] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.13:728) [2011-12-06 17:56:59.169476] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.15:726) [2011-12-06 17:56:59.169529] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.13:726) [2011-12-06 17:56:59.169581] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.30.13:724) ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
