Re: [Gluster-users] GlusterFS with CTDB setup - need of a event script?
I don't know about using ctdb, but using Corosync: the latest Filesystem resource-agent has support for GlusterFS. I've never used it, so it might not be what you're looking for. https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/Filesystem Dan From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Sam Wouters Sent: Friday, March 22, 2013 8:11 AM To: gluster-users@gluster.org Subject: [Gluster-users] GlusterFS with CTDB setup - need of a event script? Hi, I'm setting up a GlusterFS replicated cluster with CTDB and build-in glusterfs nfs export on CentOS 6.4 hosts following the documentation. Everything works fine, but I have a question about the setup: As there is currently no event script in ctdb for monitoring the gluster status, a node will not become unhealthy if for some reason the glusterd nfs export is unavailable. As such, the public ip stays at a node even when it's not serving the gluster volume or when it's not in sync. Does anybody already had a go at an ctdb event script for monitoring the gluster vol status on the host? regards, Sam ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] how well will this work
I also don’t think this is a rant. I, as well, have been following this list for a few years, and have been waiting for GlusterFS to stabilize for VM deployment. I hope this discussion helps the devs understand areas that people are waiting for. We have 2 SAN servers with Infiniband connections to a Blade Center. I would like all the KVM VMs hosted on the SAN with the ability to add more SAN servers in the future. – Currently Gluster allows this via NFS but I’ve read about performance issues. – So, right now, after 2 years of not deploying this gear (and running the VMs images on each blade), am looking for an expandable solution for the backend storage so I stop manually babying this network and install OpenNebula so I’m not the only person in our office who can manage our VM infrastructure. This does fit into the OP’s question because I would love to see GlusterFS work like this. Miles - As is right now GlusterFS is not what you want for backend VM storage. Question: “how well will this work” Answer: “horribly” Dan From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of John Mark Walker Sent: Thursday, December 27, 2012 12:39 PM To: Stephan von Krawczynski Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] how well will this work Stephan, I'm going to make this as simple as possible. Every message to this list should follow these rules: 1. be helpful 2. be constructive 3. be respectful I will not tolerate ranting that serves no purpose. If your message doesn't follow any of the rules above, then you shouldn't be posting it. This is your 2nd warning. -JM ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] about HA infrastructure for hypervisors
If you do decide to use 2 switchs: 1) for KVM Hosts use 2 nics, bridge them and run KVM on the bridge (usually br0), link each nic to a different switch 2) Interlink those switches. Dan -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Nicolas Sebrecht Sent: Wednesday, June 27, 2012 4:13 AM To: Gerald Brandt Cc: gluster-users Subject: [Gluster-users] Re: about HA infrastructure for hypervisors The 27/06/12, Gerald Brandt wrote: > Hi, > > If your switch breaks, you are done. Put each Gluster server on it's own > switch. Right. Handling switch failures isn't what I'm most worried about but I guess that I'll need to add a network link between KVM hypervisors, too. Thanks for this tip, though. > > ++ ++ > > ||--|| > > | KVM hypervisor |---+ +---| KVM hypervisor | > > || | | || > > ++ | | ++ > >| | > > +--+ +--+ > > |switch| |switch| > > +--+ +--+ > > | | | | > > +---+ | | | | +---+ > > | | | | | +-| | > > | Glusterfs 3.3 |--+ +--)| Glusterfs 3.3 | > > | server A| || server B| > > | |---+| | > > +---++---+ -- Nicolas Sebrecht ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] OT: possible to upgrade mellanox HBA firmware from ubuntu?
Glad to help! And, glad to have your modifications listed here for posterity! I couldn't recall if I had used the mft rpm from OFED. I was literally floundering around (ok not literally) trying to figure out how to upgrade the firmware and pissed off at the ridiculousness of what was required to do so. What I didn't bother to explain was how I did have the updated kernel installed, and had to run through the OFED install script to create the kernel-mft rpm to match it. With various trials runs to pick which RPMS to build - because not all successfully compiled *which was far outside of what I was trying to accomplish* - all done on a test system because I wasn't about to install compilers on a production server. Dan From: Harry Mangalam [mailto:harry.manga...@uci.edu] Sent: Wednesday, November 09, 2011 12:19 PM To: gluster-users@gluster.org Cc: Dan Cyr Subject: Re: [Gluster-users] OT: possible to upgrade mellanox HBA firmware from ubuntu? I can verify that the CentOS approach works as Dan described it with the caveats prefixed by ##hjm, using a testbed machine running a fresh install of CentOS6. Thanks to Dan for such a nice outline. hjm On Tuesday 08 November 2011 12:11:44 Dan Cyr wrote: > Harry, > > I had these same problems when I upgraded my firmware. > > Here's what I ended up doing (Centos6 or RHEL6 *not 6.1*): > > If you use the default kernel: 2.6.32_71.el6 (don't run yum > update), then you can use the kernel-mft RPM included with OFED > (kernel-mft-2.7.0-2.6.32_71.el6.x86_64.x86_64.rpm) > > yum -y install mstflint > wget "http://www.mellanox.com/downloads/ofed/MLNX_OFED_LINUX-1.5.3-1.0. 0-rhel6-x86_64.iso" mkdir -p /mnt/iso > mount -o loop MLNX_OFED_LINUX-1.5.3-1.0.0-rhel6-x86_64.iso /mnt/iso/ > yum -y > install /mnt/iso/RPMS/kernel-mft-2.7.0-2.6.32_71.el6.x86_64.x86_64.rpm > --nogpgcheck ##hjm: you need to install the mst app from the mft rpm that is ##hjm: distributed in the above iso image ##hjm: sudo rpm -i /mnt/iso/RPMS/mft-2.7.0-20.x86_64.rpm > mst start > ls /dev/mst/ # Verify the proper device is there (*_pciconf?) - if > there are none, this needs to be resolved before continuing. ##hjm: this worked fine: ls /dev/mst/ mt25204_pciconf0 mt25204_pci_cr0 > mkdir -p /usr/src/firmware > cd /usr/src/firmware > # My cards are SuperMicro UIO - replace the below commands > appropriately #wget > "ftp://ftp.supermicro.com/Firmware/InfiniBand/AOC-UINF-m2/aocuinfm > 2_20090609.zip" #unzip aocuinfm2_20090609.zip > # -allow_psid_change might not be required - check with you > hardware vendor - For me I found this link: > http://64.174.237.178/support/faqs/faq.cfm?faq=9803 #flint -d > /dev/mst/mt25418_pciconf0 -i aocuinfm2_20090609.bin -nofs > -allow_psid_change burn #reboot === ##hjm: found this latest firmware upgrade. <http://www.mellanox.com/content/pages.php?pg=custom_firmware_table> And once you get the appro package and download and unpack it, the VERY latest firmware needs to be created with mlxburn like this: $ cd ~hjm/fw-25204-rel-1_2_940 $ sudo mlxburn -fw ./fw-25204-rel.mlx -dev /dev/mst/mt25204_pci_cr0 -nofs ## and it seems to work: $ flint -d /dev/mst/mt25204_pciconf0 q Image type: Failsafe FW Version: 1.2.940 <---VERY latest fw. I.S. Version: 1 Device ID: 25204 Description: Node Port1 Sys image GUIDs: 00066a0098006e5f 00066a00a0006e5f 00066a0098006e5f Board ID: j (MT_023002) VSD: j PSID: MT_023002 === so with that, the > > Good Luck. > Dan > > From: gluster-users-boun...@gluster.org > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Harry > Mangalam Sent: Tuesday, November 08, 2011 11:45 AM > To: gluster-users@gluster.org > Subject: [Gluster-users] OT: possible to upgrade mellanox HBA > firmware from ubuntu? > > > Sorry for the OT, but this problem is preventing my further testing > of gluster and this groups seems like they might know.. > > I've been looking into this for a few days and have not run across > any success stories in upgrading firmware using Ubuntu (Mellanox > supports RH and Suse), but I've tried to install the MFT/MST > packages and it keeps erroring out for various reasons. > > If it's not possible (or very painful), I'll upgrade the card > firmware via a LiveCD/DVD of SciLinux or CentOS on another > machine. > > Other suggestions for upgrading the IB card firmware gratefully > accepted. > > Harry > > -- > > Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine > > [ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 > > MSTB Lat/L
Re: [Gluster-users] OT: possible to upgrade mellanox HBA firmware from ubuntu?
Harry, I had these same problems when I upgraded my firmware. Here's what I ended up doing (Centos6 or RHEL6 *not 6.1*): If you use the default kernel: 2.6.32_71.el6 (don't run yum update), then you can use the kernel-mft RPM included with OFED (kernel-mft-2.7.0-2.6.32_71.el6.x86_64.x86_64.rpm) yum -y install mstflint wget "http://www.mellanox.com/downloads/ofed/MLNX_OFED_LINUX-1.5.3-1.0.0-rhel6-x86_64.iso"; mkdir -p /mnt/iso mount -o loop MLNX_OFED_LINUX-1.5.3-1.0.0-rhel6-x86_64.iso /mnt/iso/ yum -y install /mnt/iso/RPMS/kernel-mft-2.7.0-2.6.32_71.el6.x86_64.x86_64.rpm --nogpgcheck mst start ls /dev/mst/ # Verify the proper device is there (*_pciconf?) - if there are none, this needs to be resolved before continuing. mkdir -p /usr/src/firmware cd /usr/src/firmware # My cards are SuperMicro UIO - replace the below commands appropriately #wget "ftp://ftp.supermicro.com/Firmware/InfiniBand/AOC-UINF-m2/aocuinfm2_20090609.zip"; #unzip aocuinfm2_20090609.zip # -allow_psid_change might not be required - check with you hardware vendor - For me I found this link: http://64.174.237.178/support/faqs/faq.cfm?faq=9803 #flint -d /dev/mst/mt25418_pciconf0 -i aocuinfm2_20090609.bin -nofs -allow_psid_change burn #reboot Good Luck. Dan From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Harry Mangalam Sent: Tuesday, November 08, 2011 11:45 AM To: gluster-users@gluster.org Subject: [Gluster-users] OT: possible to upgrade mellanox HBA firmware from ubuntu? Sorry for the OT, but this problem is preventing my further testing of gluster and this groups seems like they might know.. I've been looking into this for a few days and have not run across any success stories in upgrading firmware using Ubuntu (Mellanox supports RH and Suse), but I've tried to install the MFT/MST packages and it keeps erroring out for various reasons. If it's not possible (or very painful), I'll upgrade the card firmware via a LiveCD/DVD of SciLinux or CentOS on another machine. Other suggestions for upgrading the IB card firmware gratefully accepted. Harry -- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) -- This signature has been OCCUPIED! ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs over rdma ... not.
Try upgrading the firmware. I had this exact problem last week and the firmware update solved it. Dan From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Harry Mangalam Sent: Saturday, November 05, 2011 8:17 AM To: gluster-users@gluster.org; land...@scalableinformatics.com Subject: Re: [Gluster-users] glusterfs over rdma ... not. Thanks for the help, Joe, This looks like the problem: == $ ibv_devinfo Failed to get IB devices list: Function not implemented == There appears to be a problem.. Either i don';t have a valid IB driver or the device isn't recognized. lsmod says: $ lsmod |egrep '(ib|rdma)' rdma_cm 31030 0 ib_cm 40316 1 rdma_cm iw_cm 9385 1 rdma_cm ib_sa 22006 2 rdma_cm,ib_cm ib_addr 6766 1 rdma_cm ib_mthca 140934 0 ib_mad 41321 3 ib_cm,ib_sa,ib_mthca ib_core 64935 6 rdma_cm,ib_cm,iw_cm,ib_sa,ib_mthca,ib_mad so the card seems to be recognized and the driver is loaded (google says this card is suboptimal for such an infrastructure but I inherited this testbed so will have to deal with it for now). hjm On Friday 04 November 2011 18:32:36 Joe Landman wrote: > On 11/04/2011 09:06 PM, Harry Mangalam wrote: > > OK - finished some tests over tcp and ironed out a lot of > > problems. rdma is next; should be snap now > > > > [I must admit that this is my 1st foray into the land of IB, so > > some of the following may be obvious to a non-naive admin..] > > > except that while I can create and start the volume with rdma as transport: > First things first: > > > ibv_devinfo > > What does it report? > > > == > > > > root@pbs3:~ > > > > 622 $ gluster volume info glrdma > > > > Volume Name: glrdma > > > > Type: Distribute > > > > Status: Started > > > > Number of Bricks: 4 > > > > Transport-type: tcp,rdma > > > > Bricks: > > > > Brick1: pbs1:/data2 > > > > Brick2: pbs2:/data2 > > > > Brick3: pbs3:/data2 > > > > Brick4: pbs3:/data > > > > == > > > > I can't mount the damn thing. This seems to be a fairly frequent > > problem according to google.. Again, all servers and clients are > > ubuntu 10.04.3/64b, running self-compiled 3.3b1. > > try a > > showmount -e pbs2 > > Also, did you enable the auth bits as last time? -- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) -- This signature has been OCCUPIED! ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
