Re: [Gluster-users] Another transaction could be in progress
This mostly occurs when you run two gluster commands simultaneously. Gluster uses a lock on each peer to synchronize commands. Any command which would need to do operations on multiple peers, would first acquire this lock, and release it after doing the operation. If a command cannot acquire a lock because another command had the lock, it will fail with the above error message. It sometimes happens that a command could fail to release the lock on some peers. When this happens all further commands which need the lock will fail with the same error. In this case your only option is to restart glusterd on the peers which have the stale lock held. This will not cause any downtime as the brick processes are not affected by restarting glusterd. In your case, since you can run commands on other nodes, most likely you are running commands simultaneously or at least running a command before an old one finishes. ~kaushal On Tue, Mar 18, 2014 at 11:24 AM, Franco Broi franco.b...@iongeo.com wrote: What causes this error? And how do I get rid of it? [root@nas4 ~]# gluster vol status Another transaction could be in progress. Please try again after sometime. Looks normal on any other server. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Another transaction could be in progress
Restarted the glusterd daemons on all 4 servers, still the same. It only and always fails on the same server and it always works on the other servers. I had to reboot the server in question this morning, perhaps it's got itself in a funny state. Is the lock something that can be examined? And removed? On Tue, 2014-03-18 at 12:08 +0530, Kaushal M wrote: This mostly occurs when you run two gluster commands simultaneously. Gluster uses a lock on each peer to synchronize commands. Any command which would need to do operations on multiple peers, would first acquire this lock, and release it after doing the operation. If a command cannot acquire a lock because another command had the lock, it will fail with the above error message. It sometimes happens that a command could fail to release the lock on some peers. When this happens all further commands which need the lock will fail with the same error. In this case your only option is to restart glusterd on the peers which have the stale lock held. This will not cause any downtime as the brick processes are not affected by restarting glusterd. In your case, since you can run commands on other nodes, most likely you are running commands simultaneously or at least running a command before an old one finishes. ~kaushal On Tue, Mar 18, 2014 at 11:24 AM, Franco Broi franco.b...@iongeo.com wrote: What causes this error? And how do I get rid of it? [root@nas4 ~]# gluster vol status Another transaction could be in progress. Please try again after sometime. Looks normal on any other server. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] How to remove old NFS exports
Hi Some time ago during a testing phase I made a volume called test-volume and exported it via gluster's inbuilt NFS. I then destroyed the volume and made a new one called data but showmount -e still shows test-volume and not my new volume. I can mount the data volume from other servers, just not this particular one. Is there anyway to fix this without resorting to a reboot? Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] PLEASE READ ! We need your opinion. GSOC-2014 and the Gluster community
I had a discussion with some developers here in the office regarding this. We created a list of ideas which we thought could be suitable for student projects. I've added these to [1]. But I'm also putting them on here for more visibility. (I've tried to arrange the list in descending order of difficulty as I find it) . Glusterd services high availablity Glusterd should restart the processes it manages, bricks, nfs server, self-heal daemon quota daemon, whenever it detects they have died. . glusterfsiostat - Top like utility for glusterfs These are client side tools which will display stats from the io-stats translator. I'm not currently sure of the difference between the two. . ovirt gui for stats Have pretty graphs and tables in ovirt for the GlusterFS top and profile commands. . monitoring integrations - munin others. The more monitoring support we have for GlusterFS the better. . More compression algorithms for compression xlator The onwire compression translator should be extended to support more compression algorithms. Ideally it should be pluggable. . cinder glusterfs backup driver Write a driver for cinder, a part of openstack, to allow backup onto GlusterFS volumes . rsockets - sockets for rdma transport Coding for RDMA using the familiar socket api should lead to a more robust rdma transport . data import tool Create a tool which will allow already importing already existing data in the brick directories into the gluster volume. This is most likely going to be a special rebalance process. . rebalance improvements Improve rebalance preformance. . Improve the meta translator The meta xlator provides a /proc like interface to GlusterFS xlators. We could further improve this and make it a part of the standard volume graph. . geo-rep using rest-api This might be suitable for geo replication over WAN. Using rsync/ssh over WAN isn't too nice. . quota using underlying fs quota GlusterFS quota is currently maintained completely in GlusterFSs namespace using xattrs. We could make use of the quota capabilities of the underlying fs (XFS) for better performance. . snapshot pluggability Snapshot should be able to make use of snapshot support provided by btrfs for example. . compression at rest Lessons learnt while implementing encryption at rest can be used with the compression at rest. . file-level deduplication GlusterFS works on files. So why not have dedup at the level files as well. . composition xlator for small files Merge smallfiles into a designated large file using our own custom semantics. This can improve our small file performance. . multi master geo-rep Nothing much to say here. This has been discussed many times. Any comments on this list? ~kaushal [1] http://www.gluster.org/community/documentation/index.php/Projects On Tue, Mar 18, 2014 at 9:07 AM, Lalatendu Mohanty lmoha...@redhat.com wrote: On 03/13/2014 11:49 PM, John Mark Walker wrote: - Original Message - Welcome, Carlos. I think it's great that you're taking initiative here. +1 - I love enthusiastic fresh me^H^H^H^H^H^H^H^Hcommunity members! :) However, it's also important to set proper expectations for what a GSoC intern could reasonably be expected to achieve. I've seen some amazing stuff out of GSoC, but if we set the bar too high then we end up with incomplete code and the student doesn't learn much except frustration. This. The reason we haven't really participated in GSoC is not because we don't want to - it's because it's exceptionally difficult for a project of our scope, but that doesn't mean there aren't any possibilities. As an example, last year the Open Source Lab at OSU worked with a student to create an integration with Ganeti, which was mostly successful, and I think work has continued on that project. That's an example of a project with the right scope. IMO integration projects are ideal fits for GSoc. I can see some information in Trello back log i.e. under Ecosystem Integration. But not sure of their current status. I think we should again take look on these and see if something can be done through GSoc. 3) Accelerator node project. Some storage solutions out there offer an accelerator node, which is, in short, a, extra node with a lot of RAM, eventually fast disks (SSD), and that works like a proxy to the regular volumes. active chunks of files are moved there, logs (ZIL style) are recorded on fast media, among other things. There is NO active project for this, or trello entry, because it is something I started discussing with a few fellows just a couple of days ago. I thought of starting to play with RAM disks (tmpfs) as scratch disks, but, since we have an opportunity to do something more efficient, or at the very least start it, why not ? Looks like somebody has read the Isilon marketing materials. ;) A full production-level implementation of this, with cache consistency and so on,
Re: [Gluster-users] Another transaction could be in progress
The lock is an in-memory structure which isn't persisted. Restarting should reset the lock. You could possibly reset the lock by gdbing into the glusterd process. Since this is happening to you consistently, there is something else that is wrong. Could you please give more details on your cluster? And the glusterd logs of the misbehaving peer (if possible for all the peers). It would help in tracking it down. On Tue, Mar 18, 2014 at 12:24 PM, Franco Broi franco.b...@iongeo.com wrote: Restarted the glusterd daemons on all 4 servers, still the same. It only and always fails on the same server and it always works on the other servers. I had to reboot the server in question this morning, perhaps it's got itself in a funny state. Is the lock something that can be examined? And removed? On Tue, 2014-03-18 at 12:08 +0530, Kaushal M wrote: This mostly occurs when you run two gluster commands simultaneously. Gluster uses a lock on each peer to synchronize commands. Any command which would need to do operations on multiple peers, would first acquire this lock, and release it after doing the operation. If a command cannot acquire a lock because another command had the lock, it will fail with the above error message. It sometimes happens that a command could fail to release the lock on some peers. When this happens all further commands which need the lock will fail with the same error. In this case your only option is to restart glusterd on the peers which have the stale lock held. This will not cause any downtime as the brick processes are not affected by restarting glusterd. In your case, since you can run commands on other nodes, most likely you are running commands simultaneously or at least running a command before an old one finishes. ~kaushal On Tue, Mar 18, 2014 at 11:24 AM, Franco Broi franco.b...@iongeo.com wrote: What causes this error? And how do I get rid of it? [root@nas4 ~]# gluster vol status Another transaction could be in progress. Please try again after sometime. Looks normal on any other server. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] PLEASE READ ! We need your opinion. GSOC-2014 and the Gluster community
On 18/03/2014, at 7:05 AM, Kaushal M wrote: snip . cinder glusterfs backup driver Write a driver for cinder, a part of openstack, to allow backup onto GlusterFS volumes Is a backup version of the driver different from the existing Cinder driver? + Justin -- Open Source and Standards @ Red Hat twitter.com/realjustinclift ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] PLEASE READ ! We need your opinion. GSOC-2014 and the Gluster community
On 03/18/2014 01:12 PM, Justin Clift wrote: On 18/03/2014, at 7:05 AM, Kaushal M wrote: snip . cinder glusterfs backup driver Write a driver for cinder, a part of openstack, to allow backup onto GlusterFS volumes Is a backup version of the driver different from the existing Cinder driver? Yes, the backup driver backs up existing cinder volumes. The existing cinder driver facilitates attaching cinder volumes to nova instances. -Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Another transaction could be in progress
Sorry, didn't think to look in the log file, I can see I have bigger problems. Last time I saw this was because I had changed an IP address but this time all I did was reboot the server. I've checked all the files in vols and everything looks good. [2014-03-18 08:09:18.117040] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-03-18 08:09:18.117074] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-03-18 08:09:18.117087] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2014-03-18 08:09:18.117097] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2014-03-18 08:09:18.117107] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4 [2014-03-18 08:09:18.117117] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5 [2014-03-18 08:09:18.117128] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6 [2014-03-18 08:09:18.117138] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7 [2014-03-18 08:09:18.117148] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-8 [2014-03-18 08:09:18.117158] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-9 [2014-03-18 08:09:18.117168] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-10 [2014-03-18 08:09:18.117178] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-11 [2014-03-18 08:09:18.117196] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-12 [2014-03-18 08:09:18.117209] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-13 [2014-03-18 08:09:18.117219] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-14 [2014-03-18 08:09:18.117229] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-15 This is from another server [root@nas1 bricks]# gluster vol status Status of volume: data Gluster process PortOnline Pid -- Brick nas1-10g:/data1/gvol 49152 Y 17331 Brick nas2-10g:/data5/gvol 49160 Y 3933 Brick nas1-10g:/data2/gvol 49153 Y 17340 Brick nas2-10g:/data6/gvol 49161 Y 3942 Brick nas1-10g:/data3/gvol 49154 Y 17350 Brick nas2-10g:/data7/gvol 49162 Y 3951 Brick nas1-10g:/data4/gvol 49155 Y 17360 Brick nas2-10g:/data8/gvol 49163 Y 3960 Brick nas3-10g:/data9/gvol 49156 Y 10076 Brick nas3-10g:/data10/gvol 49157 Y 10085 Brick nas3-10g:/data11/gvol 49158 Y 10094 Brick nas3-10g:/data12/gvol 49159 Y 10108 Brick nas4-10g:/data13/gvol N/A N 8879 Brick nas4-10g:/data14/gvol N/A N 8884 Brick nas4-10g:/data15/gvol N/A N Brick nas4-10g:/data16/gvol N/A N 8892 NFS Server on localhost 2049Y 18725 NFS Server on nas3-10g 2049Y 11667 NFS Server on nas2-10g 2049Y 4980 NFS Server on nas4-10g N/A N N/A There are no active volume tasks Any ideas? On Tue, 2014-03-18 at 12:39 +0530, Kaushal M wrote: The lock is an in-memory structure which isn't persisted. Restarting should reset the lock. You could possibly reset the lock by gdbing into the glusterd process. Since this is happening to you consistently, there is something else that is wrong. Could you please give more details on your cluster? And the glusterd logs of the misbehaving peer (if possible for all the peers). It would help in tracking it down. On Tue, Mar 18, 2014 at 12:24 PM, Franco Broi franco.b...@iongeo.com wrote: Restarted the glusterd daemons on all 4 servers, still the same. It only and always fails on the same server and it always works on the other servers. I had to reboot the server in question this morning, perhaps it's got itself in a funny state. Is the lock something that can be examined? And removed? On Tue, 2014-03-18 at 12:08 +0530, Kaushal M wrote: This mostly occurs when you run two gluster commands simultaneously. Gluster uses a lock on each peer to synchronize commands. Any command which would need to
[Gluster-users] disk quotas
Hi, i am not sure since which version, but in 3.3.x and 3.4.x it's possible to set disk quotas, but still only hard limit. I also found documentations from red hat about setting soft limits, too. What's right? Gruß, Helge Systementwicklung Betrieb, USA Telefon: 17 82 helge.rid...@comdirect.de --- comdirect ausgezeichnet: Beste Bank (Euro, Mai 2013) --- comdirect bank AG Pascalkehre 15, 25451 Quickborn (HRB 4889 Pinneberg) www.comdirect.de Vorstand: Thorsten Reitmeyer (Vorsitzender), Holger Hohrein, Martina Palte Vorsitzender des Aufsichtsrats: Martin Zielke ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Another transaction could be in progress
On Tue, Mar 18, 2014 at 1:51 PM, Franco Broi franco.b...@iongeo.com wrote: Sorry, didn't think to look in the log file, I can see I have bigger problems. Last time I saw this was because I had changed an IP address but this time all I did was reboot the server. I've checked all the files in vols and everything looks good. [2014-03-18 08:09:18.117040] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-03-18 08:09:18.117074] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-03-18 08:09:18.117087] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2014-03-18 08:09:18.117097] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2014-03-18 08:09:18.117107] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4 [2014-03-18 08:09:18.117117] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5 [2014-03-18 08:09:18.117128] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6 [2014-03-18 08:09:18.117138] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7 [2014-03-18 08:09:18.117148] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-8 [2014-03-18 08:09:18.117158] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-9 [2014-03-18 08:09:18.117168] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-10 [2014-03-18 08:09:18.117178] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-11 [2014-03-18 08:09:18.117196] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-12 [2014-03-18 08:09:18.117209] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-13 [2014-03-18 08:09:18.117219] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-14 [2014-03-18 08:09:18.117229] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-15 These logs actually show no errors. They occur always during glusterd startup and need to be gotten rid of. Could you provide the logs from all the peers for the time you ran the failing command. This is from another server [root@nas1 bricks]# gluster vol status Status of volume: data Gluster process PortOnline Pid -- Brick nas1-10g:/data1/gvol 49152 Y 17331 Brick nas2-10g:/data5/gvol 49160 Y 3933 Brick nas1-10g:/data2/gvol 49153 Y 17340 Brick nas2-10g:/data6/gvol 49161 Y 3942 Brick nas1-10g:/data3/gvol 49154 Y 17350 Brick nas2-10g:/data7/gvol 49162 Y 3951 Brick nas1-10g:/data4/gvol 49155 Y 17360 Brick nas2-10g:/data8/gvol 49163 Y 3960 Brick nas3-10g:/data9/gvol 49156 Y 10076 Brick nas3-10g:/data10/gvol 49157 Y 10085 Brick nas3-10g:/data11/gvol 49158 Y 10094 Brick nas3-10g:/data12/gvol 49159 Y 10108 Brick nas4-10g:/data13/gvol N/A N 8879 Brick nas4-10g:/data14/gvol N/A N 8884 Brick nas4-10g:/data15/gvol N/A N Brick nas4-10g:/data16/gvol N/A N 8892 NFS Server on localhost 2049Y 18725 NFS Server on nas3-10g 2049Y 11667 NFS Server on nas2-10g 2049Y 4980 NFS Server on nas4-10g N/A N N/A There are no active volume tasks Any ideas? On Tue, 2014-03-18 at 12:39 +0530, Kaushal M wrote: The lock is an in-memory structure which isn't persisted. Restarting should reset the lock. You could possibly reset the lock by gdbing into the glusterd process. Since this is happening to you consistently, there is something else that is wrong. Could you please give more details on your cluster? And the glusterd logs of the misbehaving peer (if possible for all the peers). It would help in tracking it down. On Tue, Mar 18, 2014 at 12:24 PM, Franco Broi franco.b...@iongeo.com wrote: Restarted the glusterd daemons on all 4 servers, still the same. It only and always fails on the same server and it always works on the other servers. I had to reboot
[Gluster-users] gluster3.3.0 cause the ext4 system error then the linux(2.6.32-220.el6.x86_64) kernel panic...
jianglulu hello everyone,when i write the data in the server which used 2.6.32-220.el6.x86_64 kernel and gluster3.3.0 ,the server occur ext4_mb_complex_scan_group: 27 free blocks as per group info. But bitmap says 0 error and then kernel panic,so i want to ask some suggests for this: 1.is my disk has some problemes? 2.is the ext4 filesystem has some errors? 3.should i Upgrade system for linux or gluster? these are just some of the errors: EXT4-fs error (device sdf1): ext4_lookup: deleted inode referenced: 13787177 EXT4-fs (sdb1): ext4_check_descriptors: Block bitmap for group 5760 not in group (block 0)! EXT4-fs (sdb1): group descriptors corrupted! EXT4-fs (sdb1): ext4_check_descriptors: Block bitmap for group 5760 not in group (block 0)! EXT4-fs (sdb1): group descriptors corrupted! sorry for my poor EN~~___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Does QEMU offer High Availability
Hi Lucian, but Glusterfs is at least trying to replicate the QEMU image onto the other node ? That's the whole point of the replication isn’t it ? Thanks for the help, Dan -- Message: 1 Date: Mon, 17 Mar 2014 20:29:30 +0700 From: Daniel Baker i...@collisiondetection.biz To: gluster-users@gluster.org Subject: [Gluster-users] Does QEMU offer High Availability Message-ID: 5326f8ba.7010...@collisiondetection.biz Content-Type: text/plain; charset=ISO-8859-1 HI all, If I use QEMU in Glusterfs 3.4.2 can I achieve true High Availability. If I have two nodes that are clustering a QEMU image and one of them goes down will the other QEMU image take its place. Can I really achieve that ? Thanks, Dan -- Message: 2 Date: Mon, 17 Mar 2014 13:48:45 + From: Nux! n...@li.nux.ro To: gluster-users@gluster.org Subject: Re: [Gluster-users] Does QEMU offer High Availability Message-ID: c5ba49ac0de0faf0ab1257a7f1800...@li.nux.ro Content-Type: text/plain; charset=UTF-8; format=flowed On 17.03.2014 13:29, Daniel Baker wrote: HI all, If I use QEMU in Glusterfs 3.4.2 can I achieve true High Availability. If I have two nodes that are clustering a QEMU image and one of them goes down will the other QEMU image take its place. Can I really achieve that ? Dan, This is not really a discussion for the gluster lists. You should ask KVM people, but to answer you briefly: - No, KVM is just a hypervisor, it runs virtual machines and that's all it does. You need to run additional RedHat clustering stuff around it to achieve HA. HTH Lucian ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Does QEMU offer High Availability
Hi Daniel, I'm using KVM/Qemu with GlusterFS. The virtual machine images are stored on the Gluster filesystem, so there is always a copy of the virtual machine image on both cluster nodes. And you can even do a live migrate of a running guest from one node to the other. So if a host fails, all its guests will die with it, but you can restart them on the other node if you have enough resources. Regards, Paul Boven. On 03/18/2014 01:51 PM, Daniel Baker wrote: Hi Lucian, but Glusterfs is at least trying to replicate the QEMU image onto the other node ? That's the whole point of the replication isn’t it ? Thanks for the help, Dan -- Message: 1 Date: Mon, 17 Mar 2014 20:29:30 +0700 From: Daniel Baker i...@collisiondetection.biz To: gluster-users@gluster.org Subject: [Gluster-users] Does QEMU offer High Availability Message-ID: 5326f8ba.7010...@collisiondetection.biz Content-Type: text/plain; charset=ISO-8859-1 HI all, If I use QEMU in Glusterfs 3.4.2 can I achieve true High Availability. If I have two nodes that are clustering a QEMU image and one of them goes down will the other QEMU image take its place. Can I really achieve that ? Thanks, Dan -- Message: 2 Date: Mon, 17 Mar 2014 13:48:45 + From: Nux! n...@li.nux.ro To: gluster-users@gluster.org Subject: Re: [Gluster-users] Does QEMU offer High Availability Message-ID: c5ba49ac0de0faf0ab1257a7f1800...@li.nux.ro Content-Type: text/plain; charset=UTF-8; format=flowed On 17.03.2014 13:29, Daniel Baker wrote: HI all, If I use QEMU in Glusterfs 3.4.2 can I achieve true High Availability. If I have two nodes that are clustering a QEMU image and one of them goes down will the other QEMU image take its place. Can I really achieve that ? Dan, This is not really a discussion for the gluster lists. You should ask KVM people, but to answer you briefly: - No, KVM is just a hypervisor, it runs virtual machines and that's all it does. You need to run additional RedHat clustering stuff around it to achieve HA. HTH Lucian ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Paul Boven bo...@jive.nl +31 (0)521-596547 Unix/Linux/Networking specialist Joint Institute for VLBI in Europe - www.jive.nl VLBI - It's a fringe science ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Does QEMU offer High Availability
On 18.03.2014 12:51, Daniel Baker wrote: Hi Lucian, but Glusterfs is at least trying to replicate the QEMU image onto the other node ? That's the whole point of the replication isn’t it ? Dan, Of course. If so set, glusterfs will replicate your backing file, but should the VM go down on one VM, nothing will start it on the other one. You need to add more software to make this happen. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster3.3.0 cause the ext4 system error then the linux(2.6.32-220.el6.x86_64) kernel panic...
Ok... From previous posts, there still seems to be a kernel but, unaddressed by developers, which affects how ext4 works, and indeed makes its use with gluster unreliable. Specially on older versions of GLuster. Upgrading to 3.4.2 would be nice, if you can. If at all possible, use xfs instead of ext4. This is an official recommendation. prepare you filesystem with: mkfs.xfs -i size=512 /dev/yourdevicehere or any variation, BUT, keeping the inode size value at 512. Not quite sure what you mean by my disks have problems; You might mean the physical disk, the volume on your RAID (if any), your LVM volume, your gluster volume... BUT, before moving to that arena, you might want to do the steps above. If your system is a test system, upgrade to 3.4.2, and start using XFS. If it is a production system, you will have to find a way to do the transition. KR, Carlos On Tue, Mar 18, 2014 at 10:43 AM, 姜路 jll19851...@163.com wrote: jianglulu hello everyone,when i write the data in the server which used 2.6.32-220.el6.x86_64 kernel and gluster3.3.0 ,the server occur ext4_mb_complex_scan_group: 27 free blocks as per group info. But bitmap says 0 error and then kernel panic,so i want to ask some suggests for this: 1.is my disk has some problemes? 2.is the ext4 filesystem has some errors? 3.should i Upgrade system for linux or gluster? these are just some of the errors: EXT4-fs error (device sdf1): ext4_lookup: deleted inode referenced: 13787177 EXT4-fs (sdb1): ext4_check_descriptors: Block bitmap for group 5760 not in group (block 0)! EXT4-fs (sdb1): group descriptors corrupted! EXT4-fs (sdb1): ext4_check_descriptors: Block bitmap for group 5760 not in group (block 0)! EXT4-fs (sdb1): group descriptors corrupted! sorry for my poor EN~~ ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] disk quotas
- Original Message -
From: Helge Ridder helge.rid...@comdirect.de
To: gluster-users@gluster.org
Sent: Tuesday, March 18, 2014 4:30:32 AM
Subject: [Gluster-users] disk quotas
Hi,
i am not sure since which version, but in 3.3.x and 3.4.x it’s possible to
set disk quotas, but still only hard limit.
I also found documentations from red hat about setting soft limits, too.
What’s right?
There were some quota reworks that happened in 3.4, I would suggest running at
lest 3.4 if you are using quotas. But you are correct, there is a soft limit:
# gluster v quota help
Usage: volume quota VOLNAME {enable|disable|list [path ...]|remove path|
default-soft-limit percent} |
volume quota VOLNAME {limit-usage path size [percent]} |
volume quota VOLNAME {alert-time|soft-timeout|hard-timeout} {time}
IIRC the soft limit logs to the brick logs on the server and the mount log on
the client.
-b
Gruß,
Helge
Systementwicklung Betrieb, USA
Telefon: 17 82
helge.rid...@comdirect.de
---
comdirect ausgezeichnet: Beste Bank (Euro, Mai 2013)
---
comdirect bank AG
Pascalkehre 15, 25451 Quickborn
(HRB 4889 Pinneberg)
www.comdirect.de
Vorstand: Thorsten Reitmeyer (Vorsitzender), Holger Hohrein, Martina Palte
Vorsitzender des Aufsichtsrats: Martin Zielke
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster 3.4.2 geo-replication bandwith limiting
According to this BZ https://bugzilla.redhat.com/show_bug.cgi?id=764826 its possible to set rysnc bandwidth options for geo-replication on vers 3.2.1. Is this supported in 3.4.2? I just added the option referenced in the above link and the replication agreement status changed to faulty. Thanks, Steve ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] PLEASE READ ! We need your opinion. GSOC-2014 and the Gluster community
Thanks, Kaushal! The next steps are the following: - find out the deadlines for student proposals. If we're going with the Fedora umbrella, we need to get any student proposals submitted very soon - write out proposals on fedora project ideas page - submit proposals, along with proposed mentors - track down Vipul and see if he's still interested - I believe he said he was working with KP? KP - can you confirm this? We need to move quickly if we have any hope of getting projects submitted for this year. Please ask me for any help you need - if you don't know the Fedora folks involved, I can intro you. Thanks, everyone! -JM - Original Message - I had a discussion with some developers here in the office regarding this. We created a list of ideas which we thought could be suitable for student projects. I've added these to [1]. But I'm also putting them on here for more visibility. (I've tried to arrange the list in descending order of difficulty as I find it) . Glusterd services high availablity Glusterd should restart the processes it manages, bricks, nfs server, self-heal daemon quota daemon, whenever it detects they have died. . glusterfsiostat - Top like utility for glusterfs These are client side tools which will display stats from the io-stats translator. I'm not currently sure of the difference between the two. . ovirt gui for stats Have pretty graphs and tables in ovirt for the GlusterFS top and profile commands. . monitoring integrations - munin others. The more monitoring support we have for GlusterFS the better. . More compression algorithms for compression xlator The onwire compression translator should be extended to support more compression algorithms. Ideally it should be pluggable. . cinder glusterfs backup driver Write a driver for cinder, a part of openstack, to allow backup onto GlusterFS volumes . rsockets - sockets for rdma transport Coding for RDMA using the familiar socket api should lead to a more robust rdma transport . data import tool Create a tool which will allow already importing already existing data in the brick directories into the gluster volume. This is most likely going to be a special rebalance process. . rebalance improvements Improve rebalance preformance. . Improve the meta translator The meta xlator provides a /proc like interface to GlusterFS xlators. We could further improve this and make it a part of the standard volume graph. . geo-rep using rest-api This might be suitable for geo replication over WAN. Using rsync/ssh over WAN isn't too nice. . quota using underlying fs quota GlusterFS quota is currently maintained completely in GlusterFSs namespace using xattrs. We could make use of the quota capabilities of the underlying fs (XFS) for better performance. . snapshot pluggability Snapshot should be able to make use of snapshot support provided by btrfs for example. . compression at rest Lessons learnt while implementing encryption at rest can be used with the compression at rest. . file-level deduplication GlusterFS works on files. So why not have dedup at the level files as well. . composition xlator for small files Merge smallfiles into a designated large file using our own custom semantics. This can improve our small file performance. . multi master geo-rep Nothing much to say here. This has been discussed many times. Any comments on this list? ~kaushal [1] http://www.gluster.org/community/documentation/index.php/Projects On Tue, Mar 18, 2014 at 9:07 AM, Lalatendu Mohanty lmoha...@redhat.com wrote: On 03/13/2014 11:49 PM, John Mark Walker wrote: - Original Message - Welcome, Carlos. I think it's great that you're taking initiative here. +1 - I love enthusiastic fresh me^H^H^H^H^H^H^H^Hcommunity members! :) However, it's also important to set proper expectations for what a GSoC intern could reasonably be expected to achieve. I've seen some amazing stuff out of GSoC, but if we set the bar too high then we end up with incomplete code and the student doesn't learn much except frustration. This. The reason we haven't really participated in GSoC is not because we don't want to - it's because it's exceptionally difficult for a project of our scope, but that doesn't mean there aren't any possibilities. As an example, last year the Open Source Lab at OSU worked with a student to create an integration with Ganeti, which was mostly successful, and I think work has continued on that project. That's an example of a project with the right scope. IMO integration projects are ideal fits for GSoc. I can see some information in Trello back log i.e. under Ecosystem Integration. But not sure of their current status. I think we should again take look on these and see if something can be done through GSoc. 3) Accelerator node project.
[Gluster-users] Unable to access directories in wounded gluster setup
Hi, We are employing gluster to merge the disks from 3 servers into a common name-space. We just had a power outage, and when the power came back on, one of the servers had it's power supply damaged. While working on this problem, we tried to get the gluster area (/gdata) up with the the 2 working servers, so at least the files on them would be available while we repair the other server. This hasn't worked as expected. From past experience we expected to be able to see all the subdirectories with some files simply absent. What we see instead is that 4 of the top-level directories are unaccessible from client: ls /gdata ls: cannot access /gdata/test: Invalid argument ls: cannot access /gdata/projects: Invalid argument ls: cannot access /gdata/harvard-data2: Invalid argument ls: cannot access /gdata/temp_home: Invalid argument drwxr-xr-x 6 rootroot 152 Dec 5 12:27 harvard drwxr-xr-x 4 rootroot 74 Dec 5 12:27 harvard-archive ?? ? ? ? ?? harvard-data2 drwxr-xr-x 104 pierrel 8310 16K Dec 5 12:27 harvard-data3 drwx-- 2 rootroot 12 Dec 5 12:27 lost+found ?? ? ? ? ?? projects drwxrwxr-x 8 rootsoftware 4.1K Dec 5 12:27 software drwxr-xr-x 5 rootroot 4.1K Dec 5 12:27 src ?? ? ? ? ?? temp_home ?? ? ? ? ?? test We have tried - testing the communications between the bricks and between the client bricks (comms fine) - restarting the gluster daemons on the bricks - unmounting and remounting /gdata on the client - mounting the gluster area directly on one of the servers using mount -t glusterfs mseas-data:/gdata /gdata All give the same result. The status reported by gluster looks fine (given that 1 brick is down). The reports are included below. The log files from the mounting/ls tests are included after that. What should we do/look at next to solve/debug this problem? Thanks. Pat - [root@mseas-data glusterfs]# gluster --version glusterfs 3.3.1 built on Oct 11 2012 22:01:05 - [root@mseas-data bricks]# gluster volume info Volume Name: gdata Type: Distribute Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: gluster-0-0:/mseas-data-0-0 Brick2: gluster-0-1:/mseas-data-0-1 Brick3: gluster-data:/data - [root@mseas-data bricks]# gluster volume status Status of volume: gdata Gluster process PortOnline Pid -- Brick gluster-0-0:/mseas-data-0-0 24010 Y 11346 Brick gluster-data:/data24010 Y 28791 NFS Server on localhost 38467 Y 28797 NFS Server on gluster-0-0 38467 Y 11352 - [root@mseas-data bricks]# gluster peer status Number of Peers: 2 Hostname: gluster-0-1 Uuid: 978e0f76-6474-4203-8617-ed5ad7d29239 State: Peer in Cluster (Disconnected) Hostname: gluster-0-0 Uuid: 3f73f5cc-39d8-4d9a-b442-033cb074b247 State: Peer in Cluster (Connected) - = Log files from test mount from client: -- gluster-data: /var/log/glusterfs/bricks/data.log - [2014-03-18 10:43:57.149035] I [server.c:703:server_rpc_notify] 0-gdata-server: disconnecting connectionfrom compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 [2014-03-18 10:43:57.149073] I [server-helpers.c:741:server_connection_put] 0-gdata-server: Shutting down connection compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 [2014-03-18 10:43:57.149127] I [server-helpers.c:629:server_connection_destroy] 0-gdata-server: destroyed connection of compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 [2014-03-18 10:44:10.149539] I [server-handshake.c:571:server_setvolume] 0-gdata-server: accepted client from compute-3-0.local-15405-2014/03/18-10:44:06:129552-gdata-client-2-0 (version: 3.3.1) gluster-0-0: /var/log/glusterfs/bricks/data.log --- [2014-03-18 10:43:57.141122] I [server.c:703:server_rpc_notify] 0-gdata-server: disconnecting connectionfrom compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-0-0 [2014-03-18 10:43:57.141209] I [server-helpers.c:741:server_connection_put] 0-gdata-server: Shutting down connection
Re: [Gluster-users] Unable to access directories in wounded gluster setup
Hi Again, I was again going over my notes from the other times I've been in trouble. In particular, back in November I received the following reset procedure: -- If gluster-data is pingable from the other bricks, you could try detaching and retttaching it from gluster-0-0 or 0-1. 1) On gluster-0-0: `gluster peer detach gluster-data`, if that fails, `gluster peer detach gluster-data force` 2) On gluster-data: `rm -rf /var/lib/glusterd` `service glusterd restart` 3) Again on gluster-0-0: 'gluster peer probe gluster-data' -- Would such a reset be safe to try when only 2 out of 3 bricks are up? Would it be likely to help? Thanks. Pat Hi, We are employing gluster to merge the disks from 3 servers into a common name-space. We just had a power outage, and when the power came back on, one of the servers had it's power supply damaged. While working on this problem, we tried to get the gluster area (/gdata) up with the the 2 working servers, so at least the files on them would be available while we repair the other server. This hasn't worked as expected. From past experience we expected to be able to see all the subdirectories with some files simply absent. What we see instead is that 4 of the top-level directories are unaccessible from client: ls /gdata ls: cannot access /gdata/test: Invalid argument ls: cannot access /gdata/projects: Invalid argument ls: cannot access /gdata/harvard-data2: Invalid argument ls: cannot access /gdata/temp_home: Invalid argument drwxr-xr-x 6 rootroot 152 Dec 5 12:27 harvard drwxr-xr-x 4 rootroot 74 Dec 5 12:27 harvard-archive ?? ? ? ? ?? harvard-data2 drwxr-xr-x 104 pierrel 8310 16K Dec 5 12:27 harvard-data3 drwx-- 2 rootroot 12 Dec 5 12:27 lost+found ?? ? ? ? ?? projects drwxrwxr-x 8 rootsoftware 4.1K Dec 5 12:27 software drwxr-xr-x 5 rootroot 4.1K Dec 5 12:27 src ?? ? ? ? ?? temp_home ?? ? ? ? ?? test We have tried - testing the communications between the bricks and between the client bricks (comms fine) - restarting the gluster daemons on the bricks - unmounting and remounting /gdata on the client - mounting the gluster area directly on one of the servers using mount -t glusterfs mseas-data:/gdata /gdata All give the same result. The status reported by gluster looks fine (given that 1 brick is down). The reports are included below. The log files from the mounting/ls tests are included after that. What should we do/look at next to solve/debug this problem? Thanks. Pat - [root@mseas-data glusterfs]# gluster --version glusterfs 3.3.1 built on Oct 11 2012 22:01:05 - [root@mseas-data bricks]# gluster volume info Volume Name: gdata Type: Distribute Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: gluster-0-0:/mseas-data-0-0 Brick2: gluster-0-1:/mseas-data-0-1 Brick3: gluster-data:/data - [root@mseas-data bricks]# gluster volume status Status of volume: gdata Gluster process PortOnline Pid -- Brick gluster-0-0:/mseas-data-0-0 24010 Y 11346 Brick gluster-data:/data24010 Y 28791 NFS Server on localhost 38467 Y 28797 NFS Server on gluster-0-0 38467 Y 11352 - [root@mseas-data bricks]# gluster peer status Number of Peers: 2 Hostname: gluster-0-1 Uuid: 978e0f76-6474-4203-8617-ed5ad7d29239 State: Peer in Cluster (Disconnected) Hostname: gluster-0-0 Uuid: 3f73f5cc-39d8-4d9a-b442-033cb074b247 State: Peer in Cluster (Connected) - = Log files from test mount from client: -- gluster-data: /var/log/glusterfs/bricks/data.log - [2014-03-18 10:43:57.149035] I [server.c:703:server_rpc_notify] 0-gdata-server: disconnecting connectionfrom compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 [2014-03-18 10:43:57.149073] I [server-helpers.c:741:server_connection_put] 0-gdata-server: Shutting down connection compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 [2014-03-18 10:43:57.149127] I [server-helpers.c:629:server_connection_destroy] 0-gdata-server: destroyed
[Gluster-users] Different brick sizes in a volume
Hi, I've been playing around with a 2x2 distributed replicated setup with replicating group 1 having a different brick size than replicating group 2. I've been running into out of disk errors when the smaller replicating pair disks fill up. I know of the minimum free disk feature which should prevent this issue. My question is, are there features that allow gluster to smartly use different brick sizes so extra space on larger bricks do not go unused? Thanks -- Greg Waite System Administrator Computing Group, JILA University of Colorado-Boulder ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Different brick sizes in a volume
On Tue, Mar 18, 2014 at 1:42 PM, Greg Waite I've been playing around with a 2x2 distributed replicated setup with replicating group 1 having a different brick size than replicating group 2. I've been running into out of disk errors when the smaller replicating pair disks fill up. I know of the minimum free disk feature which should prevent this issue. My question is, are there features that allow gluster to smartly use different brick sizes so extra space on larger bricks do not go unused? It looks like different sized bricks will be a core feature in 3.6 (coming soon). Correct. In fact, a lot of the logic already exists and is even in the tree. http://review.gluster.org/#/c/3573/ The trick now is to get that logic integrated into the place where rebalance calculates the new layout. Until then, you could try running that script, but I should warn you that it hasn't been looked at for over a year so you should try it out on a small test volume first to make sure it's still doing the right thing(s). I'll be glad to help with that. Another thing you can do is to divide your larger bricks in two - or three, or whatever's necessary to even things out. This means more ports, more glusterfsd processes, quite possibly some performance loss as those contend with one another, but it's something you can do *right now* that's pretty easy and bullet-proof. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] PLEASE READ ! We need your opinion. GSOC-2014 and the Gluster community
On Tue, 2014-03-18 at 12:35 +0530, Kaushal M wrote: I had a discussion with some developers here in the office regarding this. We created a list of ideas which we thought could be suitable for student projects. I've added these to [1]. But I'm also putting them on here for more visibility. (I've tried to arrange the list in descending order of difficulty as I find it) . Glusterd services high availablity Glusterd should restart the processes it manages, bricks, nfs server, self-heal daemon quota daemon, whenever it detects they have died. It might make sense to think about the interplay between this and the systemd feature set... . glusterfsiostat - Top like utility for glusterfs These are client side tools which will display stats from the io-stats translator. I'm not currently sure of the difference between the two. . ovirt gui for stats Have pretty graphs and tables in ovirt for the GlusterFS top and profile commands. . monitoring integrations - munin others. The more monitoring support we have for GlusterFS the better. . More compression algorithms for compression xlator The onwire compression translator should be extended to support more compression algorithms. Ideally it should be pluggable. . cinder glusterfs backup driver Write a driver for cinder, a part of openstack, to allow backup onto GlusterFS volumes . rsockets - sockets for rdma transport Coding for RDMA using the familiar socket api should lead to a more robust rdma transport . data import tool Create a tool which will allow already importing already existing data in the brick directories into the gluster volume. This is most likely going to be a special rebalance process. . rebalance improvements Improve rebalance preformance. . Improve the meta translator The meta xlator provides a /proc like interface to GlusterFS xlators. We could further improve this and make it a part of the standard volume graph. . geo-rep using rest-api This might be suitable for geo replication over WAN. Using rsync/ssh over WAN isn't too nice. . quota using underlying fs quota GlusterFS quota is currently maintained completely in GlusterFSs namespace using xattrs. We could make use of the quota capabilities of the underlying fs (XFS) for better performance. . snapshot pluggability Snapshot should be able to make use of snapshot support provided by btrfs for example. This would be very useful :) . compression at rest Lessons learnt while implementing encryption at rest can be used with the compression at rest. . file-level deduplication GlusterFS works on files. So why not have dedup at the level files as well. . composition xlator for small files Merge smallfiles into a designated large file using our own custom semantics. This can improve our small file performance. . multi master geo-rep Nothing much to say here. This has been discussed many times. Any comments on this list? ~kaushal [1] http://www.gluster.org/community/documentation/index.php/Projects On Tue, Mar 18, 2014 at 9:07 AM, Lalatendu Mohanty lmoha...@redhat.com wrote: On 03/13/2014 11:49 PM, John Mark Walker wrote: - Original Message - Welcome, Carlos. I think it's great that you're taking initiative here. +1 - I love enthusiastic fresh me^H^H^H^H^H^H^H^Hcommunity members! :) However, it's also important to set proper expectations for what a GSoC intern could reasonably be expected to achieve. I've seen some amazing stuff out of GSoC, but if we set the bar too high then we end up with incomplete code and the student doesn't learn much except frustration. This. The reason we haven't really participated in GSoC is not because we don't want to - it's because it's exceptionally difficult for a project of our scope, but that doesn't mean there aren't any possibilities. As an example, last year the Open Source Lab at OSU worked with a student to create an integration with Ganeti, which was mostly successful, and I think work has continued on that project. That's an example of a project with the right scope. IMO integration projects are ideal fits for GSoc. I can see some information in Trello back log i.e. under Ecosystem Integration. But not sure of their current status. I think we should again take look on these and see if something can be done through GSoc. 3) Accelerator node project. Some storage solutions out there offer an accelerator node, which is, in short, a, extra node with a lot of RAM, eventually fast disks (SSD), and that works like a proxy to the regular volumes. active chunks of files are moved there, logs (ZIL style) are recorded on fast media, among other things. There is NO active project for this, or trello entry, because it is something I started discussing with a few fellows just a couple of days ago. I thought of starting to play with RAM
Re: [Gluster-users] Replicate Over VPN
Thanks guys, for your responses! I get the digest, so I'm going to cut/paste the juicier bits into one message... And a warning... if some of my comments suggest I really don't know what I'm doing - well, that could very well be right. I'm definitely down the learning curve a way - IT is not my real job or background. -- Forwarded message -- From: Alex Chekholko ch...@stanford.edu To: gluster-users@gluster.org Cc: Date: Mon, 17 Mar 2014 11:23:15 -0700 Subject: Re: [Gluster-users] Replicate Over VPN On 03/13/2014 04:50 PM, Brock Nanson wrote: ... 2) I've seen it suggested that the write function isn't considered complete until it's complete on all bricks in the volume. My write speeds would seem to confirm this. Yes, the write will return when all replicas are written. AKA synchronous replication. Usually replication means synchronous replication. OK, so the replication is bit by bit, real time across all the replicas. 'Synchronous' meaning 'common clock' in essence. Is this correct and is there any way to cache the data and allow it to trickle over the link in the background? You're talking about asynchronous replication. Which GlusterFS calls geo-replication. Understood... so this means one direction only in reality, at least until the nut of doing the replication in both directions can be cracked. 'Asynchronous' might be a bit of a misdirection though, because it would suggest (to me at least), communication in *both* directions, but not based on the same clock. ... Geo-replication would seem to be the ideal solution, except for the fact that it apparently only works in one direction (although it was evidently hoped it would be upgraded in 3.4.0 to go in both directions I understand). So if you allow replication to be delayed, and you allow writes on both sides, how would you deal with the same file simultaneously being written on both sides. Which would win in the end? This is the big question of course, and I think the answer requires more knowledge than I have relating to how the replication process occurs. In my unsophisticated way, I would assume that under the hood, gluster would sound something like this whenever a new file is written to Node A: 1) Samba wants to write a file, I'm awake! 2) Hey Node B, wake up, we're about to start writing some bits synchronously. File is called 'junk.txt'. 3) OK, we've both opened that file for writing... 3) Samba, start your transmission. 4) 'write, write, write', in Node A/B perfect harmony 5) Close that file and make sure the file listing is updated. This bit level understanding is something I don't have. At some point, the directory listing would be updated to show the new or updated file. When does that happen? Before or after the file is written? So to answer your question about which file would be win if simultaneously written, I need to understand whether simply having the file opened for writing is enough to take control of it. That is, can Node A tell Node B that junk.txt is going to be written, thus preventing Node B from accepting a local write request? If this is the case, then gluster would only need to send enough information from Node A to Node B to indicate the write was coming and that the file is off limits until further notice. The write could occur as fast as possible on the local node, and dribble across the VPN as fast as the link allows to the other. So #4 above would be 'write, write, write as fast as each node reasonably can, but not necessarily in harmony'. And if communication was broken during the process, the heal function would be called upon to sort it out when communication is restored. So are there any configuration tricks (write-behind, compression etc) that might help me out? Is there a way to fool geo-replication into working in both directions, recognizing my application isn't seeing serious read/write activity and some reasonable amount of risk is acceptable? You're basically talking about running rsyncs in both directions. How will you handle any file conflicts? Yes, I suppose in a way I am, but not based on a cron job... it would ideally be a full time synchronization, like gluster does, but without the requirement of perfect Synchronicity (wasn't that a Police album?). Assuming my kindergarten understanding above could be applied here, the file conflicts would presumably only exist if the VPN link went down, preventing the 'open the file for writing' command to be completed on both ends. If the link went down part way through a dribbling write to Node B, the healing process would presumably have a go at fixing the problem after the link is reinstated. If someone wrote to the remote copy during the outage, the typical heal issues would come into play. -- Alex Chekholko ch...@stanford.edu -- Forwarded message -- From: Alex Chekholko ch...@stanford.edu To: gluster-users@gluster.org Cc:
Re: [Gluster-users] Different brick sizes in a volume
On 03/18/2014 11:58 AM, Jeff Darcy wrote: On Tue, Mar 18, 2014 at 1:42 PM, Greg Waite I've been playing around with a 2x2 distributed replicated setup with replicating group 1 having a different brick size than replicating group 2. I've been running into out of disk errors when the smaller replicating pair disks fill up. I know of the minimum free disk feature which should prevent this issue. My question is, are there features that allow gluster to smartly use different brick sizes so extra space on larger bricks do not go unused? It looks like different sized bricks will be a core feature in 3.6 (coming soon). Correct. In fact, a lot of the logic already exists and is even in the tree. http://review.gluster.org/#/c/3573/ The trick now is to get that logic integrated into the place where rebalance calculates the new layout. Until then, you could try running that script, but I should warn you that it hasn't been looked at for over a year so you should try it out on a small test volume first to make sure it's still doing the right thing(s). I'll be glad to help with that. Another thing you can do is to divide your larger bricks in two - or three, or whatever's necessary to even things out. This means more ports, more glusterfsd processes, quite possibly some performance loss as those contend with one another, but it's something you can do *right now* that's pretty easy and bullet-proof. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users Thanks for the input! Sorry to go off topic, my gluster experience extends to only my cobbled together test bed, but how would I go about implementing that? If this is currently being tested ill be happy to take part in testing it! -- Greg Waite System Administrator Computing Group, JILA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Openstack Havana Gluster /var/lib/instances issues
I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster3.3.0 cause the ext4 system error then the linux(2.6.32-220.el6.x86_64) kernel panic...
Same here, Had ext4 in production with distributed replicated setup and after adding 3rd set of replicas (1x1) things went really really bad.. This was with 2.6.32.xxx kernel and Gluster 3.2.5 Had to rebuild with XFS and upgraded Gluster to 3.4.2 yesterday and all seems good so far.. v On Tue 18 Mar 2014 14:07:59, Carlos Capriotti wrote: Ok... From previous posts, there still seems to be a kernel but, unaddressed by developers, which affects how ext4 works, and indeed makes its use with gluster unreliable. Specially on older versions of GLuster. Upgrading to 3.4.2 would be nice, if you can. If at all possible, use xfs instead of ext4. This is an official recommendation. prepare you filesystem with: mkfs.xfs -i size=512 /dev/yourdevicehere or any variation, BUT, keeping the inode size value at 512. Not quite sure what you mean by my disks have problems; You might mean the physical disk, the volume on your RAID (if any), your LVM volume, your gluster volume... BUT, before moving to that arena, you might want to do the steps above. If your system is a test system, upgrade to 3.4.2, and start using XFS. If it is a production system, you will have to find a way to do the transition. KR, Carlos On Tue, Mar 18, 2014 at 10:43 AM, 姜路 jll19851...@163.com wrote: jianglulu hello everyone,when i write the data in the server which used 2.6.32-220.el6.x86_64 kernel and gluster3.3.0 ,the server occur ext4_mb_complex_scan_group: 27 free blocks as per group info. But bitmap says 0 error and then kernel panic,so i want to ask some suggests for this: 1.is my disk has some problemes? 2.is the ext4 filesystem has some errors? 3.should i Upgrade system for linux or gluster? these are just some of the errors: EXT4-fs error (device sdf1): ext4_lookup: deleted inode referenced: 13787177 EXT4-fs (sdb1): ext4_check_descriptors: Block bitmap for group 5760 not in group (block 0)! EXT4-fs (sdb1): group descriptors corrupted! EXT4-fs (sdb1): ext4_check_descriptors: Block bitmap for group 5760 not in group (block 0)! EXT4-fs (sdb1): group descriptors corrupted! sorry for my poor EN~~ ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Regards Viktor Villafuerte Optus Internet Engineering t: 02 808-25265 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
It might be more useful to look at your hypervisor log and/or client log. On March 18, 2014 3:04:48 PM PDT, Aydelott, Ryan M. ry...@mcs.anl.gov wrote: I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] PLEASE READ ! We need your opinion. GSOC-2014 and the Gluster community
On Tue, Mar 18, 2014 at 9:08 PM, John Mark Walker johnm...@gluster.org wrote: Thanks, Kaushal! The next steps are the following: - find out the deadlines for student proposals. If we're going with the Fedora umbrella, we need to get any student proposals submitted very soon The deadline is this Friday, March 21. We've got 3 days more. - write out proposals on fedora project ideas page This is the ideas page - http://fedoraproject.org/wiki/Summer_coding_ideas_for_2014 I'm not sure if we should be adding all the ideas I listed earlier there as some are pretty easy to finish whereas others will take longer than the GSOC timeline to finish. Also, the ideas need to be in a specific format. If I could get help doing this, we could add all. - submit proposals, along with proposed mentors - track down Vipul and see if he's still interested - I believe he said he was working with KP? KP - can you confirm this? KP is on vacation and will be back in April, but I can confirm on his behalf as he had talked to me about this. Since KP isn't around, Vijay said he'll be reaching out to Vipul. We need to move quickly if we have any hope of getting projects submitted for this year. Please ask me for any help you need - if you don't know the Fedora folks involved, I can intro you. Thanks, everyone! -JM - Original Message - I had a discussion with some developers here in the office regarding this. We created a list of ideas which we thought could be suitable for student projects. I've added these to [1]. But I'm also putting them on here for more visibility. (I've tried to arrange the list in descending order of difficulty as I find it) . Glusterd services high availablity Glusterd should restart the processes it manages, bricks, nfs server, self-heal daemon quota daemon, whenever it detects they have died. . glusterfsiostat - Top like utility for glusterfs These are client side tools which will display stats from the io-stats translator. I'm not currently sure of the difference between the two. . ovirt gui for stats Have pretty graphs and tables in ovirt for the GlusterFS top and profile commands. . monitoring integrations - munin others. The more monitoring support we have for GlusterFS the better. . More compression algorithms for compression xlator The onwire compression translator should be extended to support more compression algorithms. Ideally it should be pluggable. . cinder glusterfs backup driver Write a driver for cinder, a part of openstack, to allow backup onto GlusterFS volumes . rsockets - sockets for rdma transport Coding for RDMA using the familiar socket api should lead to a more robust rdma transport . data import tool Create a tool which will allow already importing already existing data in the brick directories into the gluster volume. This is most likely going to be a special rebalance process. . rebalance improvements Improve rebalance preformance. . Improve the meta translator The meta xlator provides a /proc like interface to GlusterFS xlators. We could further improve this and make it a part of the standard volume graph. . geo-rep using rest-api This might be suitable for geo replication over WAN. Using rsync/ssh over WAN isn't too nice. . quota using underlying fs quota GlusterFS quota is currently maintained completely in GlusterFSs namespace using xattrs. We could make use of the quota capabilities of the underlying fs (XFS) for better performance. . snapshot pluggability Snapshot should be able to make use of snapshot support provided by btrfs for example. . compression at rest Lessons learnt while implementing encryption at rest can be used with the compression at rest. . file-level deduplication GlusterFS works on files. So why not have dedup at the level files as well. . composition xlator for small files Merge smallfiles into a designated large file using our own custom semantics. This can improve our small file performance. . multi master geo-rep Nothing much to say here. This has been discussed many times. Any comments on this list? ~kaushal [1] http://www.gluster.org/community/documentation/index.php/Projects On Tue, Mar 18, 2014 at 9:07 AM, Lalatendu Mohanty lmoha...@redhat.com wrote: On 03/13/2014 11:49 PM, John Mark Walker wrote: - Original Message - Welcome, Carlos. I think it's great that you're taking initiative here. +1 - I love enthusiastic fresh me^H^H^H^H^H^H^H^Hcommunity members! :) However, it's also important to set proper expectations for what a GSoC intern could reasonably be expected to achieve. I've seen some amazing stuff out of GSoC, but if we set the bar too high then we end up with incomplete code and the student doesn't learn much except frustration. This. The reason we haven't really participated in GSoC is not because we don't want to - it's
[Gluster-users] gluster bug 991084
Hi All, I seem to have hit glusterbug 991084 https://bugzilla.redhat.com/show_bug.cgi?id=991084 I try solution: (vol=machines; brick=/export/sdb1/brick; setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/$vol/info | cut -d= -f2 | sed 's/-//g') $brick) grep: /var/lib/glusterd/vols/machines/info: No such file or directory And then this : root@cluster1:/home/admincit# gluster volume start gv0 force volume start: gv0: failed: Volume id mismatch for brick 192.168.100.170:/export/sdb1/brick. Expected volume id e9e31a9d-e194-4cbb-851c-a837ebc753d0, volume id ---- found This has come about because I replaced the HD the brick was on. Is there another solution I can try or does anyone know what I have to tweak in the above commands to get this volume id thing sorted ? Thanks for the help, Dan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
From the pastebin it seems like its getting a ton of I/O errors. It would be good to look at gluster logs (/var/log/glusterfs) and whether that carries any errors, since I/O error is a generic error that glusterfs can return for many reasons. On Wed, Mar 19, 2014 at 3:34 AM, Aydelott, Ryan M. ry...@mcs.anl.govwrote: I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
