Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Though remove-brick is not an usual act we would do for Gluster volume, this has consistently failed ending in corrupted gluster volume after Sharding has been turned on. For bug1387878, it's very similar to what i had encountered in ESXi world. Add-brick, would run successful, but virtual-machine files would crash after rebalance in one of my environments. That did not happen in my another environment under same version (3.7.16). Difference between 2 was one is changing from Replicate to Distributed-Replicate, but they are still configured with only 2-replicas. i will have to test 3.8.* with Ganesha to see how it goes. On Mon, Nov 14, 2016 at 8:29 AM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 1016-11-14 17:01 GMT+01:00 Vijay Bellur : > > Accessing sharded data after disabling sharding is something that we > > did not visualize as a valid use case at any point in time. Also, you > > could access the contents by enabling sharding again. Given these > > factors I think this particular problem has not been prioritized by > > us. > > That's not true. > If you have VMs running on a sharded volume and you disable sharding, > with the VM still running, everything crash and could lead to data loss, > as VM > will be unable to find their filesystem and so on, qemu currupts the > image and so on. > > If I write to a file that was shareded, (in example a log file), now > when you disable the shard, > the application would write the existing file (the one that was the > first shard). > If you reenable sharding, you lost some data > > Example: > > 128MB file. shard set to 64MB. You have 2 chunks: shard1+shard2 > > Now you are writing to the file: > > > > > > > + are placed on shard1, + are placed on shard2 > > If you disable the shard and write some extra data, , then > would be placed after in shard1 (growing more than 64MB) > and not on shard3 > > If you re-enable shard, is lost, as gluster would expect it as > shard3. and I think gluster will read only the first 64MB from shard1. > If gluster read the whole file, you'll get something like this: > > > > > > > > in a text file this is bad, in a VM image, this mean data > loss/corruption almost impossible to fix. > > > > As with many other projects, we are in a stage today where the number > > of users and testers far outweigh the number of developers > > contributing code. With this state it becomes hard to prioritize > > problems from a long todo list for developers. If valuable community > > members like you feel strongly about a bug or feature that need > > attention of developers, please call such issues out on the mailing > > list. We will be more than happy to help. > > That's why i've asked for less feature and more stability. > If you have to prioritize, please choose all bugs that could lead to > data corruption or similiar. > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Features and stability are not mutually exclusive. Sometimes instability is cured by adding a feature. Fixing a bug is not something that's solved better by having more developers work on it. Sometimes fixing one bug exposed a problem elsewhere. Using free open source community projects with your own hardware and system design weights the responsibility to test more heavily on yourself. If that's not a risk you can afford, you might consider contracting with a 3rd party which has "certified" installation parameters. IMHO. On November 14, 2016 8:29:00 AM PST, Gandalf Corvotempesta wrote: >1016-11-14 17:01 GMT+01:00 Vijay Bellur : >> Accessing sharded data after disabling sharding is something that we >> did not visualize as a valid use case at any point in time. Also, you >> could access the contents by enabling sharding again. Given these >> factors I think this particular problem has not been prioritized by >> us. > >That's not true. >If you have VMs running on a sharded volume and you disable sharding, >with the VM still running, everything crash and could lead to data >loss, as VM >will be unable to find their filesystem and so on, qemu currupts the >image and so on. > >If I write to a file that was shareded, (in example a log file), now >when you disable the shard, >the application would write the existing file (the one that was the >first shard). >If you reenable sharding, you lost some data > >Example: > >128MB file. shard set to 64MB. You have 2 chunks: shard1+shard2 > >Now you are writing to the file: > > > > > > >+ are placed on shard1, + are placed on shard2 > >If you disable the shard and write some extra data, , then >would be placed after in shard1 (growing more than 64MB) >and not on shard3 > >If you re-enable shard, is lost, as gluster would expect it as >shard3. and I think gluster will read only the first 64MB from shard1. >If gluster read the whole file, you'll get something like this: > > > > > > > >in a text file this is bad, in a VM image, this mean data >loss/corruption almost impossible to fix. > > >> As with many other projects, we are in a stage today where the number >> of users and testers far outweigh the number of developers >> contributing code. With this state it becomes hard to prioritize >> problems from a long todo list for developers. If valuable community >> members like you feel strongly about a bug or feature that need >> attention of developers, please call such issues out on the mailing >> list. We will be more than happy to help. > >That's why i've asked for less feature and more stability. >If you have to prioritize, please choose all bugs that could lead to >data corruption or similiar. >___ >Gluster-users mailing list >Gluster-users@gluster.org >http://www.gluster.org/mailman/listinfo/gluster-users -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
1016-11-14 17:01 GMT+01:00 Vijay Bellur : > Accessing sharded data after disabling sharding is something that we > did not visualize as a valid use case at any point in time. Also, you > could access the contents by enabling sharding again. Given these > factors I think this particular problem has not been prioritized by > us. That's not true. If you have VMs running on a sharded volume and you disable sharding, with the VM still running, everything crash and could lead to data loss, as VM will be unable to find their filesystem and so on, qemu currupts the image and so on. If I write to a file that was shareded, (in example a log file), now when you disable the shard, the application would write the existing file (the one that was the first shard). If you reenable sharding, you lost some data Example: 128MB file. shard set to 64MB. You have 2 chunks: shard1+shard2 Now you are writing to the file: + are placed on shard1, + are placed on shard2 If you disable the shard and write some extra data, , then would be placed after in shard1 (growing more than 64MB) and not on shard3 If you re-enable shard, is lost, as gluster would expect it as shard3. and I think gluster will read only the first 64MB from shard1. If gluster read the whole file, you'll get something like this: in a text file this is bad, in a VM image, this mean data loss/corruption almost impossible to fix. > As with many other projects, we are in a stage today where the number > of users and testers far outweigh the number of developers > contributing code. With this state it becomes hard to prioritize > problems from a long todo list for developers. If valuable community > members like you feel strongly about a bug or feature that need > attention of developers, please call such issues out on the mailing > list. We will be more than happy to help. That's why i've asked for less feature and more stability. If you have to prioritize, please choose all bugs that could lead to data corruption or similiar. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
2016-11-14 16:55 GMT+01:00 Krutika Dhananjay : > The only way to fix it is to have sharding be part of the graph *even* if > disabled, > except that in this case, its job should be confined to aggregating the > already > sharded files during reads but NOT shard new files that are created, since > it is > supposed to "act" disabled. This is a slightly bigger change and this is why > I had > suggested the workaround at > https://bugzilla.redhat.com/show_bug.cgi?id=1355846#c1 > back then. Why not keeping the shard xlator always on but set on a very high value so that shard is never happening? Something at 100GB (just as proof of concept) > FWIW, the documentation [1] does explain how to disable sharding the right > way and has been in existence ever since sharding was first released in > 3.7.0. > > [1] - > http://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/ Ok but: 1) that's for 3.7 *beta1*. I'm using 3.8 2) "advisable" doesn't mean "you have to". It's an advice, not the only way to disable a feature 3) i'm talking about a confirm to add in the cli, nothing strange. all software ask for a confirm when bad things could happens. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Mon, Nov 14, 2016 at 8:54 AM, Niels de Vos wrote: > On Mon, Nov 14, 2016 at 04:50:44PM +0530, Pranith Kumar Karampuri wrote: > > On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta < > > gandalf.corvotempe...@gmail.com> wrote: > > > > > 2016-11-14 11:50 GMT+01:00 Pranith Kumar Karampuri < > pkara...@redhat.com>: > > > > To make gluster stable for VM images we had to add all these new > features > > > > and then fix all the bugs Lindsay/Kevin reported. We just fixed a > > > corruption > > > > issue that can happen with replace-brick which will be available in > 3.9.0 > > > > and 3.8.6. The only 2 other known issues that can lead to > corruptions are > > > > add-brick and the bug you filed Gandalf. Krutika just 5 minutes back > saw > > > > something that could possibly lead to the corruption for the > add-brick > > > bug. > > > > Is that really the Root cause? We are not sure yet, we need more > time. > > > > Without Lindsay/Kevin/David Gossage's support this workload would > have > > > been > > > > in much worse condition. These bugs are not easy to re-create thus > not > > > easy > > > > to fix. At least that has been Krutika's experience. > > > > > > Ok, but this changes should be placed in a "test" version and not > > > marked as stable. > > > I don't see any development release, only stable releases here. > > > Do you want all features ? Try the "beta/rc/unstable/alpha/dev" > version. > > > Do you want the stable version without known bugs but slow on VMs > > > workload? Use the "-stable" version. > > > > > > If you relase as stable, users tend to upgrade their cluster and use > > > the newer feature (that you are marking as stable). > > > What If I upgrade a production cluster to a stable version and try to > > > add-brick that lead to data corruption ? > > > I have to restore terabytes worth of data? Gluster is made for > > > scale-out, what I my cluster was made with 500TB of VMs ? > > > Try to restore 500TB from a backup > > > > > > This is unacceptable. add-brick/replace-brick should be common "daily" > > > operations. You should heavy check these for regression or bug. > > > > > > > This is a very good point. Adding other maintainers. > > Obviously this is unacceptible for versions that have sharding as a > functional (not experimental) feature. All supported features are > expected to function without major problems (like corruption) for all > standard Gluster operations. Add-brick/replace-brick are surely such > Gluster operations. > > Of course it is possible that this does not always happen, and our tests > did not catch the problem. In that case, we really need to have a bug > report with all the details, and preferably a script that can be used to > reproduce and detect the failure. > I believe this bug relates to this particular issue raised in this email chain. https://bugzilla.redhat.com/show_bug.cgi?id=1387878 Kevin found bug, and Lindsay filed report after she was able to recreate it. > > FWIW sharding has several open bugs (like any other component), but it > is not immediately clear to me if the problem reported in this email is > in Bugzilla yet. These are the bugs that are expected to get fixed in > upcoming minor releases: > https://bugzilla.redhat.com/buglist.cgi?component= > sharding&f1=bug_status&f2=version&o1=notequals&o2= > notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline > > HTH, > Niels > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Mon, Nov 14, 2016 at 10:38 AM, Gandalf Corvotempesta wrote: > 2016-11-14 15:54 GMT+01:00 Niels de Vos : >> Obviously this is unacceptible for versions that have sharding as a >> functional (not experimental) feature. All supported features are >> expected to function without major problems (like corruption) for all >> standard Gluster operations. Add-brick/replace-brick are surely such >> Gluster operations. > > Is sharding an experimental feature even in 3.8 ? > Because in 3.8 announcement, it's declared stable: > http://blog.gluster.org/2016/06/glusterfs-3-8-released/ > "Sharding is now stable for VM image storage. " > sharding was an experimental feature in 3.7. Based on the feedback that we received in testing, we called it out as stable in 3.8. The add-brick related issue is something that none of us encountered in testing and we will determine how we can avoid missing such problems in the future. >> FWIW sharding has several open bugs (like any other component), but it >> is not immediately clear to me if the problem reported in this email is >> in Bugzilla yet. These are the bugs that are expected to get fixed in >> upcoming minor releases: >> >> https://bugzilla.redhat.com/buglist.cgi?component=sharding&f1=bug_status&f2=version&o1=notequals&o2=notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline > > My issue with sharding was reported in bugzilla on 2016-07-12 > 4 months for a IMHO, critical bug. > > If you disable sharding on a sharded volume with existing shared data, > you corrupt every existing file. Accessing sharded data after disabling sharding is something that we did not visualize as a valid use case at any point in time. Also, you could access the contents by enabling sharding again. Given these factors I think this particular problem has not been prioritized by us. As with many other projects, we are in a stage today where the number of users and testers far outweigh the number of developers contributing code. With this state it becomes hard to prioritize problems from a long todo list for developers. If valuable community members like you feel strongly about a bug or feature that need attention of developers, please call such issues out on the mailing list. We will be more than happy to help. Having explained the developer perspective, I do apologize for any inconvenience you might have encountered from this particular bug. Thanks! Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Yes. I apologise for the delay. Disabling sharding would knock the translator itself off the client stack, and being that sharding is the actual (and the only) translator that has the knowledge of how to interpret sharded files, and how to aggregate them, removing the translator from the stack will make all shards start to appear like isolated files with no way to interpret the correlation between the individual pieces. The only way to fix it is to have sharding be part of the graph *even* if disabled, except that in this case, its job should be confined to aggregating the already sharded files during reads but NOT shard new files that are created, since it is supposed to "act" disabled. This is a slightly bigger change and this is why I had suggested the workaround at https://bugzilla.redhat.com/show_bug.cgi?id=1355846#c1 back then. FWIW, the documentation [1] does explain how to disable sharding the right way and has been in existence ever since sharding was first released in 3.7.0. [1] - http://staged-gluster-docs.readthedocs.io/en/release3.7. 0beta1/Features/shard/ -Krutika On Mon, Nov 14, 2016 at 9:08 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-11-14 15:54 GMT+01:00 Niels de Vos : > > Obviously this is unacceptible for versions that have sharding as a > > functional (not experimental) feature. All supported features are > > expected to function without major problems (like corruption) for all > > standard Gluster operations. Add-brick/replace-brick are surely such > > Gluster operations. > > Is sharding an experimental feature even in 3.8 ? > Because in 3.8 announcement, it's declared stable: > http://blog.gluster.org/2016/06/glusterfs-3-8-released/ > "Sharding is now stable for VM image storage. " > > > FWIW sharding has several open bugs (like any other component), but it > > is not immediately clear to me if the problem reported in this email is > > in Bugzilla yet. These are the bugs that are expected to get fixed in > > upcoming minor releases: > > https://bugzilla.redhat.com/buglist.cgi?component= > sharding&f1=bug_status&f2=version&o1=notequals&o2= > notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline > > My issue with sharding was reported in bugzilla on 2016-07-12 > 4 months for a IMHO, critical bug. > > If you disable sharding on a sharded volume with existing shared data, > you corrupt every existing file. > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Mon, Nov 14, 2016 at 8:24 PM, Niels de Vos wrote: > On Mon, Nov 14, 2016 at 04:50:44PM +0530, Pranith Kumar Karampuri wrote: > > On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta < > > gandalf.corvotempe...@gmail.com> wrote: > > > > > 2016-11-14 11:50 GMT+01:00 Pranith Kumar Karampuri < > pkara...@redhat.com>: > > > > To make gluster stable for VM images we had to add all these new > features > > > > and then fix all the bugs Lindsay/Kevin reported. We just fixed a > > > corruption > > > > issue that can happen with replace-brick which will be available in > 3.9.0 > > > > and 3.8.6. The only 2 other known issues that can lead to > corruptions are > > > > add-brick and the bug you filed Gandalf. Krutika just 5 minutes back > saw > > > > something that could possibly lead to the corruption for the > add-brick > > > bug. > > > > Is that really the Root cause? We are not sure yet, we need more > time. > > > > Without Lindsay/Kevin/David Gossage's support this workload would > have > > > been > > > > in much worse condition. These bugs are not easy to re-create thus > not > > > easy > > > > to fix. At least that has been Krutika's experience. > > > > > > Ok, but this changes should be placed in a "test" version and not > > > marked as stable. > > > I don't see any development release, only stable releases here. > > > Do you want all features ? Try the "beta/rc/unstable/alpha/dev" > version. > > > Do you want the stable version without known bugs but slow on VMs > > > workload? Use the "-stable" version. > > > > > > If you relase as stable, users tend to upgrade their cluster and use > > > the newer feature (that you are marking as stable). > > > What If I upgrade a production cluster to a stable version and try to > > > add-brick that lead to data corruption ? > > > I have to restore terabytes worth of data? Gluster is made for > > > scale-out, what I my cluster was made with 500TB of VMs ? > > > Try to restore 500TB from a backup > > > > > > This is unacceptable. add-brick/replace-brick should be common "daily" > > > operations. You should heavy check these for regression or bug. > > > > > > > This is a very good point. Adding other maintainers. > I think Pranith's intention here was to bring to other maintainers' attention the point about development releases vs stable releases although his inline comment may have been a bit out-of-place (I was part of the discussion that took place before this reply of his, in office today, hence taking the liberty to clarify). -Krutika > Obviously this is unacceptible for versions that have sharding as a > functional (not experimental) feature. All supported features are > expected to function without major problems (like corruption) for all > standard Gluster operations. Add-brick/replace-brick are surely such > Gluster operations. > > Of course it is possible that this does not always happen, and our tests > did not catch the problem. In that case, we really need to have a bug > report with all the details, and preferably a script that can be used to > reproduce and detect the failure. > > FWIW sharding has several open bugs (like any other component), but it > is not immediately clear to me if the problem reported in this email is > in Bugzilla yet. These are the bugs that are expected to get fixed in > upcoming minor releases: > https://bugzilla.redhat.com/buglist.cgi?component= > sharding&f1=bug_status&f2=version&o1=notequals&o2= > notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline > > HTH, > Niels > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
2016-11-14 15:54 GMT+01:00 Niels de Vos : > Obviously this is unacceptible for versions that have sharding as a > functional (not experimental) feature. All supported features are > expected to function without major problems (like corruption) for all > standard Gluster operations. Add-brick/replace-brick are surely such > Gluster operations. Is sharding an experimental feature even in 3.8 ? Because in 3.8 announcement, it's declared stable: http://blog.gluster.org/2016/06/glusterfs-3-8-released/ "Sharding is now stable for VM image storage. " > FWIW sharding has several open bugs (like any other component), but it > is not immediately clear to me if the problem reported in this email is > in Bugzilla yet. These are the bugs that are expected to get fixed in > upcoming minor releases: > > https://bugzilla.redhat.com/buglist.cgi?component=sharding&f1=bug_status&f2=version&o1=notequals&o2=notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline My issue with sharding was reported in bugzilla on 2016-07-12 4 months for a IMHO, critical bug. If you disable sharding on a sharded volume with existing shared data, you corrupt every existing file. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Mon, Nov 14, 2016 at 04:50:44PM +0530, Pranith Kumar Karampuri wrote: > On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta < > gandalf.corvotempe...@gmail.com> wrote: > > > 2016-11-14 11:50 GMT+01:00 Pranith Kumar Karampuri : > > > To make gluster stable for VM images we had to add all these new features > > > and then fix all the bugs Lindsay/Kevin reported. We just fixed a > > corruption > > > issue that can happen with replace-brick which will be available in 3.9.0 > > > and 3.8.6. The only 2 other known issues that can lead to corruptions are > > > add-brick and the bug you filed Gandalf. Krutika just 5 minutes back saw > > > something that could possibly lead to the corruption for the add-brick > > bug. > > > Is that really the Root cause? We are not sure yet, we need more time. > > > Without Lindsay/Kevin/David Gossage's support this workload would have > > been > > > in much worse condition. These bugs are not easy to re-create thus not > > easy > > > to fix. At least that has been Krutika's experience. > > > > Ok, but this changes should be placed in a "test" version and not > > marked as stable. > > I don't see any development release, only stable releases here. > > Do you want all features ? Try the "beta/rc/unstable/alpha/dev" version. > > Do you want the stable version without known bugs but slow on VMs > > workload? Use the "-stable" version. > > > > If you relase as stable, users tend to upgrade their cluster and use > > the newer feature (that you are marking as stable). > > What If I upgrade a production cluster to a stable version and try to > > add-brick that lead to data corruption ? > > I have to restore terabytes worth of data? Gluster is made for > > scale-out, what I my cluster was made with 500TB of VMs ? > > Try to restore 500TB from a backup > > > > This is unacceptable. add-brick/replace-brick should be common "daily" > > operations. You should heavy check these for regression or bug. > > > > This is a very good point. Adding other maintainers. Obviously this is unacceptible for versions that have sharding as a functional (not experimental) feature. All supported features are expected to function without major problems (like corruption) for all standard Gluster operations. Add-brick/replace-brick are surely such Gluster operations. Of course it is possible that this does not always happen, and our tests did not catch the problem. In that case, we really need to have a bug report with all the details, and preferably a script that can be used to reproduce and detect the failure. FWIW sharding has several open bugs (like any other component), but it is not immediately clear to me if the problem reported in this email is in Bugzilla yet. These are the bugs that are expected to get fixed in upcoming minor releases: https://bugzilla.redhat.com/buglist.cgi?component=sharding&f1=bug_status&f2=version&o1=notequals&o2=notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline HTH, Niels signature.asc Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-11-14 11:50 GMT+01:00 Pranith Kumar Karampuri : > > To make gluster stable for VM images we had to add all these new features > > and then fix all the bugs Lindsay/Kevin reported. We just fixed a > corruption > > issue that can happen with replace-brick which will be available in 3.9.0 > > and 3.8.6. The only 2 other known issues that can lead to corruptions are > > add-brick and the bug you filed Gandalf. Krutika just 5 minutes back saw > > something that could possibly lead to the corruption for the add-brick > bug. > > Is that really the Root cause? We are not sure yet, we need more time. > > Without Lindsay/Kevin/David Gossage's support this workload would have > been > > in much worse condition. These bugs are not easy to re-create thus not > easy > > to fix. At least that has been Krutika's experience. > > Ok, but this changes should be placed in a "test" version and not > marked as stable. > I don't see any development release, only stable releases here. > Do you want all features ? Try the "beta/rc/unstable/alpha/dev" version. > Do you want the stable version without known bugs but slow on VMs > workload? Use the "-stable" version. > > If you relase as stable, users tend to upgrade their cluster and use > the newer feature (that you are marking as stable). > What If I upgrade a production cluster to a stable version and try to > add-brick that lead to data corruption ? > I have to restore terabytes worth of data? Gluster is made for > scale-out, what I my cluster was made with 500TB of VMs ? > Try to restore 500TB from a backup > > This is unacceptable. add-brick/replace-brick should be common "daily" > operations. You should heavy check these for regression or bug. > This is a very good point. Adding other maintainers. > > > One more take away is to get the > > documentation right. Lack of documentation led Alex to try the worst > > possible combo for storing VMs on gluster. So we as community failed in > some > > way there as well. > > > > Krutika will be sending out VM usecase related documentation after > > 28th of this month. If you have any other feedback, do let us know. > > Yes, lack of updated docs or a reference architecture is a big issue. > -- Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
2016-11-14 11:50 GMT+01:00 Pranith Kumar Karampuri : > To make gluster stable for VM images we had to add all these new features > and then fix all the bugs Lindsay/Kevin reported. We just fixed a corruption > issue that can happen with replace-brick which will be available in 3.9.0 > and 3.8.6. The only 2 other known issues that can lead to corruptions are > add-brick and the bug you filed Gandalf. Krutika just 5 minutes back saw > something that could possibly lead to the corruption for the add-brick bug. > Is that really the Root cause? We are not sure yet, we need more time. > Without Lindsay/Kevin/David Gossage's support this workload would have been > in much worse condition. These bugs are not easy to re-create thus not easy > to fix. At least that has been Krutika's experience. Ok, but this changes should be placed in a "test" version and not marked as stable. I don't see any development release, only stable releases here. Do you want all features ? Try the "beta/rc/unstable/alpha/dev" version. Do you want the stable version without known bugs but slow on VMs workload? Use the "-stable" version. If you relase as stable, users tend to upgrade their cluster and use the newer feature (that you are marking as stable). What If I upgrade a production cluster to a stable version and try to add-brick that lead to data corruption ? I have to restore terabytes worth of data? Gluster is made for scale-out, what I my cluster was made with 500TB of VMs ? Try to restore 500TB from a backup This is unacceptable. add-brick/replace-brick should be common "daily" operations. You should heavy check these for regression or bug. > One more take away is to get the > documentation right. Lack of documentation led Alex to try the worst > possible combo for storing VMs on gluster. So we as community failed in some > way there as well. > > Krutika will be sending out VM usecase related documentation after > 28th of this month. If you have any other feedback, do let us know. Yes, lack of updated docs or a reference architecture is a big issue. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Which data corruption issue is this? Could you point me to the bug report on bugzilla? -Krutika On Sat, Nov 12, 2016 at 4:28 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > Il 12 nov 2016 10:21, "Kevin Lemonnier" ha scritto: > > We've had a lot of problems in the past, but at least for us 3.7.12 (and > 3.7.15) > > seems to be working pretty well as long as you don't add bricks. We > started doing > > multiple little clusters and abandonned the idea of one big cluster, had > no > > issues since :) > > > > Well, adding bricks could be usefull... :) > > Having to create multiple cluster is not a solution and is much more > expansive. > And if you corrupt data from a single cluster you still have issues > > I think would be better to add less features and focus more to stability. > In a software defined storage, stability and consistency are the most > important things > > I'm also subscribed to moosefs and lizardfs mailing list and I don't > recall any single data corruption/data loss event > > In gluster, after some days of testing I've found a huge data corruption > issue that is still unfixed on bugzilla. > If you change the shard size on a populated cluster, you break all > existing data. > Try to do this on a cluster with working VMs and see what happens > a single cli command break everything and is still unfixed. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Sat, Nov 12, 2016 at 4:28 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > Il 12 nov 2016 10:21, "Kevin Lemonnier" ha scritto: > > We've had a lot of problems in the past, but at least for us 3.7.12 (and > 3.7.15) > > seems to be working pretty well as long as you don't add bricks. We > started doing > > multiple little clusters and abandonned the idea of one big cluster, had > no > > issues since :) > > > > Well, adding bricks could be usefull... :) > > Having to create multiple cluster is not a solution and is much more > expansive. > And if you corrupt data from a single cluster you still have issues > > I think would be better to add less features and focus more to stability. > First of all, thanks to all the folks who contributed to this thread. We value your feedback. In gluster-users and ovirt-community we saw people trying gluster and complain about heal times and split-brains. So we had to fix bugs in quorum in 3-way replication; then we started working on features like sharding for better heal times and arbiter volumes for cost benefits. To make gluster stable for VM images we had to add all these new features and then fix all the bugs Lindsay/Kevin reported. We just fixed a corruption issue that can happen with replace-brick which will be available in 3.9.0 and 3.8.6. The only 2 other known issues that can lead to corruptions are add-brick and the bug you filed Gandalf. Krutika just 5 minutes back saw something that could possibly lead to the corruption for the add-brick bug. Is that really the Root cause? We are not sure yet, we need more time. Without Lindsay/Kevin/David Gossage's support this workload would have been in much worse condition. These bugs are not easy to re-create thus not easy to fix. At least that has been Krutika's experience. Take away from this mail thread for me is: I think it is important to educate users about why we are adding new features. People are coming to the conclusion that only bug fixing corresponds to stabilization and not features. It is a wrong perception. Without the work that went into adding all those new features above in gluster, most probably you guys wouldn't have given gluster another chance because it used to be unusable before these features for VM workloads. One more take away is to get the documentation right. Lack of documentation led Alex to try the worst possible combo for storing VMs on gluster. So we as community failed in some way there as well. Krutika will be sending out VM usecase related documentation after 28th of this month. If you have any other feedback, do let us know. In a software defined storage, stability and consistency are the most > important things > > I'm also subscribed to moosefs and lizardfs mailing list and I don't > recall any single data corruption/data loss event > > In gluster, after some days of testing I've found a huge data corruption > issue that is still unfixed on bugzilla. > If you change the shard size on a populated cluster, you break all > existing data. > Try to do this on a cluster with working VMs and see what happens > a single cli command break everything and is still unfixed. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > -- Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Sat, Nov 12, 2016 at 2:11 PM, Kevin Lemonnier wrote: > > > > On the other hand at home, I tried to use GlusterFS for VM images in a > > simple replica 2 setup with Pacemaker for HA. VMs were constantly > > failing en masse even without making any changes. Very often the images > > got corrupted and had to be restored from backups. This was over a year > > ago but motivated me to try the VMs on MooseFS. Since then I've not had > > Yeah, there has been a lot of bad versions for VM, anything < 3.7.12 has > either huge heal problems or random corruption at runtime. That's why > I keep 3.7.12 everywhere, I know it works well at least with my config, > and since I have no use for the new feature why take the risk to update ? > > Interesting comments on MooseFS, I've seen it but never tried it yet > because of the single server managing the cluster, seems like a huge > risk. Guess there must be ways to have that role failover or something. > > I've locked myself more or less into the version of 3.8 I am running on as well right now as I just don't feel inclined to have to worry if anything in an update will give me issues. -- > Kevin Lemonnier > PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Il 12 nov 2016 9:04 PM, "Alex Crow" ha scritto: IMHO GlusterFS would be a great > product if it tried to: > > a) Add less features per release, and/or slowing down the release cycle. > Maybe have a "Feature" > those that need to try new, well, features. > b) Concentrate on issues like split-brain, healing, and scaling online > without data loss. Seems to be a common theme on the list where healing > doesn't work without tinkering. It should really "just work". > c) Have a peek at BeeGFS. It's a very well-performing FS that has its > focus on HPC. You can't stand to lose many thousands of CPU-hours of > work if your FS goes down, and it has to be fast. > > The biggest question for me is what is the target market for GlusterFS? > Is it: > > HPC (performance, reliability on the large scale, ie loss of one file is > OK, all not, no funky features) > VM storage (much the same as HPC but large file performance required,no > loss or corruption of blocks within a file) > General File (medium performance OK, small file and random access > paramount, resilience and consistency need to be 99.999%, features such > as ACLs and XATTRs, snapshots required) > > i think if the documentation/wiki addressed these questions it would > make it easier for newcomers to evaluate the product. Totally agree > This needs to be a warning or clearly documented. If you lose a couple > of PB of data in a professional role, I'd not fancy your employment > prospects. I've always had the feeling that GlusterFS is a bit of a > playground for new features and the only way to really have a stable > storage system is to stump up the cash to RedHat (and we've purchased a > lot of RHEL/RHEV licences), but having so many problems in the community > version really even puts me off buying the full package! > Again, totally agree ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
> > On the other hand at home, I tried to use GlusterFS for VM images in a > simple replica 2 setup with Pacemaker for HA. VMs were constantly > failing en masse even without making any changes. Very often the images > got corrupted and had to be restored from backups. This was over a year > ago but motivated me to try the VMs on MooseFS. Since then I've not had Yeah, there has been a lot of bad versions for VM, anything < 3.7.12 has either huge heal problems or random corruption at runtime. That's why I keep 3.7.12 everywhere, I know it works well at least with my config, and since I have no use for the new feature why take the risk to update ? Interesting comments on MooseFS, I've seen it but never tried it yet because of the single server managing the cluster, seems like a huge risk. Guess there must be ways to have that role failover or something. -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 signature.asc Description: Digital signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
> Sure, but thinking about it later we realised that it might be for the better. > I believe when sharding is enabled the shards will be dispersed across all the > replica sets, making it that losing a replica set will kill all your VMs. > > Imagine a 16x3 volume for example, losing 2 bricks could bring the whole thing > down if they happen to be in the same replica set. (I might be wrong about the > way gluster disperse shards, it's my understanding only, never had the chance > to test it). > With multiple small clusters, we have the same disk space in the end but not > that problem, it's a bit more annoying to manage but for now that's allright. > >>I'm also subscribed to moosefs and lizardfs mailing list and I don't >>recall any single data corruption/data loss event >> > Never used those, might be just because there are less users ? Really have no > idea, > maybe you are right. I can add to this. I've been using MooseFS for general file storage with Samba for over a year now for >25 million files shared to 350+ users. I've *never* lost even a single file. We had some issues with permissions but that needed a couple of lines added to our smb.conf (CTDB cluster). On the other hand at home, I tried to use GlusterFS for VM images in a simple replica 2 setup with Pacemaker for HA. VMs were constantly failing en masse even without making any changes. Very often the images got corrupted and had to be restored from backups. This was over a year ago but motivated me to try the VMs on MooseFS. Since then I've not had a single problem with unexpected downtime or corruption. It's not the fastest FS in the world but it's well balanced and has a focus on consistency and reliability, Documentation clearly explains where all the chunks of your files will be so you can clearly define your resilience and recovery strategies. IMHO GlusterFS would be a great product if it tried to: a) Add less features per release, and/or slowing down the release cycle. Maybe have a "Feature" branch like RozoFS, with a separate Stable and Testing/Current. Stable is safe, Testing is risky, and "Feature" is for those that need to try new, well, features. b) Concentrate on issues like split-brain, healing, and scaling online without data loss. Seems to be a common theme on the list where healing doesn't work without tinkering. It should really "just work". c) Have a peek at BeeGFS. It's a very well-performing FS that has its focus on HPC. You can't stand to lose many thousands of CPU-hours of work if your FS goes down, and it has to be fast. The biggest question for me is what is the target market for GlusterFS? Is it: HPC (performance, reliability on the large scale, ie loss of one file is OK, all not, no funky features) VM storage (much the same as HPC but large file performance required,no loss or corruption of blocks within a file) General File (medium performance OK, small file and random access paramount, resilience and consistency need to be 99.999%, features such as ACLs and XATTRs, snapshots required) i think if the documentation/wiki addressed these questions it would make it easier for newcomers to evaluate the product. > >>If you change the shard size on a populated cluster,A you break all >>existing data. > This needs to be a warning or clearly documented. If you lose a couple of PB of data in a professional role, I'd not fancy your employment prospects. I've always had the feeling that GlusterFS is a bit of a playground for new features and the only way to really have a stable storage system is to stump up the cash to RedHat (and we've purchased a lot of RHEL/RHEV licences), but having so many problems in the community version really even puts me off buying the full package! Cheers Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
> >doing so means 3 times cost, 3 times disks to add and manage and so on >is not "commodity" It's the exact same cost. All the clusters have replica 3, there is absolutly no difference in cost if they are separate or not. It's not as easy to manage sure, but it's not like I'm tweaking the conf everyday. > >If you have to create multiple cluster andA not using the scale-out >feature why don't you use drbd or similiar? DRBD 8 doesn't support 3 replicas, and DRBD 9 last time I tried it was basically unusable (well, not surprising for a beta). And on top of that, gluster heals after a network outage is transparent and automatic, DRBD's heal are a huge pain. Since we are using OVH servers, the network really can't be trusted unfortunatly, we do have a lot of heals happening. We use DRBD a lot when we need normal master / slave replication, it's great for that, but to run VMs I really don't like it. One of our client has a proxmox 3 cluster with DRBD 8, everytime there is a little problem with the network it's horrible to fix, compared to gluster. -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 signature.asc Description: Digital signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Il 12 nov 2016 19:29, "Kevin Lemonnier" ha scritto: > I don't understand the issue. Let's say I can fit 30 VMs on a 3 node cluster, > whenever I need to create the VM 31 I just order 3 nodes and replicate the > exact same cluster. I get the exact same performances as on the first cluster, > since it's the same hardware. For a while it'll even be a bit better since > there is only one VM on it :) doing so means 3 times cost, 3 times disks to add and manage and so on is not "commodity" If you have to create multiple cluster and not using the scale-out feature why don't you use drbd or similiar? ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
2016-11-12 10:21 GMT+01:00 Kevin Lemonnier : > We've had a lot of problems in the past, but at least for us 3.7.12 (and > 3.7.15) > seems to be working pretty well as long as you don't add bricks. We started > doing > multiple little clusters and abandonned the idea of one big cluster, had no > issues since :) I was thinking about this. If you meant creating multiple volumes, ok, but having to create multiple clusters is a bad idea. Gluster performs better with multiple nodes, if you have to split the infrastructure (and nodes) in multiple cluster, you'll affect the performance. Is something like to create multiple enterprise SAN to avoid data corruption. Data corruption must be address with firmware/software updates, not by creating multiple storages. if you create 2 gluster storages, you'll get the same issues in multiple storages. It's a bad workaround, not a solution. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Il 12 nov 2016 16:13, "David Gossage" ha scritto: > > also maybe a code monkey to sit at my keyboard and screech at me whenever I type sudo so I pay attention to what I am about to do. > Obviously yes, but for destructive operation a confirm should always asked almost every software does it. In example, with mdadm you can't remove a disk from the array without failing it first. This is a sort of confirm. Many other operation require a double confirm by the user ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On Sat, Nov 12, 2016 at 7:42 AM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > Il 12 nov 2016 14:27, "Lindsay Mathieson" > ha scritto: > > > > gluster volume reset *finger twitch* > > > > > > And boom! volume gone. > > > > There are too many destructive operations in gluster :) > > More security on stored data please! > I like Lindsay's idea of a lock setting or a flag that adds a confirmation response requirement after every gluster command run. also maybe a code monkey to sit at my keyboard and screech at me whenever I type sudo so I pay attention to what I am about to do. > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Il 12 nov 2016 14:27, "Lindsay Mathieson" ha scritto: > > gluster volume reset *finger twitch* > > > And boom! volume gone. > There are too many destructive operations in gluster :) More security on stored data please! ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
On 12/11/2016 9:58 PM, Gandalf Corvotempesta wrote: Exactly. I've proposed a warning in the cli when changing the shard size but this is still unfixed and this is scaring me it's a critical bug, IMHO, and should be addressed asap or any user could destroy the whole cluster with a simple command and no warning at all. gluster volume reset *finger twitch* And boom! volume gone. Feature request: Ability to *lock* volume settings -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Il 12 nov 2016 12:53, "Kevin Lemonnier" ha scritto: > Sure, but thinking about it later we realised that it might be for the better. > I believe when sharding is enabled the shards will be dispersed across all the > replica sets, making it that losing a replica set will kill all your VMs. > > Imagine a 16x3 volume for example, losing 2 bricks could bring the whole thing > down if they happen to be in the same replica set. (I might be wrong about the > way gluster disperse shards, it's my understanding only, never had the chance > to test it). > With multiple small clusters, we have the same disk space in the end but not > that problem, it's a bit more annoying to manage but for now that's allright. I don't use EC because i really love the "gluster feature" to have plain files stored and not encoded in any way. > Not really shocked there. Guess the cli should warn you when you try re-setting > the option though, that would be nice. Exactly. I've proposed a warning in the cli when changing the shard size but this is still unfixed and this is scaring me it's a critical bug, IMHO, and should be addressed asap or any user could destroy the whole cluster with a simple command and no warning at all. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
> >Having to create multiple cluster is not a solution and is much more >expansive. >And if you corrupt data from a single cluster you still have issues > Sure, but thinking about it later we realised that it might be for the better. I believe when sharding is enabled the shards will be dispersed across all the replica sets, making it that losing a replica set will kill all your VMs. Imagine a 16x3 volume for example, losing 2 bricks could bring the whole thing down if they happen to be in the same replica set. (I might be wrong about the way gluster disperse shards, it's my understanding only, never had the chance to test it). With multiple small clusters, we have the same disk space in the end but not that problem, it's a bit more annoying to manage but for now that's allright. > >I'm also subscribed to moosefs and lizardfs mailing list and I don't >recall any single data corruption/data loss event > Never used those, might be just because there are less users ? Really have no idea, maybe you are right. >If you change the shard size on a populated cluster,A you break all >existing data. Not really shocked there. Guess the cli should warn you when you try re-setting the option though, that would be nice. -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 signature.asc Description: Digital signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Il 12 nov 2016 10:21, "Kevin Lemonnier" ha scritto: > We've had a lot of problems in the past, but at least for us 3.7.12 (and 3.7.15) > seems to be working pretty well as long as you don't add bricks. We started doing > multiple little clusters and abandonned the idea of one big cluster, had no > issues since :) > Well, adding bricks could be usefull... :) Having to create multiple cluster is not a solution and is much more expansive. And if you corrupt data from a single cluster you still have issues I think would be better to add less features and focus more to stability. In a software defined storage, stability and consistency are the most important things I'm also subscribed to moosefs and lizardfs mailing list and I don't recall any single data corruption/data loss event In gluster, after some days of testing I've found a huge data corruption issue that is still unfixed on bugzilla. If you change the shard size on a populated cluster, you break all existing data. Try to do this on a cluster with working VMs and see what happens a single cli command break everything and is still unfixed. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
>Don't get me wrong but I'm seeing too many "critical" issues like file >corruptions, crashes or similiar recently >Is gluster ready for production? >I'm scared about placing our production VMs (more or less 80) on gluster, >in case of corruption I'll loose everything We've had a lot of problems in the past, but at least for us 3.7.12 (and 3.7.15) seems to be working pretty well as long as you don't add bricks. We started doing multiple little clusters and abandonned the idea of one big cluster, had no issues since :) -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 signature.asc Description: Digital signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Il 12 nov 2016 03:29, "Krutika Dhananjay" ha scritto: > > Hi, > > Yes, this has been reported before by Lindsay Mathieson and Kevin Lemonnier on this list. > We just found one issue with replace-brick that we recently fixed. > Don't get me wrong but I'm seeing too many "critical" issues like file corruptions, crashes or similiar recently Is gluster ready for production? I'm scared about placing our production VMs (more or less 80) on gluster, in case of corruption I'll loose everything ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks
Hi, Yes, this has been reported before by Lindsay Mathieson and Kevin Lemonnier on this list. We just found one issue with replace-brick that we recently fixed. In your case, are you doing add-brick and changing the replica count (say from 2 -> 3) or are you adding "replica-count" number of bricks every time? -Krutika On Sat, Nov 12, 2016 at 6:40 AM, ML Wong wrote: > Have anyone encounter this behavior? > > Running 3.7.16 from centos-gluster37, on CentOS 7.2 with NFS-Ganesha > 2.3.0. VMs are running fine without problems and with Sharding on. However, > when i either do a "add-brick" or "remove-brick start force". VM files will > then be corrupted, and the VM will not be able to boot anymore. > > So far, as i access files through regular NFS, all regular files, or > directories seems to be accessible fine. I am not sure if this somehow > relate to bug1318136, but any help will be appreciated. Or, m i missing any > settings? Below is the vol info of gluster volume. > > Volume Name: nfsvol1 > Type: Distributed-Replicate > Volume ID: 06786467-4c8a-48ad-8b1f-346aa8342283 > Status: Started > Number of Bricks: 2 x 2 = 4 > Transport-type: tcp > Bricks: > Brick1: stor4:/data/brick1/nfsvol1 > Brick2: stor5:/data/brick1/nfsvol1 > Brick3: stor1:/data/brick1/nfsvol1 > Brick4: stor2:/data/brick1/nfsvol1 > Options Reconfigured: > features.shard-block-size: 64MB > features.shard: on > ganesha.enable: on > features.cache-invalidation: off > nfs.disable: on > performance.readdir-ahead: on > nfs-ganesha: enable > cluster.enable-shared-storage: enable > > thanks, > Melvin > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
