Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec
On 10/29/2015 06:11 PM, Jeff Darcy wrote: I want to understand if there is a possibility of exposing these as different modules that we can mix and match, using options. It’s not only possible, but it’s easier than you might think. If an option is set (cluster.nsr IIRC) then we replace cluster/afr with cluster/nsr-client and then add some translators to the server-side stack. A year ago that was just one nsr-server translator. The journaling part has already been split out, and I plan to do the same with the leader-election parts (making them usable for server-side AFR or EC) as well. It shouldn’t be hard to control the addition and removal of these and related translators (e.g. index) with multiple options instead of just one. The biggest stumbling block I’ve actually hit when trying to do this with AFR on the server side is the *tests*, many of which can’t handle delays on the client side while the server side elects leaders and cross-connects peers. That’s all solvable. It just would have taken more time than I had available for the experiment. precisely. I think switching is not that difficult once we make sure healing is complete. Switching is a rare operation IMO so we can probably ask the users to do stop/choose-new-value/start the volume after choosing the options. This way is simpler than to migrate between the volumes where you have to probably copy the data. The two sets of metadata are *entirely* disjoint, which puts us in a good position compared e.g. to DHT/tiering which had overlaps. As long as the bricks are “clean” switching back and forth should be simple. In fact I expect to do this a lot when we get to characterizing performance etc. Good to hear this. choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we come up with better metadata journals/stores it should be easy to plug them is what I'm thinking. The idea I have is based on the workload, users should be able to decide which pair of synchronization/metadata works best for them (Or we can also recommend based on our tests). Wanted to seek your inputs. Absolutely. As I’m sure you’re tired of hearing, I believe NSR will outperform AFR by a significant margin for most workloads and configurations. I wouldn’t be the project’s initiator/leader if I didn’t believe that, but I’m OK if others disagree. We’ll find out eventually. ;) More importantly, “most” is still not “all”. Even by my own reckoning, there are cases in which AFR will perform better or be preferable for other reasons. EC’s durability and space-efficiency advantages make an even stronger case for preserving both kinds of data paths and metadata arrangements. That’s precisely why I want to make the journaling and leader-election parts more generic. All the best for your endeavors! Lets make users happy. Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec
On 10/29/2015 02:06 AM, Pranith Kumar Karampuri wrote: hi, I want to understand how are you guys planning to integrate NSR volumes to the existing CLIs. Here are some thoughts I had, wanted to know your thoughts: At the heart of both the replication/ec schemes we have 1) synchronization mechanisms a) afr,ec does it using locks b) nsr does it using leader election 2) Metadata to figure out the healing/reconciliation aspects a) afr,ec does it using xattrs b) nsr does it using journals I want to understand if there is a possibility of exposing these as different modules that we can mix and match, using options. If the users choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we come up with better metadata journals/stores it should be easy to plug them is what I'm thinking. The idea I have is based on the workload, users should be able to decide which pair of synchronization/metadata works best for them (Or we can also recommend based on our tests). I wanted to add some more *insights* (or stir up the mess) based on the DHT2 MDS/DS split. With this case, we could (that may even change to a would) have NSR/AFR *only* on the MDS side of things and no EC there, and have any of EC/AFR/NSR on the DS side of things. The theory being, EC provides data space efficiency and replicates meta-data anyway, so instead of having that on the MDS side of things, which does not have any data, we may want to stick to replication methods for availability reasons. I am adding this here, although this is orthogonal to the current discussion, just to provide a perspective into our thought process. Wanted to seek your inputs. Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec
> >> I want to understand if there is a possibility of exposing these as > >> different modules that we can mix and match, using options. It’s not only possible, but it’s easier than you might think. If an option is set (cluster.nsr IIRC) then we replace cluster/afr with cluster/nsr-client and then add some translators to the server-side stack. A year ago that was just one nsr-server translator. The journaling part has already been split out, and I plan to do the same with the leader-election parts (making them usable for server-side AFR or EC) as well. It shouldn’t be hard to control the addition and removal of these and related translators (e.g. index) with multiple options instead of just one. The biggest stumbling block I’ve actually hit when trying to do this with AFR on the server side is the *tests*, many of which can’t handle delays on the client side while the server side elects leaders and cross-connects peers. That’s all solvable. It just would have taken more time than I had available for the experiment. > precisely. I think switching is not that difficult once we make sure > healing is complete. Switching is a rare operation IMO so we can > probably ask the users to do stop/choose-new-value/start the volume > after choosing the options. This way is simpler than to migrate > between the volumes where you have to probably copy the data. The two sets of metadata are *entirely* disjoint, which puts us in a good position compared e.g. to DHT/tiering which had overlaps. As long as the bricks are “clean” switching back and forth should be simple. In fact I expect to do this a lot when we get to characterizing performance etc. > >> choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future > >> if we come up with better metadata journals/stores it should be > >> easy to plug them is what I'm thinking. The idea I have is based on > >> the workload, users should be able to decide which pair of > >> synchronization/metadata works best for them (Or we can also > >> recommend based on our tests). Wanted to seek your inputs. Absolutely. As I’m sure you’re tired of hearing, I believe NSR will outperform AFR by a significant margin for most workloads and configurations. I wouldn’t be the project’s initiator/leader if I didn’t believe that, but I’m OK if others disagree. We’ll find out eventually. ;) More importantly, “most” is still not “all”. Even by my own reckoning, there are cases in which AFR will perform better or be preferable for other reasons. EC’s durability and space-efficiency advantages make an even stronger case for preserving both kinds of data paths and metadata arrangements. That’s precisely why I want to make the journaling and leader-election parts more generic. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec
On 10/29/2015 12:18 PM, Venky Shankar wrote: On Thu, Oct 29, 2015 at 11:36 AM, Pranith Kumar Karampuri wrote: hi, I want to understand how are you guys planning to integrate NSR volumes to the existing CLIs. Here are some thoughts I had, wanted to know your thoughts: At the heart of both the replication/ec schemes we have 1) synchronization mechanisms a) afr,ec does it using locks b) nsr does it using leader election 2) Metadata to figure out the healing/reconciliation aspects a) afr,ec does it using xattrs b) nsr does it using journals I want to understand if there is a possibility of exposing these as different modules that we can mix and match, using options. If the users Do you mean abstracting it out during volume creation? At a high level this could be in the form of client or server side replication. Not that AFR cannot be used on the server side (you'd know better than me), but, if at all this level of abstraction is used, we'd need to default to what fits best in what use case (as you already mentioned below) but still retaining the flexibility to override it. precisely. I think switching is not that difficult once we make sure healing is complete. Switching is a rare operation IMO so we can probably ask the users to do stop/choose-new-value/start the volume after choosing the options. This way is simpler than to migrate between the volumes where you have to probably copy the data. Pranith choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we come up with better metadata journals/stores it should be easy to plug them is what I'm thinking. The idea I have is based on the workload, users should be able to decide which pair of synchronization/metadata works best for them (Or we can also recommend based on our tests). Wanted to seek your inputs. Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec
On Thu, Oct 29, 2015 at 11:36 AM, Pranith Kumar Karampuri wrote: > hi, > I want to understand how are you guys planning to integrate NSR > volumes to the existing CLIs. Here are some thoughts I had, wanted to know > your thoughts: > At the heart of both the replication/ec schemes we have > 1) synchronization mechanisms >a) afr,ec does it using locks >b) nsr does it using leader election > 2) Metadata to figure out the healing/reconciliation aspects >a) afr,ec does it using xattrs >b) nsr does it using journals > > I want to understand if there is a possibility of exposing these as > different modules that we can mix and match, using options. If the users Do you mean abstracting it out during volume creation? At a high level this could be in the form of client or server side replication. Not that AFR cannot be used on the server side (you'd know better than me), but, if at all this level of abstraction is used, we'd need to default to what fits best in what use case (as you already mentioned below) but still retaining the flexibility to override it. > choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we come > up with better metadata journals/stores it should be easy to plug them is > what I'm thinking. The idea I have is based on the workload, users should be > able to decide which pair of synchronization/metadata works best for them > (Or we can also recommend based on our tests). Wanted to seek your inputs. > > Pranith > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] pluggability of some aspects in afr/nsr/ec
hi, I want to understand how are you guys planning to integrate NSR volumes to the existing CLIs. Here are some thoughts I had, wanted to know your thoughts: At the heart of both the replication/ec schemes we have 1) synchronization mechanisms a) afr,ec does it using locks b) nsr does it using leader election 2) Metadata to figure out the healing/reconciliation aspects a) afr,ec does it using xattrs b) nsr does it using journals I want to understand if there is a possibility of exposing these as different modules that we can mix and match, using options. If the users choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we come up with better metadata journals/stores it should be easy to plug them is what I'm thinking. The idea I have is based on the workload, users should be able to decide which pair of synchronization/metadata works best for them (Or we can also recommend based on our tests). Wanted to seek your inputs. Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel