Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
I was away on a small vacation last week, so I haven't been able to reply to this thread till now. There has been quite some discussion while I was away. This kind of discussion was exactly what we wanted. I've read through the thread, and I'd like to summarize what I feel is the general feeling shared by all. - GlusterD has shortcomings in its store and membership functionality and is in need of improvement. This is the immediate goal that we were targeting. We started this discussion because we did not want to start the development without hearing what the community has to say. - We'd like to move to using external utilities to manage some of the things GlusterD currently does. This is because we don't want to burden ourselves with creating new implementations for doing tasks other tools do well already. This is the reason for our exploration of consul. Consul does distributed simple storage and membership well, and could replace the existing store and peer membership mechanism as implemented in GlusterD. - There is also a need to improve and build upon the current cluster management facilities provided by GlusterD. Users want to be able to do more using GlusterD than what is currently possible. This, IMO, mainly is with regards to monitoring and automation. Users want more information from GlusterD regarding the cluster state, and want to run automated tasks based on the obtained information. We'd like to move these kind of improvements away from core GlusterD as well, and this is where the automation tools like Puppet, Saltstack, etc. come in. - Users want GlusterFS to remain easy to deploy With the talk of all the external utilities, some users are concerned GlusterFS is going to become hard to deploy, with even small deployments needing lots of dependencies to be pulled in. This is a something I am concerned with as well, as ease of deployment is one of the notable characters of GlusterFS and we shouldn't lose it. With these details in mind, I have an initial idea on how GlusterD-2.0 and GlusterFS-Quattro are going to shape up. GlusterD will continue to exist and will do most of the functions it performs today. Where ever possible it would delegate it's functions to external utilities. We need to make sure that these external utilities don't need any further configuration from the user to be usable (how we are going to accomplish this is a problem on its own). Retaining GlusterD in this way should satisfy the easy to deploy character. The higher level automation and monitoring capabilities will be handled by tools like Puppet, Saltstack, Chef etc. We would probably need to pick one among these for initial community support. I have no preference among these, but since we already have a puppet-gluster module, I think Puppet would be the most likely choice. GlusterD would be extended to provide more internal, cluster information to these tools, for them to make their intelligent decisions. The choice of programming language and external utilities we make, will decide how hard achieving the above would be. For the present we (GlusterD maintainers, KP and me, and other GlusterD contributers) would like to start off GlusterD-2.0 by using Consul for membership and config storage. The initial implementation would probably only just have the minimum cluster management functions, and would mainly be a POC or prototype kind of implementation. We'll try to keep it clean and reusable for later. We have been planning to use Go as the language of development, as we believe it is easy for C developers to pick up and provides features that make it easier to write distributed systems. There has been another mail thread on the topic of language choices, which should probably answer any questions on this decision. What does the list think of my thoughts above? ~kaushal ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
On 15/09/2014, at 8:19 PM, Kaushal M wrote: snip For the present we (GlusterD maintainers, KP and me, and other GlusterD contributers) would like to start off GlusterD-2.0 by using Consul for membership and config storage. The initial implementation would probably only just have the minimum cluster management functions, and would mainly be a POC or prototype kind of implementation. We'll try to keep it clean and reusable for later. We have been planning to use Go as the language of development, as we believe it is easy for C developers to pick up and provides features that make it easier to write distributed systems. There has been another mail thread on the topic of language choices, which should probably answer any questions on this decision. What does the list think of my thoughts above? To be clear, this is just an experimental proof-of-concept yeah? If so, how much time is expected to be needed for it, what are the success/failure criteria (to judge it at the end), and when is this PoC expected to be completed? + Justin -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Has anyone looked into whether LogCabin can provide the consistent small storage based on RAFT for Gluster? https://github.com/logcabin/logcabin I have no experience with using it so I cannot say if it is good or suitable. I do know the following project uses it and it's just not as easy to setup as Gluster is - it also has Zookeeper support etc. https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of James Sent: Friday, September 12, 2014 4:17 AM To: Krishnan Parthasarathi Cc: Gluster Devel; gluster-us...@gluster.org; Balamurugan Arumugam Subject: Re: [Gluster-users] [Gluster-devel] Proposal for GlusterD-2.0 On Fri, Sep 12, 2014 at 12:02 AM, Krishnan Parthasarathi kpart...@redhat.com wrote: - Original Message - On Thu, Sep 11, 2014 at 4:55 AM, Krishnan Parthasarathi kpart...@redhat.com wrote: I think using Salt as the orchestration framework is a good idea. We would still need to have a consistent distributed store. I hope Salt has the provision to use one of our choice. It could be consul or something that satisfies the criteria for choosing alternate technology. I would wait for a couple of days for the community to chew on this and share their thoughts. If we have a consensus on this, we could 'port' the 'basic'[1] volume management commands to a system built using Salt and see for real how it fits our use case. Thoughts? I disagree. I think puppet + puppet-gluster would be a good idea :) One advantage is that the technology is already proven, and there's a working POC. Feel free to prove me wrong, or to request any features that it's missing. ;) I am glad you joined this discussion. I was expecting you to join earlier :) Assuming non-sarcasm, then thank you :) I didn't join earlier, because 1) I'm not a hardcore algorithmist like most of you are and, 2) I'm busy a lot :P IIUC, puppet-gluster uses glusterd to perform glusterfs deployments. I think it's important to consider puppet given its acceptance.What are your thoughts on building 'glusterd' using puppet? I think I can describe my proposal simply, and then give the reason why... Proposal: glusterd shouldn't go away or aim to greatly automate / do much more than it does today already (with a few exceptions). puppet-gluster should be used as a higher layer abstraction to do the complex management. More features would still need to be added to address every use case and corner case, but I think we're a good deal of the way there. My work on automatically growing gluster volumes was demo-ed as a POC but never finished and pushed to git master. I have no comment on language choices or rewrites of glusterd itself, since functionality wise it mostly works for me. Why? The reasons this makes a lot of sense: 1) Higher level declarative languages can guarantee a lot of safety in terms of avoiding incorrect operations. It's easy to get the config management graph to error out, which typically means there is a bug in the code to be fixed. In this scenario, no code actually runs! This means your data won't get accidentally hurt, or put into a partial state. 2) Lines of code to accomplish certain things in puppet might be an order of magnitude less than in a typical imperative language. Statistically speaking, by keeping LOC down, the logic can be more concise, and have fewer bugs. This also lets us reason about things from a higher POV. 3) Understanding the logic in puppet can be easier than reading a pile of c or go code. This is why you can look at a page of python and understand, but staring at three pages of assembly is useless. In any case, I don't think it's likely that Gluster will end up using puppet, although I do hope people will think about this a bit more and at least consider it seriously. Since many people are not very familiar with configuration management, please don't be shy if you'd like to have a quick chat about it, and maybe a little demo to show you what's truly possible. HTH, James The proposal mail describes the functions glusterd performs today. With that as a reference could you elaborate on how we could use puppet to perform some (or all) the functions of glusterd? ~KP ___ Gluster-users mailing list gluster-us...@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
it also has Zookeeper support etc. - just to correct this and remove the perception that LogCabin somehow requires Zookeeper or works with it. LogCabin as I understand is the C++ implementation of a small store based on the Raft consensus protocol - to provide a consistent and a small distributed store. The Zookeeper part is a RAMCloud thing for storing co-ordinator information to an external cluster and it does not come with LogCabin. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Prasad, Nirmal Sent: Friday, September 12, 2014 5:58 PM To: James; Krishnan Parthasarathi Cc: Balamurugan Arumugam; gluster-us...@gluster.org; Gluster Devel Subject: Re: [Gluster-users] [Gluster-devel] Proposal for GlusterD-2.0 Has anyone looked into whether LogCabin can provide the consistent small storage based on RAFT for Gluster? https://github.com/logcabin/logcabin I have no experience with using it so I cannot say if it is good or suitable. I do know the following project uses it and it's just not as easy to setup as Gluster is - it also has Zookeeper support etc. https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of James Sent: Friday, September 12, 2014 4:17 AM To: Krishnan Parthasarathi Cc: Gluster Devel; gluster-us...@gluster.org; Balamurugan Arumugam Subject: Re: [Gluster-users] [Gluster-devel] Proposal for GlusterD-2.0 On Fri, Sep 12, 2014 at 12:02 AM, Krishnan Parthasarathi kpart...@redhat.com wrote: - Original Message - On Thu, Sep 11, 2014 at 4:55 AM, Krishnan Parthasarathi kpart...@redhat.com wrote: I think using Salt as the orchestration framework is a good idea. We would still need to have a consistent distributed store. I hope Salt has the provision to use one of our choice. It could be consul or something that satisfies the criteria for choosing alternate technology. I would wait for a couple of days for the community to chew on this and share their thoughts. If we have a consensus on this, we could 'port' the 'basic'[1] volume management commands to a system built using Salt and see for real how it fits our use case. Thoughts? I disagree. I think puppet + puppet-gluster would be a good idea :) One advantage is that the technology is already proven, and there's a working POC. Feel free to prove me wrong, or to request any features that it's missing. ;) I am glad you joined this discussion. I was expecting you to join earlier :) Assuming non-sarcasm, then thank you :) I didn't join earlier, because 1) I'm not a hardcore algorithmist like most of you are and, 2) I'm busy a lot :P IIUC, puppet-gluster uses glusterd to perform glusterfs deployments. I think it's important to consider puppet given its acceptance.What are your thoughts on building 'glusterd' using puppet? I think I can describe my proposal simply, and then give the reason why... Proposal: glusterd shouldn't go away or aim to greatly automate / do much more than it does today already (with a few exceptions). puppet-gluster should be used as a higher layer abstraction to do the complex management. More features would still need to be added to address every use case and corner case, but I think we're a good deal of the way there. My work on automatically growing gluster volumes was demo-ed as a POC but never finished and pushed to git master. I have no comment on language choices or rewrites of glusterd itself, since functionality wise it mostly works for me. Why? The reasons this makes a lot of sense: 1) Higher level declarative languages can guarantee a lot of safety in terms of avoiding incorrect operations. It's easy to get the config management graph to error out, which typically means there is a bug in the code to be fixed. In this scenario, no code actually runs! This means your data won't get accidentally hurt, or put into a partial state. 2) Lines of code to accomplish certain things in puppet might be an order of magnitude less than in a typical imperative language. Statistically speaking, by keeping LOC down, the logic can be more concise, and have fewer bugs. This also lets us reason about things from a higher POV. 3) Understanding the logic in puppet can be easier than reading a pile of c or go code. This is why you can look at a page of python and understand, but staring at three pages of assembly is useless. In any case, I don't think it's likely that Gluster will end up using puppet, although I do hope people will think about this a bit more and at least consider it seriously. Since many people are not very familiar with configuration management, please don't be shy if you'd like to have a quick chat about it, and maybe a little demo to show you what's truly possible. HTH, James The proposal mail describes the functions glusterd
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
- Original Message - From: Jeff Darcy jda...@redhat.com To: Balamurugan Arumugam b...@gluster.com Cc: Justin Clift jus...@gluster.org, gluster-us...@gluster.org, Gluster Devel gluster-devel@gluster.org Sent: Thursday, September 11, 2014 7:45:52 PM Subject: Re: [Gluster-users] [Gluster-devel] Proposal for GlusterD-2.0 Yes. I came across Salt currently for unified management for storage to manage gluster and ceph which is still in planning phase. I could think of a complete requirement of infra requirement to solve from glusterd to unified management. Calamari ceph management already uses Salt. It would be the ideal solution with Salt (or any infra) if gluster, ceph and unified management uses. I think the idea of using Salt (or similar) is interesting, but it's also key that Ceph still has its mon cluster as well. (Is mon calamari an *intentional* Star Wars reference?) As I see it, glusterd or anything we use to replacement has multiple responsibilities: (1) Track the current up/down state of cluster members and resources. (2) Store configuration and coordinate changes to it. (3) Orchestrate complex or long-running activities (e.g. rebalance). (4) Provide service discovery (current portmapper). Salt and its friends clearly shine at (2) and (3), though they outsource the actual data storage to an external data store. With such a data store, (4) becomes pretty trivial. The sticking point for me is (1). How does Salt handle that need, or how might it be satisfied on top of the facilities Salt does provide? I can see *very* clearly how to do it on top of etcd or consul. Could those in fact be used for Salt's data store? It seems like Salt shouldn't need a full-fledged industrial strength database, just something with high consistency/availability and some basic semantics. Maybe we should try to engage with the Salt developers to come up with ideas. Or find out exactly what functionality they found still needs to be in the mon cluster and not in Salt. Salt has a way to push events to salt-master from salt-minions. At salt-master, we would extend reactor[1] to handle such (any) events. This helps to solve (1). Regards, Bala [1] http://docs.saltstack.com/en/latest/topics/reactor/ ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Has anyone looked into whether LogCabin can provide the consistent small storage based on RAFT for Gluster? https://github.com/logcabin/logcabin I have no experience with using it so I cannot say if it is good or suitable. I do know the following project uses it and it's just not as easy to setup as Gluster is - it also has Zookeeper support etc. https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud LogCabin is the canonical implementation of Raft, by the author of the Raft protocol, so it was the first implementation I looked at. Sad to say, it didn't seem that stable. AFAIK RAMCloud - itself an academic project - is the only user, whereas etcd and consul are being used by multiple projects and in production. Also, I found the etcd code at least more readable than LogCabin despite the fact that I've worked in C++ before and had never seen any Go code until that time. Then again, those were early days for all three projects (consul didn't even exist yet) so things might have changed. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Bala, I think using Salt as the orchestration framework is a good idea. We would still need to have a consistent distributed store. I hope Salt has the provision to use one of our choice. It could be consul or something that satisfies the criteria for choosing alternate technology. I would wait for a couple of days for the community to chew on this and share their thoughts. If we have a consensus on this, we could 'port' the 'basic'[1] volume management commands to a system built using Salt and see for real how it fits our use case. Thoughts? [1] basic commands - peer-probe, volume-create, volume-start and volume-add-brick ~KP - Original Message - - Original Message - From: Justin Clift jus...@gluster.org To: Balamurugan Arumugam b...@gluster.com Cc: Kaushal M kshlms...@gmail.com, gluster-us...@gluster.org, Gluster Devel gluster-devel@gluster.org Sent: Thursday, September 11, 2014 7:33:52 AM Subject: Re: [Gluster-users] [Gluster-devel] Proposal for GlusterD-2.0 On 11/09/2014, at 2:46 AM, Balamurugan Arumugam wrote: snip WRT glusterd problem, I see Salt already resolves most of them at infrastructure level. Its worth considering it. Salt used to have (~12 months ago) a reputation for being really buggy. Any idea if that's still the case? I heard from various presentations about that due to zeromq 2.x issues. With zeromq 3.x, its all gone. But we could explore more on stability point of view. Apart from that though, using Salt is an interesting idea. :) Yes. I came across Salt currently for unified management for storage to manage gluster and ceph which is still in planning phase. I could think of a complete requirement of infra requirement to solve from glusterd to unified management. Calamari ceph management already uses Salt. It would be the ideal solution with Salt (or any infra) if gluster, ceph and unified management uses. Regards, Bala ___ Gluster-users mailing list gluster-us...@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
- Original Message - From: Krishnan Parthasarathi kpart...@redhat.com To: Balamurugan Arumugam barum...@redhat.com Cc: gluster-us...@gluster.org, Gluster Devel gluster-devel@gluster.org Sent: Thursday, September 11, 2014 2:25:45 PM Subject: Re: [Gluster-users] [Gluster-devel] Proposal for GlusterD-2.0 Bala, I think using Salt as the orchestration framework is a good idea. We would still need to have a consistent distributed store. I hope Salt has the provision to use one of our choice. It could be consul or something that satisfies the criteria for choosing alternate technology. I would wait for a couple of days for the community to chew on this and share their thoughts. If we have a consensus on this, we could 'port' the 'basic'[1] volume management commands to a system built using Salt and see for real how it fits our use case. Thoughts? For distributed store, I would think of MongoDB which provides distributed/replicated/highly available/master read-write/slave read-only database. Lets get what community think about SaltStack and/or MongoDB. [1] basic commands - peer-probe, volume-create, volume-start and volume-add-brick Regards, Bala ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
On 11/09/2014, at 10:16 AM, Balamurugan Arumugam wrote: snip For distributed store, I would think of MongoDB which provides distributed/replicated/highly available/master read-write/slave read-only database. Lets get what community think about SaltStack and/or MongoDB. Is this relevant for MongoDB still? (it's from 12+ months ago) http://aphyr.com/posts/284-call-me-maybe-mongodb + Justin -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Yes. I came across Salt currently for unified management for storage to manage gluster and ceph which is still in planning phase. I could think of a complete requirement of infra requirement to solve from glusterd to unified management. Calamari ceph management already uses Salt. It would be the ideal solution with Salt (or any infra) if gluster, ceph and unified management uses. I think the idea of using Salt (or similar) is interesting, but it's also key that Ceph still has its mon cluster as well. (Is mon calamari an *intentional* Star Wars reference?) As I see it, glusterd or anything we use to replacement has multiple responsibilities: (1) Track the current up/down state of cluster members and resources. (2) Store configuration and coordinate changes to it. (3) Orchestrate complex or long-running activities (e.g. rebalance). (4) Provide service discovery (current portmapper). Salt and its friends clearly shine at (2) and (3), though they outsource the actual data storage to an external data store. With such a data store, (4) becomes pretty trivial. The sticking point for me is (1). How does Salt handle that need, or how might it be satisfied on top of the facilities Salt does provide? I can see *very* clearly how to do it on top of etcd or consul. Could those in fact be used for Salt's data store? It seems like Salt shouldn't need a full-fledged industrial strength database, just something with high consistency/availability and some basic semantics. Maybe we should try to engage with the Salt developers to come up with ideas. Or find out exactly what functionality they found still needs to be in the mon cluster and not in Salt. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
I'm so glad to read this. I was thinking the same thing. On Sep 11, 2014, at 7:22 AM, Jeff Darcy jda...@redhat.com wrote: For distributed store, I would think of MongoDB which provides distributed/replicated/highly available/master read-write/slave read-only database. Lets get what community think about SaltStack and/or MongoDB. I definitely do not think MongoDB is the right tool for this job. I'm not one of those people who just bash MongoDB out of fashion, either. I frequently defend them against such attacks, and I used MongoDB for some work on CloudForms a while ago. However, a full MongoDB setup carries a pretty high operational complexity, to support high scale and rich features . . . which we don't need. This part of our system doesn't need sharding. It doesn't need complex ad-hoc query capability. If we don't need those features, we *certainly* don't need the complexity that comes with them. We need something with the very highest levels of reliability and consistency, with as little complexity as possible to go with that. Even its strongest advocates would probably agree that MongoDB doesn't fit those requirements very well. ___ Gluster-users mailing list gluster-us...@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
On 11/09/2014, at 2:46 AM, Balamurugan Arumugam wrote: snip WRT glusterd problem, I see Salt already resolves most of them at infrastructure level. Its worth considering it. Salt used to have (~12 months ago) a reputation for being really buggy. Any idea if that's still the case? Apart from that though, using Salt is an interesting idea. :) + Justin -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
- Original Message - From: Justin Clift jus...@gluster.org To: Balamurugan Arumugam b...@gluster.com Cc: Kaushal M kshlms...@gmail.com, gluster-us...@gluster.org, Gluster Devel gluster-devel@gluster.org Sent: Thursday, September 11, 2014 7:33:52 AM Subject: Re: [Gluster-users] [Gluster-devel] Proposal for GlusterD-2.0 On 11/09/2014, at 2:46 AM, Balamurugan Arumugam wrote: snip WRT glusterd problem, I see Salt already resolves most of them at infrastructure level. Its worth considering it. Salt used to have (~12 months ago) a reputation for being really buggy. Any idea if that's still the case? I heard from various presentations about that due to zeromq 2.x issues. With zeromq 3.x, its all gone. But we could explore more on stability point of view. Apart from that though, using Salt is an interesting idea. :) Yes. I came across Salt currently for unified management for storage to manage gluster and ceph which is still in planning phase. I could think of a complete requirement of infra requirement to solve from glusterd to unified management. Calamari ceph management already uses Salt. It would be the ideal solution with Salt (or any infra) if gluster, ceph and unified management uses. Regards, Bala ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Is there any reason not to consider zookeeper? I did bring up that idea a while ago. I'm no Java fan myself, but still I was surprised by the vehemence of the reactions. To put it politely, many seemed to consider the dependency on Java unacceptable for both resource and security reasons. Some community members said that they'd be forced to switch to another DFS if we went that way. It didn't seem like a very promising direction to explore further. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
That's disappointing. I can certainly understand wanting to keep dependencies small, but that sounds like FUD more than a reasoned argument. I do not envy your position navigating such waters. On Sep 8, 2014, at 2:38 PM, Jeff Darcy jda...@redhat.com wrote: Is there any reason not to consider zookeeper? I did bring up that idea a while ago. I'm no Java fan myself, but still I was surprised by the vehemence of the reactions. To put it politely, many seemed to consider the dependency on Java unacceptable for both resource and security reasons. Some community members said that they'd be forced to switch to another DFS if we went that way. It didn't seem like a very promising direction to explore further. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Is there any reason not to consider zookeeper? The 3.4 release is quite stable and due to a large number of users, bugs are fixed and its quirks are known. I like the idea of RAFT. The paper is well written and very compelling. The last time I read it, a number of critical issues were glossed over - for instance, log compaction and pruning. Systems must be correct in both theory and implementation. Although many raft-based sustems have cropped in the last year or so since RAFT was published, I don't judge their use to be significant compared to zookeeper. Quality only comes with maturity and many workloads bashing on it. The last time I needed to build up a new distributed system, I had written up some notes about etcd vs zookeeper. Perhaps you will find them helpful, or motivate some new questions before you make your decision. https://docs.google.com/document/d/1FOnLD26W9iQ2CUZ-jVCn7o0OrX8KPH7QGeokN4tA_j4/edit On Monday, September 8, 2014, Jonathan Barber jonathan.bar...@gmail.com wrote: On 8 September 2014 05:05, Krishnan Parthasarathi kpart...@redhat.com wrote: Bulk of current GlusterD code deals with keeping the configuration of the cluster and the volumes in it consistent and available across the nodes. The current algorithm is not scalable (N^2 in no. of nodes) and doesn't prevent split-brain of configuration. This is the problem area we are targeting for the first phase. As part of the first phase, we aim to delegate the distributed configuration store. We are exploring consul [1] as a replacement for the existing distributed configuration store (sum total of /var/lib/glusterd/* across all nodes). Consul provides distributed configuration store which is consistent and partition tolerant. By moving all Gluster related configuration information into consul we could avoid split-brain situations. Did you get a chance to go over the following questions while making the decision? If yes could you please share the info. What are the consistency guarantees for changing the configuration in case of network partitions? specifically when there are 2 nodes and 1 of them is not reachable? consistency guarantees when there are more than 2 nodes? What are the consistency guarantees for reading configuration in case of network partitions? consul uses Raft[1] distributed consensus algorithm internally for maintaining consistency. The Raft consensus algorithm is proven to be correct. I will be going through the workings of the algorithm soon. I will share my answers to the above questions after that. Thanks for the questions, it is important for the user to understand the behaviour of a system especially under failure. I am considering adding a FAQ section to this proposal, where questions like the above would go, once it gets accepted and makes it to the feature page. [1] - https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf The following article provides some results on how Consul works following partitioning, actually testing whether it recovers successfully: http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul It gives Consul a positive review. HTH ~KP ___ Gluster-users mailing list gluster-us...@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Jonathan Barber jonathan.bar...@gmail.com ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Bulk of current GlusterD code deals with keeping the configuration of the cluster and the volumes in it consistent and available across the nodes. The current algorithm is not scalable (N^2 in no. of nodes) and doesn't prevent split-brain of configuration. This is the problem area we are targeting for the first phase. As part of the first phase, we aim to delegate the distributed configuration store. We are exploring consul [1] as a replacement for the existing distributed configuration store (sum total of /var/lib/glusterd/* across all nodes). Consul provides distributed configuration store which is consistent and partition tolerant. By moving all Gluster related configuration information into consul we could avoid split-brain situations. Did you get a chance to go over the following questions while making the decision? If yes could you please share the info. What are the consistency guarantees for changing the configuration in case of network partitions? specifically when there are 2 nodes and 1 of them is not reachable? consistency guarantees when there are more than 2 nodes? What are the consistency guarantees for reading configuration in case of network partitions? consul uses Raft[1] distributed consensus algorithm internally for maintaining consistency. The Raft consensus algorithm is proven to be correct. I will be going through the workings of the algorithm soon. I will share my answers to the above questions after that. Thanks for the questions, it is important for the user to understand the behaviour of a system especially under failure. I am considering adding a FAQ section to this proposal, where questions like the above would go, once it gets accepted and makes it to the feature page. [1] - https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf ~KP ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
- Original Message - As part of the first phase, we aim to delegate the distributed configuration store. We are exploring consul [1] as a replacement for the existing distributed configuration store (sum total of /var/lib/glusterd/* across all nodes). Consul provides distributed configuration store which is consistent and partition tolerant. By moving all Gluster related configuration information into consul we could avoid split-brain situations. Overall, I like the idea. But I think you knew that. ;) Thanks. I am glad you like it :-) Is the idea to run consul on all nodes as we do with glusterd, or to run it only on a few nodes (similar to Ceph's mon cluster) and then use them to coordinate membership etc. for the rest? It is not set in stone, but I think we should have consul running only on a subset of the nodes in the cluster, similar to Ceph's mon cluster approach. ~KP ___ Gluster-users mailing list gluster-us...@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
On 09/05/2014 03:51 PM, Kaushal M wrote: GlusterD performs the following functions as the management daemon for GlusterFS: - Peer membership management - Maintains consistency of configuration data across nodes (distributed configuration store) - Distributed command execution (orchestration) - Service management (manage GlusterFS daemons) - Portmap service for GlusterFS daemons This proposal aims to delegate the above functions to technologies that solve these problems well. We aim to do this in a phased manner. The technology alternatives we would be looking for should have the following properties, - Open source - Vibrant community - Good documentation - Easy to deploy/manage This would allow GlusterD's architecture to be more modular. We also aim to make GlusterD's architecture as transparent and observable as possible. Separating out these functions would allow us to do that. Bulk of current GlusterD code deals with keeping the configuration of the cluster and the volumes in it consistent and available across the nodes. The current algorithm is not scalable (N^2 in no. of nodes) and doesn't prevent split-brain of configuration. This is the problem area we are targeting for the first phase. As part of the first phase, we aim to delegate the distributed configuration store. We are exploring consul [1] as a replacement for the existing distributed configuration store (sum total of /var/lib/glusterd/* across all nodes). Consul provides distributed configuration store which is consistent and partition tolerant. By moving all Gluster related configuration information into consul we could avoid split-brain situations. Did you get a chance to go over the following questions while making the decision? If yes could you please share the info. What are the consistency guarantees for changing the configuration in case of network partitions? specifically when there are 2 nodes and 1 of them is not reachable? consistency guarantees when there are more than 2 nodes? What are the consistency guarantees for reading configuration in case of network partitions? Pranith All development efforts towards this proposal would happen in parallel to the existing GlusterD code base. The existing code base would be actively maintained until GlusterD-2.0 is production-ready. This is in alignment with the GlusterFS Quattro proposals on making GlusterFS scalable and easy to deploy. This is the first phase ground work towards that goal. Questions and suggestions are welcome. ~kaushal [1] : http://www.consul.io/ ___ Gluster-users mailing list gluster-us...@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
On 09/06/2014 05:55 PM, Pranith Kumar Karampuri wrote: On 09/05/2014 03:51 PM, Kaushal M wrote: GlusterD performs the following functions as the management daemon for GlusterFS: - Peer membership management - Maintains consistency of configuration data across nodes (distributed configuration store) - Distributed command execution (orchestration) - Service management (manage GlusterFS daemons) - Portmap service for GlusterFS daemons This proposal aims to delegate the above functions to technologies that solve these problems well. We aim to do this in a phased manner. The technology alternatives we would be looking for should have the following properties, - Open source - Vibrant community - Good documentation - Easy to deploy/manage This would allow GlusterD's architecture to be more modular. We also aim to make GlusterD's architecture as transparent and observable as possible. Separating out these functions would allow us to do that. Bulk of current GlusterD code deals with keeping the configuration of the cluster and the volumes in it consistent and available across the nodes. The current algorithm is not scalable (N^2 in no. of nodes) and doesn't prevent split-brain of configuration. This is the problem area we are targeting for the first phase. As part of the first phase, we aim to delegate the distributed configuration store. We are exploring consul [1] as a replacement for the existing distributed configuration store (sum total of /var/lib/glusterd/* across all nodes). Consul provides distributed configuration store which is consistent and partition tolerant. By moving all Gluster related configuration information into consul we could avoid split-brain situations. Did you get a chance to go over the following questions while making the decision? If yes could you please share the info. What are the consistency guarantees for changing the configuration in case of network partitions? specifically when there are 2 nodes and 1 of them is not reachable? consistency guarantees when there are more than 2 nodes? What are the consistency guarantees for reading configuration in case of network partitions? Consul documentation claims that it can recover from network partition. http://www.consul.io/docs/internals/jepsen.html However having said that we are yet to do this POC. ~Atin Pranith All development efforts towards this proposal would happen in parallel to the existing GlusterD code base. The existing code base would be actively maintained until GlusterD-2.0 is production-ready. This is in alignment with the GlusterFS Quattro proposals on making GlusterFS scalable and easy to deploy. This is the first phase ground work towards that goal. Questions and suggestions are welcome. ~kaushal [1] : http://www.consul.io/ ___ Gluster-users mailing list gluster-us...@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
On 05/09/2014, at 11:21 AM, Kaushal M wrote: snip As part of the first phase, we aim to delegate the distributed configuration store. We are exploring consul [1] Does this mean we'll need to learn Go as well as C and Python? If so, that doesn't sound completely optimal. :/ That being said, a lot of distributed/networked computing projects seem to be written in it these days. Is Go specifically a good language for our kind of challenges, or is it more a case of the new shiny? + Justin -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
On 5 Sep 2014, at 12:21, Kaushal M kshlms...@gmail.com wrote: - Peer membership management - Maintains consistency of configuration data across nodes (distributed configuration store) - Distributed command execution (orchestration) - Service management (manage GlusterFS daemons) - Portmap service for GlusterFS daemons Isn't some of this covered by crm/corosync/pacemaker/heartbeat? Marcus -- Marcus Bointon Technical Director, Synchromedia Limited Creators of http://www.smartmessages.net/ UK 1CRM solutions http://www.syniah.com/ mar...@synchromedia.co.uk | http://www.synchromedia.co.uk/ signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
- Original Message - On 05/09/2014, at 11:21 AM, Kaushal M wrote: snip As part of the first phase, we aim to delegate the distributed configuration store. We are exploring consul [1] Does this mean we'll need to learn Go as well as C and Python? If so, that doesn't sound completely optimal. :/ consul is written in Go. I don't think that forces us to write our code in Go. That being said, a lot of distributed/networked computing projects seem to be written in it these days. Is Go specifically a good language for our kind of challenges, or is it more a case of the new shiny? I would prefer to write new code in a language other than C. But that's me. I haven't written a lot of Go to comment :-( ~KP ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
- Original Message - On 5 Sep 2014, at 12:21, Kaushal M kshlms...@gmail.com wrote: - Peer membership management - Maintains consistency of configuration data across nodes (distributed configuration store) - Distributed command execution (orchestration) - Service management (manage GlusterFS daemons) - Portmap service for GlusterFS daemons Isn't some of this covered by crm/corosync/pacemaker/heartbeat? Maybe, I haven't explored the above technologies. consul can be used for all the different functions that glusterd performs today. consul has service discovery and leadership algorithms for executing commands in the cluster, in its 'toolbox'. Portmap service can be implemented as a dictionary with glusterfs-server processes as the key and their ports as the value. We can build further sophistication as we find use for. ~KP ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Proposal for GlusterD-2.0
Isn't some of this covered by crm/corosync/pacemaker/heartbeat? Sorta, kinda, mostly no. Those implement virtual synchrony, which is closely related to consensus but not quite the same even in a formal CS sense. In practice, using them is *very* different. Two jobs ago, I inherited a design based on the idea that if everyone starts at the same state and handles the same messages in the same order (in that case they were using Spread) then they'd all stay consistent. Sounds great in theory, right? Unfortunately, in practice it meant that returning a node which had missed messages to a consistent state was our problem, and it was an unreasonably complex one. Debugging failure-during-recovery problems in that code was some of the least fun I ever had at that job. A consensus protocol, with its focus on consistency of data rather than consistency of communication, seems like a better fit for what we're trying to achieve. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
