Re: Puppet modules for Ceph
On Tue, Aug 7, 2012 at 6:51 AM, Jonathan D. Proulx j...@csail.mit.edu wrote: :Juju seems to provide a real-time notification mechanism between :peers, using it's name-relation-changed hook. Other CM frameworks :may need to step up their game, or be subject to the keep re-running :chef-client until it works limitation. Have you lloked at puppet kick puppet-kick(8) -- Remotely control puppet agent I haven't really used puppet, but that doesn't look like it's peer to peer, the manpage talks about reading the master's config file. I may be wrong in that. The other nice thing Juju is let you transfer data between the peers in a relationship -- e.g. we could transport the crypto secret to a new OSD that way. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On Tue, Jul 31, 2012 at 12:09:13PM -0700, Tommi Virtanen wrote: :Juju seems to provide a real-time notification mechanism between :peers, using it's name-relation-changed hook. Other CM frameworks :may need to step up their game, or be subject to the keep re-running :chef-client until it works limitation. Have you lloked at puppet kick puppet-kick(8) -- Remotely control puppet agent SYNOPSIS Trigger a puppet agent run on a set of hosts. I'm not using it in my environment and it does require some specific puppet config on the agent side to make it work so less than ideal, but might atleast provide a path. -Jon -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On Tue, Jul 24, 2012 at 6:15 AM, loic.dach...@enovance.com wrote: Note that if puppet client was run on nodeB before it was run on nodeA, all three steps would have been run in sequence instead of being spread over two puppet client invocations. Unfortunately, even that is not enough. The relevant keys (client.admin, client.bootstrap-osd, later bootstrap-mds radosgw etc also) can only be created once the mons have reached quorum. This is some time after they have started, even in the best case. Making the puppet/chef run wait for that sounds like a bad idea; especially since I use further chef-client runs to feed ceph-mon information about its peers, which may be necessary for it to ever reach quorum. While I can find ways of making the key generation happen as soon as quorum is reached, communicating the keys to other nodes only happens at the mercy of the configuration management system; both puppet and chef seem to be in the mindset of run every N minutes option. So even if we generate the keys best case 2 seconds after ceph-mon startup, it needs a full configuration manager run on the source node, and then a run on the destination node, before OSD bring-up etc can succeed. I have found no satisfying solution to this, so far. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On Tue, 31 Jul 2012, Tommi Virtanen wrote: On Tue, Jul 24, 2012 at 6:15 AM, loic.dach...@enovance.com wrote: Note that if puppet client was run on nodeB before it was run on nodeA, all three steps would have been run in sequence instead of being spread over two puppet client invocations. Unfortunately, even that is not enough. The relevant keys (client.admin, client.bootstrap-osd, later bootstrap-mds radosgw etc also) can only be created once the mons have reached quorum. This is some time after they have started, even in the best case. Making the puppet/chef run wait for that sounds like a bad idea; especially since I use further chef-client runs to feed ceph-mon information about its peers, which may be necessary for it to ever reach quorum. While I can find ways of making the key generation happen as soon as quorum is reached, communicating the keys to other nodes only happens at the mercy of the configuration management system; both puppet and chef seem to be in the mindset of run every N minutes option. So even if we generate the keys best case 2 seconds after ceph-mon startup, it needs a full configuration manager run on the source node, and then a run on the destination node, before OSD bring-up etc can succeed. I have found no satisfying solution to this, so far. It is also possible to feed initial keys to the monitors during the 'mkfs' stage. If the keys can be agreed on somehow beforehand, then they will already be in place when the initial quorum is reached. Not sure if that helps in this situation or not... sage -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On Tue, Jul 31, 2012 at 11:51 AM, Sage Weil s...@inktank.com wrote: It is also possible to feed initial keys to the monitors during the 'mkfs' stage. If the keys can be agreed on somehow beforehand, then they will already be in place when the initial quorum is reached. Not sure if that helps in this situation or not... Yeah, we're going that way for the mon. key in the chef cookbooks (to get the mons talking to each other at all, that *has* to be done that way), but putting more and more stuff there is not very nice. Your typical CM framework does not let the recipe run arbitrary code at that sort of an instantiation time, and pushing this work on the admin makes it laborous and brittle; what happens when we need a new type of a bootstrap-foo key? Get all admins to cram an extra entry into their environment json? http://ceph.com/docs/master/config-cluster/chef/#configure-your-ceph-environment That just does not seem like a good way. Juju seems to provide a real-time notification mechanism between peers, using it's name-relation-changed hook. Other CM frameworks may need to step up their game, or be subject to the keep re-running chef-client until it works limitation. If the CM makes it safe to trigger a run manually (e.g. sudo chef-client whenever you feel like it), we can trigger that locally when we finally create the keys. This still doesn't help the receiving side to notice any faster. If the CM makes it safe for us to change node attributes outside of the full CM run, we can do trigger that when we finally create the keys. Chef seems to have a full overwrites only semantic, so this is probably not safe with it. And as above, this does not help the receiving side to notice that it has information to fetch. What I want to do longer term, is make the Chef cookbook for Ceph very thin, push everything except the cross-node communication into Ceph proper, and then write a mkcephfs v2.0 that uses SSH connections as appropriate, from a central workstation host that can SSH anywhere, to trigger these actions ASAP. Then that becomes the goal for CM frameworks: provide me a communication mechanism between these nodes that can do *this*. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
Hi Francois On 18 Jul 2012, at 14:58, François Charlier wrote: Hi, I'm currently working on writing a Puppet module for Ceph. As after some research I found no existing module, I'll start from scratch but I would be glad to hear from people who would already have started working or this or having any idea or pointers regarding this subject. Thanks, +1 for puppet modules. I'm also interested in the possibility of puppet modules for ceph, we've been looking at evaluating ceph here at our site and one of the things we'd like to do is automate deployments in vagrant vm's for testing/development. Thanks, Jimmy. -- Senior Software Engineer, Digital Repository of Ireland (DRI) Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. http://www.tchpc.tcd.ie/ -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On 7/18/12 8:58 AM, François Charlier wrote: Hi, I'm currently working on writing a Puppet module for Ceph. As after some research I found no existing module, I'll start from scratch but I would be glad to hear from people who would already have started working or this or having any idea or pointers regarding this subject. Thanks, [ By the way, I'm fc on #ceph ! ] Hi Francois, That's great! You might want to look at the chef work that has been done as a base to start from. I'm not very familiar with what is in place, but Tommi or Dan may chime in later with more details. Some of the folks from Mediawiki were actually just talking about puppet modules yesterday on the IRC channel so they may be interested in collaborating too. Thanks, Mark -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On Wed, Jul 18, 2012 at 6:58 AM, François Charlier francois.charl...@enovance.com wrote: I'm currently working on writing a Puppet module for Ceph. As after some research I found no existing module, I'll start from scratch but I would be glad to hear from people who would already have started working or this or having any idea or pointers regarding this subject. Hi. I don't remember anyone actively working on puppet modules for Ceph. A quick search gives me just this: http://git.sans.ethz.ch/?p=puppet-modules/ceph;a=summary The Chef cookbook at https://github.com/ceph/ceph-cookbooks is starting to get into a pretty good stage. It radically changes how we do deployment and management, so I'd recommend you look at it in detail, and don't imitate mkcephfs. We've been actively changing core Ceph to make deployment and management simpler; I think the best proof of that is that the cookbook is already shorter than the mkcephfs shell script, and will probably just become a thinner layer in the future. The Juju charms for Ceph are also adopting a model quite close to what the Chef cookbook does. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On Wed, Jul 18, 2012 at 2:59 PM, Teyo Tyree t...@puppetlabs.com wrote: As you probably know, Puppet Labs is based in Portland. Are you attending OScon? It might be a good opportunity for us to have some face to face hacking time on a Puppet module. Let me know if you would like for us to arrange sometime to get together if you are in town. Sorry, I'm not at OScon, I'm intentionally limiting my travel right now. A large chunk of Inktank is based in Los Angeles, so we're not far away even outside of conferences. Frankly, we still have a bit of cleanup work to do on the chef cookbook side, and you'd probably be most productive writing puppet modules once that stuff is all flushed out. Soon, we'll start to de-emphasize mkcephfs in favor of other, more flexible, deployment mechanisms; I think bringing together some Puppet, Juju and Chef experts at that point would be most beneficial. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Puppet modules for Ceph
On Wed, Jul 18, 2012 at 3:26 PM, Teyo Tyree t...@puppetlabs.com wrote: Ha, that would be an interesting experiment indeed. I think Francois would like to have the Puppet module done sooner rather than later. Are the current Chef cookbooks functional enough for us to get started with them as a reference? I think so. They Work For Me(tm). The ugly stuff is mostly things like needing to wait a few rounds due to Chef's asynchronous data store, and it's missing just about all internal documentation; the user-visible aspects have a decent write-up, but nothing explains e.g. what exactly /var/lib/ceph/bootstrap-osd/ is about. I'd love to help you work through that though, so please keep talking to me and make me explain everything in enough detail. It's just that I don't have anything except source to give you right now. The end user Chef deployment docs are at http://ceph.com/docs/master/install/chef/ http://ceph.com/docs/master/config-cluster/chef/ The cookbook is at https://github.com/ceph/ceph-cookbooks and currently it assumes Ubuntu 12.04. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
