Cluster deployment and management are important things to make our work finally adopted by users. From my point of view I think there are two things we can improve further: * user friendly cluster auto deployment (we have puppet recipes, but users still need to manually ship puppet code and configurations) * topology layout (for example, specify zookeeper to be deployed on node 3~5)
BIGTOP-1702 has a great potential to make cluster deployment more user friendly. That is users don't need to deal with those trivial things like ship bigtop puppet code and hiera configurations or install those prerequisites all over the cluster. To answer cos's question in BIGTOP-1702: although I am not sure what you mean by link those managed servers for further provisioning According to the readme of vagrant-managed-servers, vagrant up and destroy are re-interpreted as "linking//unlinking". After those servers are linked, theoretically, we can directly leverage our Vagrantfile in "vagrant-puppet-vm" to do the whole cluster deployment by one-click. So, this definitely will take advantage of existing puppet recipes. And here's my thought on following questions: - how that topology will be specified? - how it is going to be compatible with what we have right now? Not like I try to preserve what we have right now, as there's not much to preserve. We don't really have a way to desribe a cluster's topology at the moment anyway. - in case we need to describe more complex topology - how would be done? I don't think BIGTOP-1702 support topology settings. Regarding to the feature, I think we should support topology specification in our puppet recipes. We currently do not support this yet, but I believe Michael is heading toward this way. 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <c...@apache.org>: > I think it is a great idea! > > And, indeed, it's just a very small step - make it user-friendly ;) > > I believe what you're proposing below is a much better way to achieve what > I've been trying by wrapping Puppet and content distribution into Gradle > (you > referred to it 'our own deployer' below). That was a bad idea, in > retrospect. > And for sure - we need the functionality of this kind: we need something > that > let anyone to deploy a cluster to a pre-provisioned nodes quickly without > additional troubles! > > I am also a big fan of getting rid of lengthy and at times cryptic pages > explaining the installation process. > > Now, the solution you have in mind will be helping with a typical cluster > topology like we are usually setting up with head_node, and other flat > things > of this nature, right? In other words - a simple, "traditional" cluster > layout? If so, may I ask a few questions: > - how that topology will be specified? > - how it is going to be compatible with what we have right now? Not like I > try to preserve what we have right now, as there's not much to preserve. > We don't really have a way to desribe a cluster's topology at the > moment anyway. > - in case we need to describe more complex topology - how would be done? > > I also presume, that the proposed solution will take advantage of existing > Puppet recipes, right? > > Thanks in advance for your answers! > Cos > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote: > > Hi folks. > > > > This last year we've made a lot of progress, > > - we can now easily build bigtop, > > - and we can test a fresh build in output/ it in vagrant boxes locally > > > > Whats next? > > > > In my mind, there is only one last step, to make bigtop a consumer facing > > product: We need to make it so you can simply provision a cluster in the > > cloud. > > As you may recall, David (student at RPI, intern at analytics company > last > > summer), asked how to do this, and spent about three weeks on it. > Recently > > Ata Turk, working on the Massachusets Open Cloud @ Boston University, and > > former Yahoo researchers, also asked me the same thing. > > > > Alternative? > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so on, > > which mostly are an untested, human readable version of our vagrant > > recipes. yikes ! > > > > How to implement a ssh cloud bigtop installation ? > > > > We could roll our own deployer, but in doing so, we would have to make > > semantics for: > > > > - possible need for machiene reboots > > - syncing local folders to remote folders > > - installing (and reinstalling) > > > > luckily, our buddy Vagrant already does this for us (respectively , with > > the "reload","synced.folder", and "up" options). > > > > Additionally, *** I THINK *** we can literally use the exact same vagrant > > recipes which we are already using to test --- so we will have a great > user > > experience, and really easy to reproduce bugs and test deployments. > > > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions, > suggestions > > on the implementation or feature? I think it will be easy, but also, it > > will be one of the most powerful things we add to bigtop, basically > > allowing users to easily use the system... maybe even too easy :) > > > > -- > > jay vyas >
