On May 1, 2013, at 7:27 AM, Dharmesh Kakadia <dharmesh.kaka...@research.iiit.ac.in> wrote:
> Sebastien and Chiradeep, thanks for the comments !! That clarified a lot of > things. I just read Chiradeep's blog ( > http://cloudierthanthou.wordpress.com/2013/04/26/stackmate-execute-cloudformation-templates-on-cloudstack/) > which details the service. > > I am proposing a server side implementation of cloudformation. > > I misunderstood the ReST and Query API. Thanks for correcting. Information > here(http://gehrcke.de/2009/06/aws-about-api/) helped me. In case we want > to use existing AWS tools for cloudformation, we also would be designing > Query API, not ReST. > > Sorry for the confusion regarding cloudmonkey. I was proposing to integrate > cloudformation API into cloudstack source code, directly and add > corresponding support in cloudmonkey. But as you suggested, it might be > easy to start with prototype decoupled from cloudstack (Uses cloudstack API > and does not reside in cloudstack). I assume by existing cloudformation > tools you mean AWS tools( > http://aws.amazon.com/developertools/AWS-CloudFormation). Reusing them will > be a really good idea. Yes first step should be with cloud formation (stackmate) outside cloudstack. And I strongly suggest that you make sure it is compatible with the AWS tools. > > There are lot of options for configuration mgmt tools. I have used knife > previously and good to know that it has cloudstack plugin based on fog ( > https://github.com/fifthecho/knife-cloudstack-fog). Reasons rundeck looked > better was support for rollbacking and is full workflow execution engine. > Finally rundeck can use chef/puppet. I have seen provisonr/whirr and they > look promising. Definitely a lot to explore here !! Yes, I have nothing against rundeck. But it would be one more dependency. > > Thanks for suggesting clear proposal. > > Thanks, > Dharmesh > > > > > > On Wed, May 1, 2013 at 3:07 PM, Sebastien Goasguen <run...@gmail.com> wrote: > >> >> On Apr 30, 2013, at 4:59 PM, Chiradeep Vittal <chiradeep.vit...@citrix.com> >> wrote: >> >>> >>> >>> On 4/30/13 5:01 AM, "Sebastien Goasguen" <run...@gmail.com> wrote: >>> >>>> Dharmesh, see in-line >>>> >>>> On Apr 30, 2013, at 5:34 AM, Dharmesh Kakadia <dhkaka...@gmail.com> >> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am Dharmesh Kakdia and interested in project "Integration project to >>>>> deploy and use Mesos on a CloudStack based cloud" ( >>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1784) >>>>> >>>>> I am working on proposal and want to get feedback. Please provide >>>>> suggestions :) >>>>> >>>>> * >>>>> >>>>> Abstract: >>>>> >>>>> The project aims to bring cloudformation[1] like service to cloudstack. >>>>> One >>>>> of the prime use-case is cluster computing frameworks on cloudstack. A >>>>> cloudformation service will give users and administrators of cloudstack >>>>> ability to manage and control a set of resources easily. The >>>>> cloudformation >>>>> will allow booting and configuring a set of VMs and form a cluster. >>>>> Simple >>>>> example would be LAMP stack. More complex clusters such as mesos or >>>>> hadoop >>>>> cluster requires a little more advanced configuration. There is already >>>>> some work done by Chiradeep Vittal at this front [5] using route and >>>> >>>> it's using ruote: http://ruote.rubyforge.org >>>> >>>>> sinatra. In this project, I will implement cloudformation service and >>>>> demonstrate how to run mesos cluster using it. >>>> >>>> You will create cloud formation templates that describe a mesos cluster >>>> >>>>> >>>>> Mesos: >>>>> >>>>> Mesos is a resource management platform for clusters [2]. It aims to >>>>> increase resource utilization of clusters by sharing cluster resources >>>>> among multiple processing frameworks(like MapReduce, MPI, Graph >>>>> Processing) >>>>> or multiple instances of same framework. It provides efficient resource >>>>> isolation through use of containers. Uses zookeeper for state >>>>> maintenance >>>>> and fault tolerance. >>>>> >>>>> What can run on mesos ? >>>>> >>>>> Spark: A cluster computing framework based on the Resilient Distributed >>>>> Datasets (RDDs) abstraction. RDD is more generalized than MapReduce and >>>>> can >>>>> support iterative and interactive computation while retaining fault >>>>> tolerance, scalability, data locality etc. >>>>> >>>>> Hadoop: Hadoop is fault tolerant and scalable distributed computing >>>>> framework based on MapReduce abstraction. >>>>> >>>>> Begel: A graph processing framework based on pregel. >>>>> >>>>> and other frameworks like MPI, Hypertable. >>>>> >>>>> How to deploy mesos >>>>> >>>>> Mesos provides cluster installation scripts [7] for cluster deployment. >>>>> There are also scripts available to deploy a cluster on Amazon EC2 [8]. >>>> >>>> It would be nice to see if these scripts can be used as is with the >>>> CloudStack EC2 service. >>>> >>>>> >>>>> Deliverables: >>>>> >>>>> 1. Cloudformation service implementation on cloudstack. >>>>> >>>>> 2. Integration of cloudformation with cloudmonkey, CLI tool. >>>> >>>> 2. is a little confusing. I believe that what Chiradeep prototype runs >> on >>>> the client side. What is needed is a server side implementation. >>>> That way we could use existing cloudformation cli tools to talk to it. >>>> I don't understand where cloudmonkey comes into play. CloudMonkey is a >>>> cli for the CloudStack API. Unless you plan to integrate the >>>> cloudformation API directly in the cloudstack source code, the >>>> integration you propose is not clear to me. >>>> >>> >>> Sebastien is correct. I intend to put in the query API server around the >>> core of stack mate soon (as soon as I'm done helping on the internal >>> loadbalancer). This will be written in Ruby. >>> >>> >> >> Dharmesh I suggest you propose the following: >> >> 1-Deploy CloudStack and understand instance configuration/contextualization >> 2-Test and deploy Mesos on a set of CloudStack based VM, manually. >> Design/propose an automation framework. >> 3-Test stackmate and engage chiradeep (report bugs, make suggestion, make >> pull request) >> 4-Create cloud formation template to provision a Mesos Cluster >> 5-Compare with Apache Whirr or other cluster provisioning tools. >> 6-Potentially if you see a link with cloudmonkey, see how you could extend >> it to talk to stackmate in a similar manner that it talks to CloudStack. >> >> >> You are pretty close and this is a very exciting projects, so go ahead, >> modify a bit your proposal and submit it. >> >> Deadline for applications is this Friday May 3rd. >> >> -sebastien >> >> >> >>
