storm-deploy
I am trying to deploy storm on cloud (AWS). After running lein-deps inside storm-deploy folder it shows me this: java.io.FileNotFoundException: /home/ec2-user/storm-deploy/target/stale/extract- native.dependencies (No such file or directory) at java.io.FileOutputStream.open (FileOutputStream.java:-2) java.io.FileOutputStream.init (FileOutputStream.java:221) clojure.java.io/fn (io.clj:230) clojure.java.io$fn__8641$G__8604__8648.invoke (io.clj:69) clojure.java.io/fn (io.clj:166) clojure.java.io$fn__8654$G__8608__8661.invoke (io.clj:69) clojure.java.io$writer.doInvoke (io.clj:119) clojure.lang.RestFn.invoke (RestFn.java:410) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.RestFn.applyTo (RestFn.java:132) clojure.core$apply.invoke (core.clj:626) clojure.core$spit.doInvoke (core.clj:6403) clojure.lang.RestFn.invoke (RestFn.java:425) leiningen.core.classpath$when_stale.doInvoke (classpath.clj:78) clojure.lang.RestFn.invoke (RestFn.java:580) leiningen.core.classpath$resolve_dependencies.doInvoke (classpath.clj:357) clojure.lang.RestFn.invoke (RestFn.java:425) leiningen.deps$deps.invoke (deps.clj:135) leiningen.deps$deps.invoke (deps.clj:114) clojure.lang.Var.invoke (Var.java:379) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.Var.applyTo (Var.java:700) clojure.core$apply.invoke (core.clj:626) leiningen.core.main$partial_task$fn__6071.doInvoke (main.clj:253) clojure.lang.RestFn.invoke (RestFn.java:410) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.RestFn.applyTo (RestFn.java:132) clojure.lang.AFunction$1.doInvoke (AFunction.java:29) clojure.lang.RestFn.applyTo (RestFn.java:137) clojure.core$apply.invoke (core.clj:626) leiningen.core.main$apply_task.invoke (main.clj:303) leiningen.core.main$resolve_and_apply.invoke (main.clj:309) leiningen.core.main$_main$fn__6136.invoke (main.clj:377) leiningen.core.main$_main.doInvoke (main.clj:366) clojure.lang.RestFn.invoke (RestFn.java:408) clojure.lang.Var.invoke (Var.java:379) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.Var.applyTo (Var.java:700) clojure.core$apply.invoke (core.clj:624) clojure.main$main_opt.invoke (main.clj:315) clojure.main$main.doInvoke (main.clj:420) clojure.lang.RestFn.invoke (RestFn.java:436) clojure.lang.Var.invoke (Var.java:388) clojure.lang.AFn.applyToHelper (AFn.java:160) clojure.lang.Var.applyTo (Var.java:700) clojure.main.main (main.java:37)
Re: Can be storm-deploy script used for deploying storm 0.9.x version on Amazon?
Hello, Sasi! Thank your very much for your response.Due to the fact that I would like to deploy also a Storm application on Amazon, it really helps to spare time . If it possible, just as an information, do you have some measurement on how much time did take the cluster to be up and running? Best regards, Florin On Wed, Mar 26, 2014 at 8:36 PM, Sasi Panja sasi.pa...@gmail.com wrote: I was able to deploy and run 0.9.0-rc2 on EC2 using https://github.com/nathanmarz/storm-deploy After following the instructions from the wiki and updating the configuration files, the following command worked for me : *lein deploy-storm --start --name yourclustername --branch master --commit 0.9.0-rc2* I am using Leiningen 2.3.4 All the workers, master and zookkeeper servers were up and running in the cluster # CLUSTERS CONFIG FILE *under storm-deploy/conf* nimbus.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu nimbus.hardware: m1.xlarge supervisor.count: 5 supervisor.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu on us-west-2 supervisor.hardware: m1.xlarge zookeeper.count: 1 zookeeper.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu zookeeper.hardware: m1.large *config.clj under ~/.pallet * (defpallet :services { :default { :blobstore-provider aws-s3 :provider aws-ec2 :environment {:user {:username storm ; this must be storm :private-key-path /home/ubuntu/.ssh/id_rsa :public-key-path /home/ubuntu/.ssh/id_rsa.pub} :aws-user-id } :identity :credential XX :jclouds.regions us-west-2 } }) -Sasi On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin spicoflo...@gmail.comwrote: Hello! I would like to know what changes should be applied to the storm-deploy script (https://github.com/nathanmarz/storm-deploy) in order to install it on Amazon? Thank you in advance. Regards, Florin
Re: Can be storm-deploy script used for deploying storm 0.9.x version on Amazon?
current storm-deply script currently points to https://github.com/nathanmarz/storm 0.9.0.1 is the latest tag that works - Yi - Yi On Wed, Mar 26, 2014 at 6:01 AM, Spico Florin spicoflo...@gmail.com wrote: Hello! I would like to know what changes should be applied to the storm-deploy script (https://github.com/nathanmarz/storm-deploy) in order to install it on Amazon? Thank you in advance. Regards, Florin
Re: Can be storm-deploy script used for deploying storm 0.9.x version on Amazon?
Hi Florin, The cluster consisted of 1 master, 1 zookeeper, and 5 workers (test env) and it took about 10 minutes for everything to be up and view the UI console. The storm-deploy script was run from another ec2 machine running in the same zone. I was then able to run my topology in less than 2 minutes. As Marc mentioned, it might be worth to try Michael's Wirbelsturm script, which seems quite elegant and powerful from the github/blog documentation. -Sasi On Thu, Mar 27, 2014 at 4:15 AM, Marc Vaillant vaill...@animetrics.comwrote: Hi Florin, I just wanted to suggest that you also look at Wirbelsturm by Michael Noll as an alternative to storm-deploy https://github.com/miguno/wirbelsturm. I think that you will find it more complete, better documented, and more mainstream because it uses vagrant and puppet instead of pallet and jclouds. I highly recommend reading Michael's blog post about it: http://www.michael-noll.com/blog/2014/03/17/wirbelsturm-one-click-deploy-storm-kafka-clusters-with-vagrant-puppet Best, Marc On Thu, Mar 27, 2014 at 10:19:24AM +0200, Spico Florin wrote: Hello, Sasi! Thank your very much for your response.Due to the fact that I would like to deploy also a Storm application on Amazon, it really helps to spare time . If it possible, just as an information, do you have some measurement on how much time did take the cluster to be up and running? Best regards, Florin On Wed, Mar 26, 2014 at 8:36 PM, Sasi Panja sasi.pa...@gmail.com wrote: I was able to deploy and run 0.9.0-rc2 on EC2 using https://github.com/ nathanmarz/storm-deploy After following the instructions from the wiki and updating the configuration files, the following command worked for me : lein deploy-storm --start --name yourclustername --branch master --commit 0.9.0-rc2 I am using Leiningen 2.3.4 All the workers, master and zookkeeper servers were up and running in the cluster ### # # CLUSTERS CONFIG FILE under storm-deploy/conf ### # nimbus.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu nimbus.hardware: m1.xlarge supervisor.count: 5 supervisor.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu on us-west-2 supervisor.hardware: m1.xlarge zookeeper.count: 1 zookeeper.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu zookeeper.hardware: m1.large ### # config.clj under ~/.pallet ### # (defpallet :services { :default { :blobstore-provider aws-s3 :provider aws-ec2 :environment {:user {:username storm ; this must be storm :private-key-path /home/ubuntu/.ssh/ id_rsa :public-key-path /home/ubuntu/.ssh/ id_rsa.pub} :aws-user-id } :identity :credential XX :jclouds.regions us-west-2 } }) -Sasi On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin spicoflo...@gmail.com wrote: Hello! I would like to know what changes should be applied to the storm-deploy script (https://github.com/nathanmarz/storm-deploy) in order to install it on Amazon? Thank you in advance. Regards, Florin
Can be storm-deploy script used for deploying storm 0.9.x version on Amazon?
Hello! I would like to know what changes should be applied to the storm-deploy script (https://github.com/nathanmarz/storm-deploy) in order to install it on Amazon? Thank you in advance. Regards, Florin
Re: Can be storm-deploy script used for deploying storm 0.9.x version on Amazon?
I was able to deploy and run 0.9.0-rc2 on EC2 using https://github.com/nathanmarz/storm-deploy After following the instructions from the wiki and updating the configuration files, the following command worked for me : *lein deploy-storm --start --name yourclustername --branch master --commit 0.9.0-rc2* I am using Leiningen 2.3.4 All the workers, master and zookkeeper servers were up and running in the cluster # CLUSTERS CONFIG FILE *under storm-deploy/conf* nimbus.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu nimbus.hardware: m1.xlarge supervisor.count: 5 supervisor.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu on us-west-2 supervisor.hardware: m1.xlarge zookeeper.count: 1 zookeeper.image: us-west-2/ami-ca2ca4fa #64-bit ubuntu zookeeper.hardware: m1.large *config.clj under ~/.pallet * (defpallet :services { :default { :blobstore-provider aws-s3 :provider aws-ec2 :environment {:user {:username storm ; this must be storm :private-key-path /home/ubuntu/.ssh/id_rsa :public-key-path /home/ubuntu/.ssh/id_rsa.pub} :aws-user-id } :identity :credential XX :jclouds.regions us-west-2 } }) -Sasi On Wed, Mar 26, 2014 at 3:01 AM, Spico Florin spicoflo...@gmail.com wrote: Hello! I would like to know what changes should be applied to the storm-deploy script (https://github.com/nathanmarz/storm-deploy) in order to install it on Amazon? Thank you in advance. Regards, Florin
Re: Problems with Using storm-deploy on AWS
Hi Srinath, I have switched from t1.micro to m1.large, but got the same ERROR logging again. I'm using RHEL instead of ubuntu. Do you think that was the problem? Thank you for your timely reply :) Best, Shouheng On Thu, Mar 20, 2014 at 5:43 PM, Srinath C srinat...@gmail.com wrote: Yi, I haven't seen this error, but I think you definitely need a better instance than t1.micro. I was able to successfully get it up and running with m1.large after failing to bring it up on m1.small. But you could try m1.medium and see if that works for you. On Fri, Mar 21, 2014 at 3:14 AM, Shouheng Yi shouhen...@gmail.com wrote: Hi all, I'm having a lot of troubles with deploying storm on AWS. I followed everything on https://github.com/nathanmarz/storm-deploy, but still no luck. Can somebody help me with this? Here are some of my specs: - using t1.micro, RHEL 64bit (ami-a25415cb) - $ cat ~/.pallet/config.clj (defpallet :services { :default { :blobstore-provider aws-s3 :provider aws-ec2 :environment {:user {:username storm :private-key-path /home/ec2-user/.ssh/.pem :public-key-path /home/ec2-user/.ssh/.pem.pub} :aws-user-id --} :identity xx :credential xx :jclouds.regions us-east-1 } }) - $ cat ~/storm-deploy/conf/clusters.yaml nimbus.image: us-east-1/ami-a25415cb nimbus.hardware: t1.micro supervisor.count: 2 supervisor.image: us-east-1/ami-a25415cb supervisor.hardware: t1.micro zookeeper.count: 1 zookeeper.image: us-east-1/ami-a25415cb zookeeper.hardware: t1.micro Then I got these ERRORs DEBUG provision - Finished post-configure and exec phases INFO provision - Attaching to Available Cluster... ERROR logging - Exception in thread main ERROR logging - java.lang.AssertionError: Assert failed: (= (count running-nodes) 1) (form-init8570048559822528595.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - Caused by: java.lang.AssertionError: Assert failed: (= (count running-nodes) 1) ERROR logging - at backtype.storm.crate.storm$nimbus_ip.invoke(storm.clj:25) ERROR logging - at backtype.storm.crate.storm$mk_storm_yaml.invoke(storm.clj:207) ERROR logging - at backtype.storm.provision$sync_storm_conf_dir.invoke(provision.clj:47) ERROR logging - at backtype.storm.provision$attach_BANG_.invoke(provision.clj:55) ERROR logging - at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:85) ERROR logging - at backtype.storm.provision$start_BANG_.invoke(provision.clj:91) ERROR logging - at backtype.storm.provision$_main$fn__8422.invoke(provision.clj:144) ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:130) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:483) ERROR logging - at clojure.lang.Var.invoke(Var.java:381) ERROR logging - at user$eval5$fn__7.invoke(form-init8570048559822528595.clj:1) ERROR logging - at user$eval5.invoke(form-init8570048559822528595.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424) ERROR logging - ... 13 more What is wrong with it? Can someone help me? Thanks
Re: Problems with Using storm-deploy on AWS
Yi, I haven't seen this error, but I think you definitely need a better instance than t1.micro. I was able to successfully get it up and running with m1.large after failing to bring it up on m1.small. But you could try m1.medium and see if that works for you. On Fri, Mar 21, 2014 at 3:14 AM, Shouheng Yi shouhen...@gmail.com wrote: Hi all, I'm having a lot of troubles with deploying storm on AWS. I followed everything on https://github.com/nathanmarz/storm-deploy, but still no luck. Can somebody help me with this? Here are some of my specs: - using t1.micro, RHEL 64bit (ami-a25415cb) - $ cat ~/.pallet/config.clj (defpallet :services { :default { :blobstore-provider aws-s3 :provider aws-ec2 :environment {:user {:username storm :private-key-path /home/ec2-user/.ssh/.pem :public-key-path /home/ec2-user/.ssh/.pem.pub} :aws-user-id --} :identity xx :credential xx :jclouds.regions us-east-1 } }) - $ cat ~/storm-deploy/conf/clusters.yaml nimbus.image: us-east-1/ami-a25415cb nimbus.hardware: t1.micro supervisor.count: 2 supervisor.image: us-east-1/ami-a25415cb supervisor.hardware: t1.micro zookeeper.count: 1 zookeeper.image: us-east-1/ami-a25415cb zookeeper.hardware: t1.micro Then I got these ERRORs DEBUG provision - Finished post-configure and exec phases INFO provision - Attaching to Available Cluster... ERROR logging - Exception in thread main ERROR logging - java.lang.AssertionError: Assert failed: (= (count running-nodes) 1) (form-init8570048559822528595.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - Caused by: java.lang.AssertionError: Assert failed: (= (count running-nodes) 1) ERROR logging - at backtype.storm.crate.storm$nimbus_ip.invoke(storm.clj:25) ERROR logging - at backtype.storm.crate.storm$mk_storm_yaml.invoke(storm.clj:207) ERROR logging - at backtype.storm.provision$sync_storm_conf_dir.invoke(provision.clj:47) ERROR logging - at backtype.storm.provision$attach_BANG_.invoke(provision.clj:55) ERROR logging - at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:85) ERROR logging - at backtype.storm.provision$start_BANG_.invoke(provision.clj:91) ERROR logging - at backtype.storm.provision$_main$fn__8422.invoke(provision.clj:144) ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:130) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:483) ERROR logging - at clojure.lang.Var.invoke(Var.java:381) ERROR logging - at user$eval5$fn__7.invoke(form-init8570048559822528595.clj:1) ERROR logging - at user$eval5.invoke(form-init8570048559822528595.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424) ERROR logging - ... 13 more What is wrong with it? Can someone help me? Thanks
Facing Error in storm-deploy
Hi everyone, I am trying to deploy the storm on AWS cluster, and getting following error. I am using a mac machine, so these are the steps I followed: 1. downloaded lein, converted it to executable, moved to usr/local/bin and executed same. 2. did a git clone of storm deploy code. 3. cd into storm deploy and run lein deps 4. made a config.clj file as below. Also, did ssh-keygen in ~/.ssh to create a public private key pair which was used in config.clj. (defpallet :services { :default { :blobstore-provider aws-s3 :provider aws-ec2 :environment {:user {:username storm ; this must be storm :private-key-path ~/.ssh/id_rsa :public-key-path ~/.ssh/id_rsa.pub} :aws-user-id 2517} :identity :credential :jclouds.regions us-east-1 } }) Then did a: lein deploy-storm --start --name mycluster --branch 0.8.3 the error below is mentioned. I have followed the steps as described and cross checked these at multiple places, however the error remains. Any help or pointers shall be really useful, Thanks in Advance! --Gaurav INFO execute - Output: /Users/admin/.ssh/id_rsa.pub DEBUG execute - out = /Users/admin/.ssh/id_rsa.pub\n INFO execute - Output: /Users/admin/.ssh/id_rsa DEBUG execute - out = /Users/admin/.ssh/id_rsa\n INFO execute - Output: storm DEBUG execute - out = storm\n INFO execute - Output: /Users/admin/.ssh/id_rsa.pub DEBUG execute - out = /Users/admin/.ssh/id_rsa.pub\n INFO execute - Output: /Users/admin/.ssh/id_rsa DEBUG execute - out = /Users/admin/.ssh/id_rsa\n INFO execute - Output: /Users/admin/.ssh/id_rsa.pub DEBUG execute - out = /Users/admin/.ssh/id_rsa.pub\n INFO execute - Output: /Users/admin/.ssh/id_rsa DEBUG execute - out = /Users/admin/.ssh/id_rsa\n DEBUG jclouds - Found jclouds sshj driver DEBUG jclouds - extensions (:log4j :slf4j :sshj) DEBUG jclouds - options [:jclouds.regions us-east-1 :blobstore-provider aws-s3] ERROR logging - Exception in thread main ERROR logging - com.google.inject.CreationException: Guice creation errors: 1) org.jclouds.rest.RestContextorg.jclouds.aws.ec2.AWSEC2Client, A cannot be used as a key; It is not fully specified. 1 error (form-init8975416400432954481.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - Caused by: com.google.inject.CreationException: Guice creation errors: 1) org.jclouds.rest.RestContextorg.jclouds.aws.ec2.AWSEC2Client, A cannot be used as a key; It is not fully specified. 1 error ERROR logging - at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) ERROR logging - at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) ERROR logging - at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) ERROR logging - at com.google.inject.Guice.createInjector(Guice.java:95) ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:324) ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:262) ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:524) ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:504) ERROR logging - at org.jclouds.compute2$compute_service.doInvoke(compute2.clj:92) ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:147) ERROR logging - at clojure.core$apply.doInvoke(core.clj:548) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:562) ERROR logging - at pallet.compute.jclouds$eval5952$fn__5954.invoke(jclouds.clj:720) ERROR logging - at clojure.lang.MultiFn.invoke(MultiFn.java:167) ERROR logging - at pallet.compute$compute_service.doInvoke(compute.clj:36) ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:140) ERROR logging - at clojure.core$apply.invoke(core.clj:542) ERROR logging - at pallet.configure
Is ZeroMQ dependency needed in the storm-deploy cloud script when using Netty (storm 0.9.1)?
Hello! In the script used for deploying storm to cloud( https://github.com/nathanmarz/storm-deploy ) there is a dependency on ZeroMQ. If I would like to use the latest version of Storm ( https://github.com/apache/incubator-storm) with Netty support, I think this dependency should be removed. Am I right with my supposition? I look forward for your answers. Thank you. Regards, Florin