Re: Trident Kafka Spout - Ack count increasing even though no messages are processed
Im trying to deploy storm on AWS but I cannot create ~/.pallet/config.clj Any suggestion would be helpful. On Sat, Sep 27, 2014 at 10:20 PM, Nathan Marz nat...@nathanmarz.com wrote: Trident executes a batch every 500ms (by default). A batch involves a bunch of coordination messages going out to all the bolts to coordinate the batch (even if the batch is empty). So that's what you're seeing. On Fri, Sep 26, 2014 at 12:32 PM, Deepak Subhramanian deepak.subhraman...@gmail.com wrote: Hi, I deployed a basic trident topology in HDP storm cluster. I am using TransactionalTridentKafkaSpout to process messages from Kafka Queue . For some reason the ack count in Storm GUI in spout and bolt increases by 20 every second even if there is no messages in Kafka queue . Is there any reason ack is called by Trident even though no messages are getting processed Thanks, D -- Twitter: @nathanmarz http://nathanmarz.com
storm-deploy
I am trying to deploy storm on cloud (AWS). After running lein-deps inside storm-deploy folder it shows me this: java.io.FileNotFoundException: /home/ec2-user/storm-deploy/target/stale/extract- native.dependencies (No such file or directory) at java.io.FileOutputStream.open (FileOutputStream.java:-2) java.io.FileOutputStream.init (FileOutputStream.java:221) clojure.java.io/fn (io.clj:230) clojure.java.io$fn__8641$G__8604__8648.invoke (io.clj:69) clojure.java.io/fn (io.clj:166) clojure.java.io$fn__8654$G__8608__8661.invoke (io.clj:69) clojure.java.io$writer.doInvoke (io.clj:119) clojure.lang.RestFn.invoke (RestFn.java:410) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.RestFn.applyTo (RestFn.java:132) clojure.core$apply.invoke (core.clj:626) clojure.core$spit.doInvoke (core.clj:6403) clojure.lang.RestFn.invoke (RestFn.java:425) leiningen.core.classpath$when_stale.doInvoke (classpath.clj:78) clojure.lang.RestFn.invoke (RestFn.java:580) leiningen.core.classpath$resolve_dependencies.doInvoke (classpath.clj:357) clojure.lang.RestFn.invoke (RestFn.java:425) leiningen.deps$deps.invoke (deps.clj:135) leiningen.deps$deps.invoke (deps.clj:114) clojure.lang.Var.invoke (Var.java:379) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.Var.applyTo (Var.java:700) clojure.core$apply.invoke (core.clj:626) leiningen.core.main$partial_task$fn__6071.doInvoke (main.clj:253) clojure.lang.RestFn.invoke (RestFn.java:410) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.RestFn.applyTo (RestFn.java:132) clojure.lang.AFunction$1.doInvoke (AFunction.java:29) clojure.lang.RestFn.applyTo (RestFn.java:137) clojure.core$apply.invoke (core.clj:626) leiningen.core.main$apply_task.invoke (main.clj:303) leiningen.core.main$resolve_and_apply.invoke (main.clj:309) leiningen.core.main$_main$fn__6136.invoke (main.clj:377) leiningen.core.main$_main.doInvoke (main.clj:366) clojure.lang.RestFn.invoke (RestFn.java:408) clojure.lang.Var.invoke (Var.java:379) clojure.lang.AFn.applyToHelper (AFn.java:154) clojure.lang.Var.applyTo (Var.java:700) clojure.core$apply.invoke (core.clj:624) clojure.main$main_opt.invoke (main.clj:315) clojure.main$main.doInvoke (main.clj:420) clojure.lang.RestFn.invoke (RestFn.java:436) clojure.lang.Var.invoke (Var.java:388) clojure.lang.AFn.applyToHelper (AFn.java:160) clojure.lang.Var.applyTo (Var.java:700) clojure.main.main (main.java:37)
Re: Apache Storm Graduation to a TLP
Congrats! On Mon, Sep 22, 2014 at 11:41 PM, Guillaume Conte guilla...@keyade.com wrote: +1 Congrats! From: Corey Nolet cjno...@gmail.com Reply-To: user@storm.incubator.apache.org user@storm.incubator.apache.org Date: Monday, September 22, 2014 at 11:33 PM To: user@storm.incubator.apache.org user@storm.incubator.apache.org Cc: d...@storm.incubator.apache.org d...@storm.incubator.apache.org Subject: Re: Apache Storm Graduation to a TLP Congrats! On Mon, Sep 22, 2014 at 5:16 PM, P. Taylor Goetz ptgo...@gmail.com wrote: I’m pleased to announce that Apache Storm has graduated to a Top-Level Project (TLP), and I’d like to thank everyone in the Storm community for your contributions and help in achieving this important milestone. As part of the graduation process, a number of infrastructure changes have taken place: *New website url:* http://storm.apache.org *New git repo urls:* https://git-wip-us.apache.org/repos/asf/storm.git (for committer push) g...@github.com:apache/storm.git -or- https://github.com/apache/storm.git (for github pull requests) *Mailing Lists:* If you are already subscribed, you’re subscription has been migrated. New messages should be sent to the new address: [list]@storm.apache.org This includes any subscribe/unsubscribe requests. Note: The mail-archives.apache.org site will not reflect these changes until October 1. Most of these changes have already occurred and are seamless. Please update your git remotes and address books accordingly. - Taylor
Re: Debug storm
Hi Guys, I have been experiencing some problems running the examples provided in the storm-starter folder. It keeps showing Dependency problems with pom file. Cannot fetch snapshot 0.9.3 from the maven2 repository.I would appreciate your help. Thanks in advance Sent from Samsung Mobile. div Original message /divdivFrom: Vikas Agarwal vi...@infoobjects.com /divdivDate:19/09/2014 20:04 (GMT+01:00) /divdivTo: user@storm.incubator.apache.org /divdivSubject: Re: Debug storm /divdiv /divHave you tried local cluster mode? On Fri, Sep 19, 2014 at 11:17 PM, Jiang Jacky jiang0...@gmail.com wrote: Hi, Guys Is there any tools to debug storm? I can directly trigger the spout and bolt, and put the break point in my project. For now, I have to do it by unit testing for each function from testNG. If there is directly tool to startup with topology in Intellij or Eclipse, then I am able to trace my code step by step. Thanks -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
Re: metrics consumer logging stormUI data
Hey Otis, I was just registered at sematext and I can say that this is what I have been looking for.I have just one question, what about the delays between the SPM and the Storm Cluster (if they do exist), whats the worst case? I mean because these metrics are not calculated locally, but using an internet connection. Thanks ! On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Raphael, Not sure if this is what you are after, but SPM http://sematext.com/spm/ will collect and graph all Storm metrics, let you do alerting and anomaly detection on them, etc. If you want to graph custom metrics (e.g. something from your bolts), you can send them in as custom metrics https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics and again graph them, alert on them, do anomaly detection on them, stick them on dashboards, etc. If you want to emit events from your bolts, you can send events to SPM https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration, too, or you can send them to Logsene http://www.sematext.com/logsene/... can be handy for correlation with alerts and performance graphs when troubleshooting. Here are some Storm metrics graph: http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/ I hope this helps. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh raffihs...@gmail.com wrote: Hi, Using Storm/Trident, how do I register a metrics consumer to log the data I get in the StormUI ? I want to look at historical data of my topology, for example the execute latency of the topology over time, as this would give me good insight as to where things might be going wrong when the system breaks. I have been following the steps outlined in the BigData CookBook here: http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to However I am not wanting to create my own metrics, instead I just want to log the metrics that already exist built in to Storm. It is unclear to me how I am supposed to go about doing that. Thanks -- Raphael Hsieh
Trying to run test Storm App on Windows but getting problems with POM file
[INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
Re: muliple-nodes kafka cluster
[INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] Reactor Summary: [INFO] [INFO] Storm .. SUCCESS [ 2.816 s] [INFO] maven-shade-clojure-transformer SUCCESS [ 30.954 s] [INFO] Storm Core . SUCCESS [01:34 min] [INFO] storm-starter .. FAILURE [ 0.128 s] [INFO] storm-kafka SKIPPED [INFO] storm-hdfs . SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:08 min [INFO] Finished at: 2014-09-16T23:51:57+02:00 [INFO] Final Memory: 39M/411M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :storm-starter On Tue, Sep 16, 2014 at 11:46 PM, Harsha st...@harsha.io wrote: Hi Alec, Single node kafka cluster not recommended apart from using it for development. I highly recommend using multinode cluster and create a partitioned topic with replication. This not only makes it optimal to take in more data at faster rates also allows your cluster running if there is a node failure as the topic is replicated there wouldn't be huge data loss. If I am using multiple-nodes, the tradeoff is the connection time among different nodes? kafka producer api sends a message to broker either round-robin or based on partition function. please go through the kafka docs here http://kafka.apache.org/documentation.html for simple consumer and also how the replication works among multiple nodes. -Harsha On Tue, Sep 16, 2014, at 02:06 PM, Sa Li wrote: Hi, All I have been using kafka cluster in single server with three brokers, but I am thinking to build a larger kafka cluster, say 4 nodes (server), and 3 brokers in each node, so totally 12 brokers, would that be better than single node cluster? Or single node will be fair enough, since web api may push million rows into kafka cluster every day, I am kinda worry if the cluster is capable to take such much data without losing data. If I am using multiple-nodes, the tradeoff is the connection time among different nodes? thanks Alec
Re: Trying to run test Storm App on Windows but getting problems with POM file
I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for project org.apache.storm:storm-starter:storm-version-SNAPSHOT. # Must be run from the top-level directory of the Storm code repository $ mvn clean install -DskipTests=true This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/. When you run the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version of Storm in this local Maven repository at $HOME/.m2/repository. From https://github.com/apache/incubator-storm/tree/master/examples/storm-starter On Wed, Sep 17, 2014 at 12:02 AM, Nick Beenham nick.been...@gmail.com wrote: I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
Re: Trying to run test Storm App on Windows but getting problems with POM file
Yes, Im running it from top level dir (incubator-storm) with the following command: mvn clean install -DskipTests=true and still cant get it to work properly, I have tried to build it from the storm-starter and still the same, keeps telling me there is an error with dependencies. On Wed, Sep 17, 2014 at 1:45 AM, Harsha st...@harsha.io wrote: Hi , Did you changed storm/pom.xml groupIdorg.apache.storm/groupId artifactIdstorm/artifactId version0.9.3-incubating-SNAPSHOT/version and also are your running mvn install from top-level dir not from storm-starter. -Harsha On Tue, Sep 16, 2014, at 03:12 PM, Gezim Musliaj wrote: I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for project org.apache.storm:storm-starter:storm-version-SNAPSHOT. # Must be run from the top-level directory of the Storm code repository $ mvn clean install -DskipTests=true This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/. When you run the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version of Storm in this local Maven repository at $HOME/.m2/repository. From https://github.com/apache/incubator-storm/tree/master/examples/storm-starter On Wed, Sep 17, 2014 at 12:02 AM, Nick Beenham nick.been...@gmail.com wrote: I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
Re: Trying to run test Storm App on Windows but getting problems with POM file
Finally I solved the problem, I had tried building storm-starter before the main storm project. I just removed the previous folder with all of its files, extracted a new one from the incubator-storm.rar and did the build succesfully. But when I make the eclipse project files via the command mvn eclipse:eclipse inside the storm-starter folder and then I open it on Eclipse IDE Indigo I get errors like Class missing, Fix Project Setup etc. I figured it out that in the build path were a bunch of M2_HOME .jar files but I dont know if the problem has to do with the ordering or anything else. On Wed, Sep 17, 2014 at 1:57 AM, Gezim Musliaj gmusl...@gmail.com wrote: Yes, Im running it from top level dir (incubator-storm) with the following command: mvn clean install -DskipTests=true and still cant get it to work properly, I have tried to build it from the storm-starter and still the same, keeps telling me there is an error with dependencies. On Wed, Sep 17, 2014 at 1:45 AM, Harsha st...@harsha.io wrote: Hi , Did you changed storm/pom.xml groupIdorg.apache.storm/groupId artifactIdstorm/artifactId version0.9.3-incubating-SNAPSHOT/version and also are your running mvn install from top-level dir not from storm-starter. -Harsha On Tue, Sep 16, 2014, at 03:12 PM, Gezim Musliaj wrote: I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for project org.apache.storm:storm-starter:storm-version-SNAPSHOT. # Must be run from the top-level directory of the Storm code repository $ mvn clean install -DskipTests=true This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/. When you run the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version of Storm in this local Maven repository at $HOME/.m2/repository. From https://github.com/apache/incubator-storm/tree/master/examples/storm-starter On Wed, Sep 17, 2014 at 12:02 AM, Nick Beenham nick.been...@gmail.com wrote: I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
