Re: [Discussion] storm local-mode event object reuse bug

Serialization isn't free. By skipping it where possible, even in a cluster, it's worth doing so to conserve CPU resources. Using immutable objects is cheaper. Assuming you're coding in java, consider using ImmutableMap, ImmutableMap.Builder, and similar classes in the Guava library from Google.

Could not find component common for null

I'm getting the following error with storm 0.9.3 . I get this error with my topology and with the word count topology. This error occurs when storm is installed standalone but not when it is installed via Ambari. I have zookeeper declared as a dependency with the same version used by storm 0.9.

Re: Storm commercial support company

MapR also seems to provide Storm support. [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.com Mobile: 865 724 4910 [http://www.cisco.com/ass

Re: Tident: fork DAG without partitioning?

ioning? Are your filters such that a tuple will pass either filterA, or filterB, but never both? --Tom On Thu, Sep 24, 2015 at 10:35 AM, Grant Overby (groverby) mailto:grove...@cisco.com>> wrote: I have a trident topology where a portion of the DAG looks like this:

Tident: fork DAG without partitioning?

I have a trident topology where a portion of the DAG looks like this: partition - - - filterA - - - eachA / stream - - - - \ partition - - - filterB - - - eachB I believe with the above DAG, each tuple will be sent down both s

Trident: Cross thread calls

Is execute() always called from the same thread? If so, is prepare() and cleanup() called from the same thread as execute? [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.

Re: SQL exception when using Maven

Try removing provided from the postgres dependency [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.com Mobile: 865 724 4910 [http://www.cisc

Re: Clarification regarding packaging of storm-core jar

I’d leave storm/lib vanilla and package the higher versions of those libs with your topology jar using maven shade to relocate the necessary packages to avoid conflict. Here is an example from a storm topology that relocates guava (com.google.common). org.apache.maven.plugins maven-s

Re: Limiting Workers to use only 1 CORE/2GB RAM

or Company Registration Information. From: "Grant Overby (groverby)" mailto:grove...@cisco.com>> Date: Wednesday, July 8, 2015 at 2:24 PM To: "user@storm.apache.org<mailto:user@storm.apache.org>" mailto:user@storm.apache.org>> Subject: Re: Limiting Worker

Re: Limiting Workers to use only 1 CORE/2GB RAM

To limit the worker process to a single cpu: http://linux.die.net/man/1/taskset To get the correct PID (replace user name as appropriate): `su - storm -c jps | grep worker | cut -d ' ' -f 1` [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Over

Re: Supervisor believes worker has not started.

Tuesday, June 2, 2015 at 11:24 AM To: "Grant Overby (groverby)" mailto:grove...@cisco.com>>, "user@storm.apache.org<mailto:user@storm.apache.org>" mailto:user@storm.apache.org>> Subject: Re: Supervisor believes worker has not started. Which version of storm are

Re: Supervisor believes worker has not started.

Same here. The worker isn’t committing suicide; its being murdered by the supervisor. [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.com Mobile

Re: [DISCUSS] Drop Support for Java 1.6 in Storm 0.10.0

We¹re using it with 1.7, so no objection. Grant Overby Software Engineer Cisco.com grove...@cisco.com Mobile: 865 724 4910 Think before you print.This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, di

Supervisor believes worker has not started.

Supervisor is reporting that the worker “still hasn’t started” and eventually kills and restarts the worker. However; the worker has started and is processing tuples. This repeats indefinitely. Debugging steps? [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398

Re: java.lang.OutOfMemoryError

t be appropriate, if the Storm management code itself does explicit Garbage Collection. I cannot recall having searched through the codebase to find if that is the case or not. From: "Grant Overby (groverby)" mailto:grove...@cisco.com>> Reply-To: "user@storm.apache.org<

Re: java.lang.OutOfMemoryError

Why -XX:+DisableExplicitGC ? [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.com Mobile: 865 724 4910 [http://www.cisco.com/assets/swa/img/

Re: java.lang.OutOfMemoryError

Nothing too obvious jumps out at me. How long are the lines? What is worker.childopts set to? [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.com

Re: java.lang.OutOfMemoryError

GC Overhead limit exceeded usually indicates you are rapidly instantiating and releasing objects to the extent that the JVM is spending too much time in GC. You could try increasing the worker heap size, but I’d take a look at your topology first to see if you can calm down the instantiate / rel

Re: lifecycle hooks for storm worker nodes

I don’t know of worker hooks, but shouldn’t be interpreted of them not existing. Bolts have prepare and cleanup methods that could probably be hacked to implement worker lifecycle hooks. The jvm has shutdown hooks: http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#addShutdownHook(

Re: New to apache storm

I haven’t used DRPC, so I can’t speak to it. That said, Kafka is pretty awesome and can do some really jaw dropping performance. If I were you, I’d consider standardizing around Kafka. If that isn’t viable, storm topologies are directional acyclic graphs, so you can merge two streams into a sing

Re: Supervisor repeatedly killing worker

I’m not, and If I had to guess I’d say it’s likely something is going wrong with the heartbeats, but how can I go about finding out? From: Paul Poulosky mailto:ppoul...@yahoo-inc.com>> Reply-To: "user@storm.apache.org" mailto:user@storm.apache.org>>, Paul Poulosk

Supervisor repeatedly killing worker

The supervisor is reporting the worker “still hasn’t started” even though the worker is up and appears to be working. After timeout, the supervisor kills and restarts the worker. I set the timeouts really high (10 mins) to see what would happen. I though maybe a GC pause or something. The worke

Re: Bolt Type Question

but I misread inheritance hierarchy when I was looking at the code today. So this makes it even more of a potential problem as it doesn't really have a clean solution. On Wed, Apr 15, 2015 at 3:08 PM, Grant Overby (groverby) mailto:grove...@cisco.com>> wrote: “Obviously any bol

Re: Bolt Type Question

thought that it was different from IRichBolt, but I misread inheritance hierarchy when I was looking at the code today. So this makes it even more of a potential problem as it doesn't really have a clean solution. On Wed, Apr 15, 2015 at 3:08 PM, Grant Overby (groverby) mailto:grove...@ci

Re: Bolt Type Question

“Obviously any bolt must implement IBolt" BaseBasicBolt isn’t an IBolt. If possible, sticking to using the “Rich” components will solve these sorts of issues. [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com

Re: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java (default-cli) on project : An exception occurred while executing the Java class

I don’t believe hyphens are legal in package names. “Also, unless you have a very good reason for doing so, or you don’t care about nuking your computer, I strongly suggest you stop executing a development project with sudo privileges." +1 [http://www.cisco.com/web/europe/images/email/signatu

Re: POM.xml error in MAC

all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. From: bigdata hadoop mailto:hadoopst...@gmail.com>> Date: Monday, April 6, 2015 at 11:31 AM To: "Grant Overby (groverb

Re: POM.xml error in MAC

The first [ERROR] line is truncated. That line should tell you what dependencies are missing. Idea is showing hive.version property is missing (its red). Where are you setting those properties for maven? [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726

Re: JDBC connection pool shared by bolts

I used this approach. It worked well. [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.com Mobile: 865 724 4910 [http://www.cisco.com/assets/

Re: look for help about Streaming benchmark and how to have a stress testing in Storm

I found the following link useful in a similar situation. It doesn’t sound like you’re using trident, but I expect at least some of the content will still be relevant. https://gist.github.com/mrflip/5958028 [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119

Re: Distributed logs coordination

NTP? http://en.wikipedia.org/wiki/Network_Time_Protocol [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com grove...@cisco.com Mobile: 865 724 4910 [http://

Re: How to use storm's HiveBolt

If storm and hive run as different users, check your permissions in HDFS. The storm user will need permissions to write to your table (this is likely under /apps/hive/warehouse ). Similarly, the hive user will need to be able to access and modify the files created by the storm user. This can cau

Re: configuring topology.workers

RE: configuring topology.workers Thank you. To be more specific, I looked at DefaultScheduler source. It’s getting the list of available slots as node+port combination…So if a node has two slots and next node has 2 more slots, will the workers for the topology be assigned to 2 slots in one node

Re: configuring topology.workers

Topologies are assigned to supervisors in a round robin fashion by the default scheduler. You can provide other schedulers: http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/ [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192

Limit max tuples in a Storm Trident topology

Is there a way to limit the maximum number of tuples in a Trident topology? Ie, if greater than x tuples are being processed, then spouts pause. [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com