Fwd: DYNAMIC ADJUSTMENT OF NUMBER OF TASKS

Hello, I am working on extending the Storm platform and would like to know the scope of dynamically adjusting the number of tasks for a topology. I don't want to work with a worst-case ceiling on the number of tasks. Please let me know if there is/isn't a method for dynamically changing the numb

DYNAMIC ADJUSTMENT OF NUMBER OF TASKS

Hello, I am working on extending the Storm platform and would like to know the scope of dynamically adjusting the number of tasks for a topology. I don't want to work with a worst-case ceiling on the number of tasks. Please let me know if there is/isn't a method for dynamically changing the numb

Re: Storm Cluster Install and Deploy. Am I doing this right?

Hi Tim,      Overall looks good to me. we recommend adding supervisord ( a watchdog process) to monitor nimbus and supervisors and restart them incase if they fail but since you already upstart which does the same job. Thanks, Harsha On June 18, 2015 at 1:15:16 PM, Tim Molter (tim.mol...@gmail.

Re: Storm Cluster Problem: Workers not being added

I discovered my problem. Since I cloned a machine from which to add more nodes to the cluster, I didn't clean out the /mnt/storm directory and my new worker was using the same ID string as the previous worker. Deleted everything in the storm dir and restarted my worker and now it's back. So indi

Storm Cluster Problem: Workers not being added

I had a standalone storm server running nimbus, supervisor, and the ui. The nimbus server was set to localhost since it was running locally. I spun up a new server and configured just the supervisor process, pointing to the nimbus server above. I am not seeing two supervisors. They have connectiv

Storm Cluster Install and Deploy. Am I doing this right?

Hi Storm gurus, I recently took the plunge into running a topology on a cluster with 4 hosts. I was hoping someone could critique my installation set up and deployment which I wrote up here: http://knowm.org/how-to-install-a-distributed-apache-storm-cluster/ In particular, is setting up zookeepe

Re: Singleton database connection

That I don't know - mind you that, if you're running more than one worker, you're probably using a singleton _per jvm_, not a single instance across the whole cluster. You could try assigning a UUID or something like that to your singleton and printing it to the logs from every bolt, see if it's th

Re: Singleton database connection

Thanks Nathan and Javier. I ended up using lazy init singleton and things seem to be working. But now how do I know its using single instance of the session? Is there a way we can know from storm UI or and where else? -- Kushan Maskey 817.403.7500 Precocity LLC M. Miller &

use external zookeeper in storm 0.9.2

Hi, Does anybody know how to run topology in local mode with external Zookeeper in Storm 0.9.2-incubating? I tried to configure the port and server but it continues to throw out NullPointerException. Anyone can help? Thank you very much!! here is the code public static void main(String[] args)

Re: Implementing user defined header in Tuple

First thing that comes to mind is: pass a Map as one of the Values within your tuple. Name it "header". Done. On Tue, Jun 16, 2015 at 10:01 AM, Bas van de Lustgraaf < basvdlustgr...@gmail.com> wrote: > Hi Guys, > > I would like to implement the same concept of an event header as Apache > flume us

Re: Has anybody successfully run storm 0.9+ in production under reasonable load?

tried. no effect. Thanks, Fang On Mon, Jun 15, 2015 at 9:33 PM, Binh Nguyen Van wrote: > Can you try this. > > Remove -XX:+CMSScavengeBeforeRemark flag and reduce your heap size so > that YGC happen once every 2-3 seconds? > If that fix the issue then I think GC is the cause of your problem. >

Re: Singleton database connection

We had a similar issue (namely, you can't pass anything non-serializable through the Configuration or create it in the constructor). What we did is pass in the Configuration or constructor a String with the path to a properties file. From that configuration file, in the bolt.prepare method you bri

ZooKeeper ensemble by using AWS Elastic IPs (Amazon EC2)

Hello all, I have recently updated my Storm cluster to use an ensemble of zookeeper servers (3 is the exact number). I am running my VMs on EC2, and in order to avoid updating the configuration files I used Elastic IPs for each of the Nimbus and ZooKeeper servers (every time an instance restarts,

Re: Using FLUX and multiple streams to the same bolt

Thanks Taylor! On Wed, Jun 17, 2015 at 9:19 PM, P. Taylor Goetz wrote: > JIRA: > > https://issues.apache.org/jira/browse/STORM-873 > > On Jun 17, 2015, at 11:38 PM, P. Taylor Goetz wrote: > > > Romeo, > > > > I have a fix (see below). Should be included in the next release (beta > or final). I

Re: Thrift Serialisation error when starting Storm 0.10.0-beta

You need to delete all data under /storm in zookeeper. In addition you will also need to delete all data under storm-local directory from nimbus and supervisor nodes. This means you lost all your topologies so you will have to resubmit the topologies once you restart the cluster. On Jun 18, 2

Re: increasing timeout in Test (STORM_TEST_TIMEOUT_MS)

Hi Tousif, Which version of Apache Storm you are using? Because, It's working with my storm-0.9.0.x and above version. You can increase tuple time out value with configuration object in your java code. if possible show your code patch of topology creation and deployment code. On Thu, Jun 18, 20

Thrift Serialisation error when starting Storm 0.10.0-beta

Hi, We are trying to use Storm 0.10.0 beta version. When we try to start nimbus we get the following error. Please suggest ways to get rid of it :) Stacktrace: ERROR backtype.storm.daemon.nimbus - Error on initialization of server service-handler java.lang.RuntimeException: java.lang.RuntimeE

Re: increasing timeout in Test (STORM_TEST_TIMEOUT_MS)

Hi Swapnil, I tried but it did not override default Timeout in Tests same is reported here https://issues.apache.org/jira/browse/STORM-354 https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/testing.clj#L190-192 java.lang.AssertionError: Test timed out (5000ms) at ba

Re: increasing timeout in Test (STORM_TEST_TIMEOUT_MS)

Hi Tousif, add following line in you java code conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 180); just before submitting you topology. On Thu, Jun 18, 2015 at 2:44 PM, Tousif wrote: > Hello, > > I tried passing STORM_TEST_TIMEOUT_MS as env variable in eclipse and also > through System.setPr

increasing timeout in Test (STORM_TEST_TIMEOUT_MS)

Hello, I tried passing STORM_TEST_TIMEOUT_MS as env variable in eclipse and also through System.setProperty but no luck. Any one got this working ? -- Regards Tousif Khazi