subject:"Need Help"

need help: storm worker keep crashing

2019-08-29 Thread Zainal Arifin (BLOOMBERG/ 731 LEX)

Hi,
I wonder if anyone who ever used streamparse (for storm topology in python 
code), ever run into this issue I am facing right now.
Unfortunately, I don't see any error from streamparse log, but I see exception 
below in the storm worker log.
The serializer exception section seems the culprit, but it doesn't have 
sufficient information for me to figure it out.
Perhaps any of you have idea/pointer for me? Thanks!


2019-08-28 15:05:32.947 o.a.s.s.ShellSpout Thread-11-event_spout-executor[10 
10] [INFO] Launched subprocess with pid 10054
2019-08-28 15:05:32.951 o.a.s.d.executor Thread-11-event_spout-executor[10 10] 
[INFO] Opened spout event_spout:(10)
2019-08-28 15:05:32.953 o.a.s.d.executor Thread-11-event_spout-executor[10 10] 
[INFO] Activating spout event_spout:(10)
2019-08-28 15:05:32.953 o.a.s.s.ShellSpout Thread-11-event_spout-executor[10 
10] [INFO] Start checking heartbeat...
2019-08-28 15:05:32.961 o.a.s.util Thread-11-event_spout-executor[10 10] 
[ERROR] Async loop died!
java.lang.RuntimeException: pid:10054, name:event_spout exitCode:-1, 
errorString:
at 
org.apache.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:218) 
~[storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.spout.ShellSpout.sendSyncCommand(ShellSpout.java:145) 
~[storm-core-1.1.1.jar:1.1.1]
at org.apache.storm.spout.ShellSpout.activate(ShellSpout.java:266) 
~[storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.daemon.executor$fn__4962$fn__4977$fn__5008.invoke(executor.clj:641)
 ~[storm-core-1.1.1.jar:1.1.1]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) 
[storm-core-1.1.1.jar:1.1.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.RuntimeException: 
org.apache.storm.multilang.NoOutputException: Pipe to subprocess seems to be 
broken! No output read.
Serializer Exception:


at 
org.apache.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:127) 
~[storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:183) 
~[storm-core-1.1.1.jar:1.1.1]
... 6 more
2019-08-28 15:05:32.968 o.a.s.d.executor Thread-11-event_spout-executor[10 10] 
[ERROR]
java.lang.RuntimeException: pid:10054, name:event_spout exitCode:-1, 
errorString:
at 
org.apache.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:218) 
~[storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.spout.ShellSpout.sendSyncCommand(ShellSpout.java:145) 
~[storm-core-1.1.1.jar:1.1.1]
at org.apache.storm.spout.ShellSpout.activate(ShellSpout.java:266) 
~[storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.daemon.executor$fn__4962$fn__4977$fn__5008.invoke(executor.clj:641)
 ~[storm-core-1.1.1.jar:1.1.1]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) 
[storm-core-1.1.1.jar:1.1.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.RuntimeException: 
org.apache.storm.multilang.NoOutputException: Pipe to subprocess seems to be 
broken! No output read.
Serializer Exception:


at 
org.apache.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:127) 
~[storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:183) 
~[storm-core-1.1.1.jar:1.1.1]
... 6 more
2019-08-28 15:05:33.009 o.a.s.util Thread-11-event_spout-executor[10 10] 
[ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) 
[storm-core-1.1.1.jar:1.1.1]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at 
org.apache.storm.daemon.worker$fn__5632$fn__5633.invoke(worker.clj:763) 
[storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.daemon.executor$mk_executor_data$fn__4848$fn__4849.invoke(executor.clj:276)
 [storm-core-1.1.1.jar:1.1.1]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:494) 
[storm-core-1.1.1.jar:1.1.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2019-08-28 15:05:33.018 o.a.s.d.worker Thread-16 [INFO] Shutting down worker 
tmon-4-1567019114 ba5b3695-b390-4c3e-9d92-af0771f17b86 6700

Re: Need help with custom Storm Trident Metrics

2019-04-04 Thread Stig Rohde Døssing

I think on 1.x you can work around it by directly accessing the Storm
metrics registry. That's what TopologyContext would do when you use it
https://github.com/apache/storm/blob/2ade13055315b69980f228ed786c6a76efb695a7/storm-core/src/jvm/org/apache/storm/task/TopologyContext.java#L397

Den tor. 4. apr. 2019 kl. 09.29 skrev Niraj Gupta :

> Is there any way to use TopologyContext in the Trident StateFactory?
> Also, I tried directly using Dropwizard metrics JMX reporter, which worked
> on the single node setup but when I deployed on the cluster then those
> metrics via my own reporter were not visible.
>
> On Thu, Apr 4, 2019 at 12:39 PM Stig Rohde Døssing 
> wrote:
>
>> There might be an issue with the API for StateFactory here. You need a
>> TopologyContext to use the new metrics API, but makeState doesn't take one.
>> Others can correct me if this is not an issue, but IMO feel free to
>> register an issue at https://issues.apache.org/jira.
>>
>> Den tor. 4. apr. 2019 kl. 08.58 skrev Stig Rohde Døssing <
>> stigdoess...@gmail.com>:
>>
>>> As far as I can tell, the JMX reporting is only hooked up to the metrics
>>> v2 API. You're using metrics v1. Could you try to register your metric with
>>> the new metrics system? You can find documentation at
>>> https://storm.apache.org/releases/2.0.0-SNAPSHOT/metrics_v2.html.
>>>
>>> Den ons. 3. apr. 2019 kl. 15.41 skrev Niraj Gupta :
>>>
 *Storm version*: 1.0.3 I'm registering custom metric in the makeState
 of the StateFactory implementation.

 @Overridepublic State makeState(final Map conf, final IMetricsContext 
 metricsContext, final int partitionIndex, final int numPartitions) {
 ReducedMetric reducedMetric = new ReducedMetric(new MeanReducer());
 metricsContext.registerMetric("custom-metric-1", reducedMetric, 1);
 reducedMetric.update(100);}

 I have added the following jmx config in the storm.yml

 -Dcom.sun.management.jmxremote.port=3337 
 -Dcom.sun.management.jmxremote.local.only=false 
 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false

 After running the Storm cluster when I'm connecting to the 3337 port
 then I'm not seeing this metrics which I had defined.

 What am I missing?

 --
 Regards,
 Niraj Gupta
 +91 727533898
 0


 --
 Regards,
 Niraj Gupta
 +91 7275338980


 --
 Regards,
 Niraj Gupta
 +91 7275338980

>>>
>
> --
> Regards,
> Niraj Gupta
> +91 7275338980
>

Re: Need help with custom Storm Trident Metrics

2019-04-04 Thread Niraj Gupta

Is there any way to use TopologyContext in the Trident StateFactory?
Also, I tried directly using Dropwizard metrics JMX reporter, which worked
on the single node setup but when I deployed on the cluster then those
metrics via my own reporter were not visible.

On Thu, Apr 4, 2019 at 12:39 PM Stig Rohde Døssing 
wrote:

> There might be an issue with the API for StateFactory here. You need a
> TopologyContext to use the new metrics API, but makeState doesn't take one.
> Others can correct me if this is not an issue, but IMO feel free to
> register an issue at https://issues.apache.org/jira.
>
> Den tor. 4. apr. 2019 kl. 08.58 skrev Stig Rohde Døssing <
> stigdoess...@gmail.com>:
>
>> As far as I can tell, the JMX reporting is only hooked up to the metrics
>> v2 API. You're using metrics v1. Could you try to register your metric with
>> the new metrics system? You can find documentation at
>> https://storm.apache.org/releases/2.0.0-SNAPSHOT/metrics_v2.html.
>>
>> Den ons. 3. apr. 2019 kl. 15.41 skrev Niraj Gupta :
>>
>>> *Storm version*: 1.0.3 I'm registering custom metric in the makeState
>>> of the StateFactory implementation.
>>>
>>> @Overridepublic State makeState(final Map conf, final IMetricsContext 
>>> metricsContext, final int partitionIndex, final int numPartitions) {
>>> ReducedMetric reducedMetric = new ReducedMetric(new MeanReducer());
>>> metricsContext.registerMetric("custom-metric-1", reducedMetric, 1);
>>> reducedMetric.update(100);}
>>>
>>> I have added the following jmx config in the storm.yml
>>>
>>> -Dcom.sun.management.jmxremote.port=3337 
>>> -Dcom.sun.management.jmxremote.local.only=false 
>>> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false 
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>>
>>> After running the Storm cluster when I'm connecting to the 3337 port
>>> then I'm not seeing this metrics which I had defined.
>>>
>>> What am I missing?
>>>
>>> --
>>> Regards,
>>> Niraj Gupta
>>> +91 727533898
>>> 0
>>>
>>>
>>> --
>>> Regards,
>>> Niraj Gupta
>>> +91 7275338980
>>>
>>>
>>> --
>>> Regards,
>>> Niraj Gupta
>>> +91 7275338980
>>>
>>

-- 
Regards,
Niraj Gupta
+91 7275338980

Re: Need help with custom Storm Trident Metrics

2019-04-04 Thread Stig Rohde Døssing

There might be an issue with the API for StateFactory here. You need a
TopologyContext to use the new metrics API, but makeState doesn't take one.
Others can correct me if this is not an issue, but IMO feel free to
register an issue at https://issues.apache.org/jira.

Den tor. 4. apr. 2019 kl. 08.58 skrev Stig Rohde Døssing <
stigdoess...@gmail.com>:

> As far as I can tell, the JMX reporting is only hooked up to the metrics
> v2 API. You're using metrics v1. Could you try to register your metric with
> the new metrics system? You can find documentation at
> https://storm.apache.org/releases/2.0.0-SNAPSHOT/metrics_v2.html.
>
> Den ons. 3. apr. 2019 kl. 15.41 skrev Niraj Gupta :
>
>> *Storm version*: 1.0.3 I'm registering custom metric in the makeState of
>> the StateFactory implementation.
>>
>> @Overridepublic State makeState(final Map conf, final IMetricsContext 
>> metricsContext, final int partitionIndex, final int numPartitions) {
>> ReducedMetric reducedMetric = new ReducedMetric(new MeanReducer());
>> metricsContext.registerMetric("custom-metric-1", reducedMetric, 1);
>> reducedMetric.update(100);}
>>
>> I have added the following jmx config in the storm.yml
>>
>> -Dcom.sun.management.jmxremote.port=3337 
>> -Dcom.sun.management.jmxremote.local.only=false 
>> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false 
>> -Dcom.sun.management.jmxremote.authenticate=false
>>
>> After running the Storm cluster when I'm connecting to the 3337 port then
>> I'm not seeing this metrics which I had defined.
>>
>> What am I missing?
>>
>> --
>> Regards,
>> Niraj Gupta
>> +91 727533898
>> 0
>>
>>
>> --
>> Regards,
>> Niraj Gupta
>> +91 7275338980
>>
>>
>> --
>> Regards,
>> Niraj Gupta
>> +91 7275338980
>>
>

Re: Need help with custom Storm Trident Metrics

2019-04-04 Thread Stig Rohde Døssing

As far as I can tell, the JMX reporting is only hooked up to the metrics v2
API. You're using metrics v1. Could you try to register your metric with
the new metrics system? You can find documentation at
https://storm.apache.org/releases/2.0.0-SNAPSHOT/metrics_v2.html.

Den ons. 3. apr. 2019 kl. 15.41 skrev Niraj Gupta :

> *Storm version*: 1.0.3 I'm registering custom metric in the makeState of
> the StateFactory implementation.
>
> @Overridepublic State makeState(final Map conf, final IMetricsContext 
> metricsContext, final int partitionIndex, final int numPartitions) {
> ReducedMetric reducedMetric = new ReducedMetric(new MeanReducer());
> metricsContext.registerMetric("custom-metric-1", reducedMetric, 1);
> reducedMetric.update(100);}
>
> I have added the following jmx config in the storm.yml
>
> -Dcom.sun.management.jmxremote.port=3337 
> -Dcom.sun.management.jmxremote.local.only=false 
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false
>
> After running the Storm cluster when I'm connecting to the 3337 port then
> I'm not seeing this metrics which I had defined.
>
> What am I missing?
>
> --
> Regards,
> Niraj Gupta
> +91 727533898
> 0
>
>
> --
> Regards,
> Niraj Gupta
> +91 7275338980
>
>
> --
> Regards,
> Niraj Gupta
> +91 7275338980
>

Fwd: Need help with custom Storm Trident Metrics

2019-04-03 Thread Niraj Gupta

*Storm version*: 1.0.3 I'm registering custom metric in the makeState of
the StateFactory implementation.

@Overridepublic State makeState(final Map conf, final IMetricsContext
metricsContext, final int partitionIndex, final int numPartitions) {
ReducedMetric reducedMetric = new ReducedMetric(new MeanReducer());
metricsContext.registerMetric("custom-metric-1", reducedMetric, 1);
reducedMetric.update(100);}

I have added the following jmx config in the storm.yml

-Dcom.sun.management.jmxremote.port=3337
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

After running the Storm cluster when I'm connecting to the 3337 port then
I'm not seeing this metrics which I had defined.

What am I missing?

-- 
Regards,
Niraj Gupta
+91 727533898
0


-- 
Regards,
Niraj Gupta
+91 7275338980


-- 
Regards,
Niraj Gupta
+91 7275338980

Fwd: I need help to run a topology

2017-09-11 Thread 张博

-- Forwarded message --
From: 张博 <jyzhan...@gmail.com>
Date: 2017-09-12 13:29 GMT+08:00
Subject: I need help to run a topology
To: user@storm.apache.org

Hi,
I used springboot to develop a topology,but when I package it to a jar,and
run the jar,it has error.

The detail error :https://stackoverflow.com/questions/46168377/storm-run-
springboot-development-topology-has-error

Can you send a topology example that  develops by springboot.The most
important thing is the method to package it.

Thanks
Pradeep

Re: need help on running storm examples

2017-07-28 Thread Stig Rohde Døssing

Hi Himabindu,

backtype.storm.* is the old package namespace for Storm for pre-1.0.0 Storm
classes. If you are working with new code, you should be referring to
org.apache.storm.* instead. You can more than likely just do a search and
replace in your Java files for backtype.storm and replace by
org.apache.storm.

You might want to use mvn dependency:tree to check that your project
doesn't depend on older Storm jars. Also consider posting your pom.xml if
you need more help.

You could use the jartransformer class as you mentioned, but I think it was
intended as a way to allow pre-1.0.0 topologies to run on 1.0.0+, and it
would be better to update your code to refer to org.apache.storm instead.

2017-07-28 13:54 GMT+02:00 Himabindu Koppula <
himabindu.hadoopdevelo...@gmail.com>:

>
> -- Forwarded message --
> From: Himabindu Koppula <himabindu.hadoopdevelo...@gmail.com>
> Date: Fri, Jul 28, 2017 at 5:15 PM
> Subject: need help on running storm examples
> To: user@storm.apache.org
>
>
> Hi Team,
> I new to storm .I am using storm-core and strom-hbase jar files of version
> 1.0.1 .
> I am using maven 3.5 build my project.
> I am getting an error saying backtype.storm.tuple.Fields can't be
> resolved,in the
>
> class PersistentWordCount.java ,when using
>
> SimpleHBaseMapper mapper = *new** SimpleHBaseMapper()*
>
> * .withRowKeyField("word") .withColumnFields(new Fields("word")) *
>
> .withCounterFields(*new* Fields("count"))
>
> .withColumnFamily("cf");
>
> I have gone through the docuemtation and understood that,I need to add the
> following configuration client.jartransformer.class:
> "org.apache.storm.hack.StormShadeTransformer"
>
>  in storm.yaml.
>
>
>
> But,inordder to get this build successful,could you please let me know the
> steps to be followed.
>
>
> Thanks and Regards,
>
> Himabindu Koppula
>
>
>
>
>
>
>
>
>
>
>

Fwd: need help on running storm examples

2017-07-28 Thread Himabindu Koppula

-- Forwarded message --
From: Himabindu Koppula <himabindu.hadoopdevelo...@gmail.com>
Date: Fri, Jul 28, 2017 at 5:15 PM
Subject: need help on running storm examples
To: user@storm.apache.org

Hi Team,
I new to storm .I am using storm-core and strom-hbase jar files of version
1.0.1 .
I am using maven 3.5 build my project.
I am getting an error saying backtype.storm.tuple.Fields can't be
resolved,in the

class PersistentWordCount.java ,when using

SimpleHBaseMapper mapper = *new** SimpleHBaseMapper()*

* .withRowKeyField("word") .withColumnFields(new Fields("word")) *

.withCounterFields(*new* Fields("count"))

.withColumnFamily("cf");

I have gone through the docuemtation and understood that,I need to add the
following configuration client.jartransformer.class: "org.apache.storm.hack.
StormShadeTransformer"

 in storm.yaml.

But,inordder to get this build successful,could you please let me know the
steps to be followed.

Thanks and Regards,

Himabindu Koppula

Re: Need help with Unknown Error, not sure

2016-07-18 Thread Satish Duggana

Your supervisor's local host name is not getting resolved.  You can
override this by configuring storm.local.hostname with a valid hostname.

Thanks,
Satish.

On Tue, Jul 19, 2016 at 12:22 AM, Joaquin Menchaca 
wrote:

> Hi.
>
> Anyone have any suggestions how to debug this and find out what is
> happening?  Any troubleshooting tools I can use to test the functionality
> of the storm cluster?
>
> 3899 [main-EventThread] INFO  o.a.s.s.o.a.c.f.s.ConnectionStateManager - 
> State change: CONNECTED
> 3908 [main] INFO  b.s.d.supervisor - Starting Supervisor with conf 
> {"topology.builtin.metrics.bucket.size.secs" 60, "nimbus.childopts" "-Xmx5g 
> -Djava.net.preferIPv4Stack=true", "ui.filter.params" nil, 
> "storm.cluster.mode" "local", "storm.messaging.netty.client_worker_threads" 
> 1, "supervisor.run.worker.as.user" false, "topology.max.task.parallelism" 
> nil, "zmq.threads" 1, "storm.group.mapping.service" 
> "backtype.storm.security.auth.ShellBasedGroupsMapping", 
> "transactional.zookeeper.root" "/transactional", 
> "topology.sleep.spout.wait.strategy.time.ms" 1, "drpc.invocations.port" 3773, 
> "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", 
> "storm.messaging.netty.server_worker_threads" 1, 
> "topology.max.error.report.per.interval" 5, "storm.thrift.transport" 
> "backtype.storm.security.auth.SimpleTransportPlugin", "zmq.hwm" 0, 
> "storm.principal.tolocal" 
> "backtype.storm.security.auth.DefaultPrincipalToLocal", 
> "supervisor.worker.shutdown.sleep.secs" 1, "storm.zookeeper.retry.times" 5, 
> "ui.actions.enabled" true, "zmq.linger.millis" 0, "supervisor.enable" true, 
> "topology.stats.sample.rate" 0.05, "storm.messaging.netty.min_wait_ms" 1000, 
> "storm.zookeeper.port" 2000, "supervisor.heartbeat.frequency.secs" 5, 
> "topology.enable.message.timeouts" true, "drpc.worker.threads" 64, 
> "drpc.queue.size" 128, "drpc.https.keystore.password" "", "logviewer.port" 
> 8000, "nimbus.reassign" true, "topology.executor.send.buffer.size" 1024, 
> "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", 
> "ui.host" "0.0.0.0", "storm.nimbus.retry.interval.millis" 2000, 
> "nimbus.inbox.jar.expiration.secs" 3600, "dev.zookeeper.path" 
> "/tmp/dev-storm-zookeeper", "topology.acker.executors" nil, 
> "topology.fall.back.on.java.serialization" true, "storm.zookeeper.servers" 
> ["localhost"], "nimbus.thrift.threads" 64, "logviewer.cleanup.age.mins" 
> 10080, "topology.worker.childopts" nil, "topology.classpath" nil, 
> "supervisor.monitor.frequency.secs" 3, "drpc.servers" 
> ["nimbus-stormcluster.gobalto.com"], "nimbus.credential.renewers.freq.secs" 
> 600, "topology.skip.missing.kryo.registrations" true, 
> "drpc.authorizer.acl.filename" "drpc-auth-acl.yaml", 
> "storm.group.mapping.service.cache.duration.secs" 120, 
> "topology.testing.always.try.serialize" false, "nimbus.monitor.freq.secs" 10, 
> "supervisor.supervisors" [], "topology.tasks" nil, 
> "topology.bolts.outgoing.overflow.buffer.enable" false, 
> "storm.messaging.netty.socket.backlog" 500, "topology.workers" 1, 
> "storm.local.dir" "/tmp/3caeea32-21a1-4f1b-8d05-b1e47361e33f", 
> "worker.childopts" "-Xmx1536m -Djava.net.preferIPv4Stack=true", 
> "storm.auth.simple-white-list.users" [], "topology.message.timeout.secs" 30, 
> "topology.state.synchronization.timeout.secs" 60, "topology.tuple.serializer" 
> "backtype.storm.serialization.types.ListDelegateSerializer", 
> "supervisor.supervisors.commands" [], "logviewer.childopts" "-Xmx128m", 
> "topology.environment" nil, "topology.debug" false, 
> "storm.messaging.netty.max_retries" 10, "ui.childopts" "-Xmx768m 
> -Djava.net.preferIPv4Stack=true", "storm.zookeeper.session.timeout" 2, 
> "drpc.childopts" "-Xmx768m", "drpc.http.creds.plugin" 
> "backtype.storm.security.auth.DefaultHttpCredentialsPlugin", 
> "storm.zookeeper.connection.timeout" 15000, "storm.zookeeper.auth.user" nil, 
> "storm.meta.serialization.delegate" 
> "backtype.storm.serialization.GzipThriftSerializationDelegate", 
> "topology.max.spout.pending" nil, "nimbus.supervisor.timeout.secs" 60, 
> "nimbus.task.timeout.secs" 30, "drpc.port" 3772, 
> "storm.zookeeper.retry.intervalceiling.millis" 3, "nimbus.thrift.port" 
> 6627, "storm.auth.simple-acl.admins" [], "storm.nimbus.retry.times" 5, 
> "supervisor.worker.start.timeout.secs" 120, "storm.zookeeper.retry.interval" 
> 1000, "logs.users" nil, "transactional.zookeeper.port" nil, 
> "drpc.max_buffer_size" 1048576, "task.credentials.poll.secs" 30, 
> "drpc.https.keystore.type" "JKS", "topology.worker.receiver.thread.count" 1, 
> "supervisor.slots.ports" (1024 1025 1026), "topology.transfer.buffer.size" 
> 1024, "topology.worker.shared.thread.pool.size" 4, 
> "drpc.authorizer.acl.strict" false, "nimbus.file.copy.expiration.secs" 600, 
> "topology.executor.receive.buffer.size" 1024, "nimbus.task.launch.secs" 120, 
> "storm.local.mode.zmq" false, "storm.messaging.netty.buffer_size" 16384, 
>

Need help with Unknown Error, not sure

2016-07-18 Thread Joaquin Menchaca

Hi.

Anyone have any suggestions how to debug this and find out what is
happening?  Any troubleshooting tools I can use to test the functionality
of the storm cluster?

3899 [main-EventThread] INFO  o.a.s.s.o.a.c.f.s.ConnectionStateManager
- State change: CONNECTED
3908 [main] INFO  b.s.d.supervisor - Starting Supervisor with conf
{"topology.builtin.metrics.bucket.size.secs" 60, "nimbus.childopts"
"-Xmx5g -Djava.net.preferIPv4Stack=true", "ui.filter.params" nil,
"storm.cluster.mode" "local",
"storm.messaging.netty.client_worker_threads" 1,
"supervisor.run.worker.as.user" false, "topology.max.task.parallelism"
nil, "zmq.threads" 1, "storm.group.mapping.service"
"backtype.storm.security.auth.ShellBasedGroupsMapping",
"transactional.zookeeper.root" "/transactional",
"topology.sleep.spout.wait.strategy.time.ms" 1,
"drpc.invocations.port" 3773, "topology.multilang.serializer"
"backtype.storm.multilang.JsonSerializer",
"storm.messaging.netty.server_worker_threads" 1,
"topology.max.error.report.per.interval" 5, "storm.thrift.transport"
"backtype.storm.security.auth.SimpleTransportPlugin", "zmq.hwm" 0,
"storm.principal.tolocal"
"backtype.storm.security.auth.DefaultPrincipalToLocal",
"supervisor.worker.shutdown.sleep.secs" 1,
"storm.zookeeper.retry.times" 5, "ui.actions.enabled" true,
"zmq.linger.millis" 0, "supervisor.enable" true,
"topology.stats.sample.rate" 0.05, "storm.messaging.netty.min_wait_ms"
1000, "storm.zookeeper.port" 2000,
"supervisor.heartbeat.frequency.secs" 5,
"topology.enable.message.timeouts" true, "drpc.worker.threads" 64,
"drpc.queue.size" 128, "drpc.https.keystore.password" "",
"logviewer.port" 8000, "nimbus.reassign" true,
"topology.executor.send.buffer.size" 1024,
"topology.spout.wait.strategy"
"backtype.storm.spout.SleepSpoutWaitStrategy", "ui.host" "0.0.0.0",
"storm.nimbus.retry.interval.millis" 2000,
"nimbus.inbox.jar.expiration.secs" 3600, "dev.zookeeper.path"
"/tmp/dev-storm-zookeeper", "topology.acker.executors" nil,
"topology.fall.back.on.java.serialization" true,
"storm.zookeeper.servers" ["localhost"], "nimbus.thrift.threads" 64,
"logviewer.cleanup.age.mins" 10080, "topology.worker.childopts" nil,
"topology.classpath" nil, "supervisor.monitor.frequency.secs" 3,
"drpc.servers" ["nimbus-stormcluster.gobalto.com"],
"nimbus.credential.renewers.freq.secs" 600,
"topology.skip.missing.kryo.registrations" true,
"drpc.authorizer.acl.filename" "drpc-auth-acl.yaml",
"storm.group.mapping.service.cache.duration.secs" 120,
"topology.testing.always.try.serialize" false,
"nimbus.monitor.freq.secs" 10, "supervisor.supervisors" [],
"topology.tasks" nil, "topology.bolts.outgoing.overflow.buffer.enable"
false, "storm.messaging.netty.socket.backlog" 500, "topology.workers"
1, "storm.local.dir" "/tmp/3caeea32-21a1-4f1b-8d05-b1e47361e33f",
"worker.childopts" "-Xmx1536m -Djava.net.preferIPv4Stack=true",
"storm.auth.simple-white-list.users" [],
"topology.message.timeout.secs" 30,
"topology.state.synchronization.timeout.secs" 60,
"topology.tuple.serializer"
"backtype.storm.serialization.types.ListDelegateSerializer",
"supervisor.supervisors.commands" [], "logviewer.childopts"
"-Xmx128m", "topology.environment" nil, "topology.debug" false,
"storm.messaging.netty.max_retries" 10, "ui.childopts" "-Xmx768m
-Djava.net.preferIPv4Stack=true", "storm.zookeeper.session.timeout"
2, "drpc.childopts" "-Xmx768m", "drpc.http.creds.plugin"
"backtype.storm.security.auth.DefaultHttpCredentialsPlugin",
"storm.zookeeper.connection.timeout" 15000,
"storm.zookeeper.auth.user" nil, "storm.meta.serialization.delegate"
"backtype.storm.serialization.GzipThriftSerializationDelegate",
"topology.max.spout.pending" nil, "nimbus.supervisor.timeout.secs" 60,
"nimbus.task.timeout.secs" 30, "drpc.port" 3772,
"storm.zookeeper.retry.intervalceiling.millis" 3,
"nimbus.thrift.port" 6627, "storm.auth.simple-acl.admins" [],
"storm.nimbus.retry.times" 5, "supervisor.worker.start.timeout.secs"
120, "storm.zookeeper.retry.interval" 1000, "logs.users" nil,
"transactional.zookeeper.port" nil, "drpc.max_buffer_size" 1048576,
"task.credentials.poll.secs" 30, "drpc.https.keystore.type" "JKS",
"topology.worker.receiver.thread.count" 1, "supervisor.slots.ports"
(1024 1025 1026), "topology.transfer.buffer.size" 1024,
"topology.worker.shared.thread.pool.size" 4,
"drpc.authorizer.acl.strict" false, "nimbus.file.copy.expiration.secs"
600, "topology.executor.receive.buffer.size" 1024,
"nimbus.task.launch.secs" 120, "storm.local.mode.zmq" false,
"storm.messaging.netty.buffer_size" 16384,
"worker.heartbeat.frequency.secs" 1, "ui.http.creds.plugin"
"backtype.storm.security.auth.DefaultHttpCredentialsPlugin",
"storm.zookeeper.root" "/storm", "topology.tick.tuple.freq.secs" nil,
"drpc.https.port" -1, "task.refresh.poll.secs" 10,
"task.heartbeat.frequency.secs" 3, "storm.messaging.netty.max_wait_ms"
5000, "nimbus.impersonation.authorizer"
"backtype.storm.security.auth.authorizer.ImpersonationAuthorizer",
"drpc.http.port" 3774,

Re: Need Help

2016-04-27 Thread Spico Florin

Hi!
In Storm UI, please have a look at the value that you get for the Capacity
(last 10m) for your bolt. If it closer If this value  close to 1, then the
bolt is “at capacity”and is a bottleneck in your topology.Address such
bottlenecks by increasing the parallelism of the “at-capacity” bolts. This
means you should increase the num executors for this bolt, by providing the
hint parallelism in the bolt declarer or set up a numTask for this bolt to
16 and then rebalance your topology by providinng the
storm rebalance  -e  (max 16).
I hope that these help.
Regards,
 Florin

On Wed, Apr 27, 2016 at 10:54 AM, Sai Dilip Reddy Kiralam <
dkira...@aadhya-analytics.com> wrote:

>
> Hi All,
>
> I'm running a topology in local machine.my bolts parse the json data and
> insert the parsed fields into db.one of the bolt is turning into red
> color.What does it mean.
> does it indicate flow of tuples is more for that insert bolt? if yes!!
> .what should I do?Do I need it increase the parallelism of that particular
> bolt?
>
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>

Need Help

2016-04-27 Thread Sai Dilip Reddy Kiralam

Hi All,

I'm running a topology in local machine.my bolts parse the json data and
insert the parsed fields into db.one of the bolt is turning into red
color.What does it mean.
does it indicate flow of tuples is more for that insert bolt? if yes!!
.what should I do?Do I need it increase the parallelism of that particular
bolt?




*Best regards,*

*K.Sai Dilip Reddy.*

Re: Need Help regarding topology with numWorker>1

2016-02-13 Thread Matthias J. Sax

If your workload does not saturate a single machine, it will of course
be more efficient to run within a single worker as you avoid intra-JVM
communication.

As long as CPU, memory, or network of a single machine is not utilized
completely, you will not benefit from multiple workers from a
performance point of view.

Btw: I am a not sure what you mean by "I have to create a scenario..."?

On 02/12/2016 11:42 PM, Rudraneel chakraborty wrote:
> Thanks. Actually i have to create a scenario where 2 worker performs
> better than one worker. But in reality , topology with a single worker
> performs considerably better. 
> 
> I sending csv lines to kafka (5 partitions ) and reading them from a
> topology with kafka spout (parallelism hint 5)
> 
> Any thoughts? 
> 
> On Thursday, 11 February 2016, Nathan Leung  > wrote:
> 
> Any situation where you require more CPU than 1 server can provide
> for you - there are tuning parameters (e.g. localOrShuffleGrouping)
> that you can use to reduce the amount of data sent over the network too.
> 
> Any situation where you need to have tolerance in case of machine
> failure.
> 
> 
> 
> On Thu, Feb 11, 2016 at 2:58 PM, Rudraneel chakraborty
>  >
> wrote:
> 
> Topology param numWorkers i meant 
> 
> 
> On Thursday, 11 February 2016, Rudraneel chakraborty
>  >
> wrote:
> 
> More specifically , i have seen a topology performs better
> if it is assigned a single worker compared to more than one
> worker. 
> 
> I want a situation where a topology performs better with
> more than one worker.
> 
> And it doesnt matter if both workers are on same supervisor
> or different supervisor
> 
> On Thursday, 11 February 2016, Matthias J. Sax
>  wrote:
> 
> I am not sure what you mean:
> 
>   - number of worker slots per supervisor
> or
>   - topology parameter "number of workers"
> 
> Can you clarify?
> 
> -Matthias
> 
> On 02/11/2016 05:14 AM, anshu shukla wrote:
> > Not  like that..  But i have used workers equal to
> number of cores. Each
> > vm with 8 corea.
> >
> > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
> >  > > wrote:
> >
> > more than one worker on same node ? Did u use
> custom schedule r?
> > because by default, the workers would be spread
> throughout the cluster
> >
> 
> 
> 
> -- 
> Rudraneel Chakraborty
> Carleton University Real Time and Distributed Systems Reserach
> 
> 
> 
> -- 
> Rudraneel Chakraborty
> Carleton University Real Time and Distributed Systems Reserach
> 
> 
> 
> 
> -- 
> Rudraneel Chakraborty
> Carleton University Real Time and Distributed Systems Reserach
> 



signature.asc
Description: OpenPGP digital signature

Re: Need Help regarding topology with numWorker>1

2016-02-13 Thread Rudraneel chakraborty

Thank you
And i need this because i am developing a custom scheduler . And for this i
need to have topologies performing well with two worker processes instead
of one. And its becoming tough to saturate a single worker

On Saturday, 13 February 2016, Matthias J. Sax  wrote:

> If your workload does not saturate a single machine, it will of course
> be more efficient to run within a single worker as you avoid intra-JVM
> communication.
>
> As long as CPU, memory, or network of a single machine is not utilized
> completely, you will not benefit from multiple workers from a
> performance point of view.
>
> Btw: I am a not sure what you mean by "I have to create a scenario..."?
>
> On 02/12/2016 11:42 PM, Rudraneel chakraborty wrote:
> > Thanks. Actually i have to create a scenario where 2 worker performs
> > better than one worker. But in reality , topology with a single worker
> > performs considerably better.
> >
> > I sending csv lines to kafka (5 partitions ) and reading them from a
> > topology with kafka spout (parallelism hint 5)
> >
> > Any thoughts?
> >
> > On Thursday, 11 February 2016, Nathan Leung  
> > >> wrote:
> >
> > Any situation where you require more CPU than 1 server can provide
> > for you - there are tuning parameters (e.g. localOrShuffleGrouping)
> > that you can use to reduce the amount of data sent over the network
> too.
> >
> > Any situation where you need to have tolerance in case of machine
> > failure.
> >
> >
> >
> > On Thu, Feb 11, 2016 at 2:58 PM, Rudraneel chakraborty
> > 
> >  ');>>
> > wrote:
> >
> > Topology param numWorkers i meant
> >
> >
> > On Thursday, 11 February 2016, Rudraneel chakraborty
> > 
> >  ');>>
> > wrote:
> >
> > More specifically , i have seen a topology performs better
> > if it is assigned a single worker compared to more than one
> > worker.
> >
> > I want a situation where a topology performs better with
> > more than one worker.
> >
> > And it doesnt matter if both workers are on same supervisor
> > or different supervisor
> >
> > On Thursday, 11 February 2016, Matthias J. Sax
> > > wrote:
> >
> > I am not sure what you mean:
> >
> >   - number of worker slots per supervisor
> > or
> >   - topology parameter "number of workers"
> >
> > Can you clarify?
> >
> > -Matthias
> >
> > On 02/11/2016 05:14 AM, anshu shukla wrote:
> > > Not  like that..  But i have used workers equal to
> > number of cores. Each
> > > vm with 8 corea.
> > >
> > > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
> > > 
> > > >>
> wrote:
> > >
> > > more than one worker on same node ? Did u use
> > custom schedule r?
> > > because by default, the workers would be spread
> > throughout the cluster
> > >
> >
> >
> >
> > --
> > Rudraneel Chakraborty
> > Carleton University Real Time and Distributed Systems
> Reserach
> >
> >
> >
> > --
> > Rudraneel Chakraborty
> > Carleton University Real Time and Distributed Systems Reserach
> >
> >
> >
> >
> > --
> > Rudraneel Chakraborty
> > Carleton University Real Time and Distributed Systems Reserach
> >
>
>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Re: Need Help regarding topology with numWorker>1

2016-02-12 Thread Rudraneel chakraborty

Thanks for the reply. So after some thinking , i figured if i shorten the
size of the worker incoming queue , i can create the scenario where using
more than one worker might result in better performance. Any thoughts on
how to do that?

The property topology, receiver.buffer.size should be the the key as
mentioned here
http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/

But this jira issue says this property is no longer used
https://issues.apache.org/jira/plugins/servlet/mobile#issue/STORM-596
Any thoughts on how to change the incoming queue size of the workers?

On Thursday, 11 February 2016, Matthias J. Sax  wrote:

> I am not sure what you mean:
>
>   - number of worker slots per supervisor
> or
>   - topology parameter "number of workers"
>
> Can you clarify?
>
> -Matthias
>
> On 02/11/2016 05:14 AM, anshu shukla wrote:
> > Not  like that..  But i have used workers equal to number of cores. Each
> > vm with 8 corea.
> >
> > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
> > 
> > >> wrote:
> >
> > more than one worker on same node ? Did u use custom schedule r?
> > because by default, the workers would be spread throughout the
> cluster
> >
>
>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Re: Need Help regarding topology with numWorker>1

2016-02-12 Thread Rudraneel chakraborty

Thanks for the reply. So after some thinking , i figured if i shorten the
size of the worker incoming queue , i can create the scenario where using
more than one worker might result in better performance. Any thoughts on
how to do that?

The property topology, receiver.buffer.size should be the the key as
mentioned here
http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/

But this jira issue says this property is no longer used
https://issues.apache.org/jira/plugins/servlet/mobile#issue/STORM-596
Any thoughts on how to change the incoming queue size of the workers?

On Thursday, 11 February 2016, Matthias J. Sax  wrote:

> I am not sure what you mean:
>
>   - number of worker slots per supervisor
> or
>   - topology parameter "number of workers"
>
> Can you clarify?
>
> -Matthias
>
> On 02/11/2016 05:14 AM, anshu shukla wrote:
> > Not  like that..  But i have used workers equal to number of cores. Each
> > vm with 8 corea.
> >
> > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
> > 
> > >> wrote:
> >
> > more than one worker on same node ? Did u use custom schedule r?
> > because by default, the workers would be spread throughout the
> cluster
> >
>
>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Re: Need Help regarding topology with numWorker>1

2016-02-12 Thread Rudraneel chakraborty

Thanks. Actually i have to create a scenario where 2 worker performs better
than one worker. But in reality , topology with a single worker performs
considerably better.

I sending csv lines to kafka (5 partitions ) and reading them from a
topology with kafka spout (parallelism hint 5)

Any thoughts?

On Thursday, 11 February 2016, Nathan Leung  wrote:

> Any situation where you require more CPU than 1 server can provide for you
> - there are tuning parameters (e.g. localOrShuffleGrouping) that you can
> use to reduce the amount of data sent over the network too.
>
> Any situation where you need to have tolerance in case of machine failure.
>
>
>
> On Thu, Feb 11, 2016 at 2:58 PM, Rudraneel chakraborty <
> rudraneel.chakrabo...@gmail.com
> > wrote:
>
>> Topology param numWorkers i meant
>>
>>
>> On Thursday, 11 February 2016, Rudraneel chakraborty <
>> rudraneel.chakrabo...@gmail.com
>> > wrote:
>>
>>> More specifically , i have seen a topology performs better if it is
>>> assigned a single worker compared to more than one worker.
>>>
>>> I want a situation where a topology performs better with more than one
>>> worker.
>>>
>>> And it doesnt matter if both workers are on same supervisor or different
>>> supervisor
>>>
>>> On Thursday, 11 February 2016, Matthias J. Sax  wrote:
>>>
 I am not sure what you mean:

   - number of worker slots per supervisor
 or
   - topology parameter "number of workers"

 Can you clarify?

 -Matthias

 On 02/11/2016 05:14 AM, anshu shukla wrote:
 > Not  like that..  But i have used workers equal to number of cores.
 Each
 > vm with 8 corea.
 >
 > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
 >  > wrote:
 >
 > more than one worker on same node ? Did u use custom schedule r?
 > because by default, the workers would be spread throughout the
 cluster
 >


>>>
>>> --
>>> Rudraneel Chakraborty
>>> Carleton University Real Time and Distributed Systems Reserach
>>>
>>>
>>
>> --
>> Rudraneel Chakraborty
>> Carleton University Real Time and Distributed Systems Reserach
>>
>>
>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Re: Need Help regarding topology with numWorker>1

2016-02-11 Thread Rudraneel chakraborty

Topology param numWorkers i meant

On Thursday, 11 February 2016, Rudraneel chakraborty <
rudraneel.chakrabo...@gmail.com> wrote:

> More specifically , i have seen a topology performs better if it is
> assigned a single worker compared to more than one worker.
>
> I want a situation where a topology performs better with more than one
> worker.
>
> And it doesnt matter if both workers are on same supervisor or different
> supervisor
>
> On Thursday, 11 February 2016, Matthias J. Sax  > wrote:
>
>> I am not sure what you mean:
>>
>>   - number of worker slots per supervisor
>> or
>>   - topology parameter "number of workers"
>>
>> Can you clarify?
>>
>> -Matthias
>>
>> On 02/11/2016 05:14 AM, anshu shukla wrote:
>> > Not  like that..  But i have used workers equal to number of cores. Each
>> > vm with 8 corea.
>> >
>> > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
>> > > > > wrote:
>> >
>> > more than one worker on same node ? Did u use custom schedule r?
>> > because by default, the workers would be spread throughout the
>> cluster
>> >
>>
>>
>
> --
> Rudraneel Chakraborty
> Carleton University Real Time and Distributed Systems Reserach
>
>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Re: Need Help regarding topology with numWorker>1

2016-02-11 Thread Matthias J. Sax

I am not sure what you mean:

  - number of worker slots per supervisor
or
  - topology parameter "number of workers"

Can you clarify?

-Matthias

On 02/11/2016 05:14 AM, anshu shukla wrote:
> Not  like that..  But i have used workers equal to number of cores. Each
> vm with 8 corea.
> 
> On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
>  > wrote:
> 
> more than one worker on same node ? Did u use custom schedule r?
> because by default, the workers would be spread throughout the cluster 
> 



signature.asc
Description: OpenPGP digital signature

Re: Need Help regarding topology with numWorker>1

2016-02-11 Thread Rudraneel chakraborty

More specifically , i have seen a topology performs better if it is
assigned a single worker compared to more than one worker.

I want a situation where a topology performs better with more than one
worker.

And it doesnt matter if both workers are on same supervisor or different
supervisor

On Thursday, 11 February 2016, Matthias J. Sax  wrote:

> I am not sure what you mean:
>
>   - number of worker slots per supervisor
> or
>   - topology parameter "number of workers"
>
> Can you clarify?
>
> -Matthias
>
> On 02/11/2016 05:14 AM, anshu shukla wrote:
> > Not  like that..  But i have used workers equal to number of cores. Each
> > vm with 8 corea.
> >
> > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
> > 
> > >> wrote:
> >
> > more than one worker on same node ? Did u use custom schedule r?
> > because by default, the workers would be spread throughout the
> cluster
> >
>
>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Re: Need Help regarding topology with numWorker>1

2016-02-11 Thread Nathan Leung

Any situation where you require more CPU than 1 server can provide for you
- there are tuning parameters (e.g. localOrShuffleGrouping) that you can
use to reduce the amount of data sent over the network too.

Any situation where you need to have tolerance in case of machine failure.



On Thu, Feb 11, 2016 at 2:58 PM, Rudraneel chakraborty <
rudraneel.chakrabo...@gmail.com> wrote:

> Topology param numWorkers i meant
>
>
> On Thursday, 11 February 2016, Rudraneel chakraborty <
> rudraneel.chakrabo...@gmail.com> wrote:
>
>> More specifically , i have seen a topology performs better if it is
>> assigned a single worker compared to more than one worker.
>>
>> I want a situation where a topology performs better with more than one
>> worker.
>>
>> And it doesnt matter if both workers are on same supervisor or different
>> supervisor
>>
>> On Thursday, 11 February 2016, Matthias J. Sax  wrote:
>>
>>> I am not sure what you mean:
>>>
>>>   - number of worker slots per supervisor
>>> or
>>>   - topology parameter "number of workers"
>>>
>>> Can you clarify?
>>>
>>> -Matthias
>>>
>>> On 02/11/2016 05:14 AM, anshu shukla wrote:
>>> > Not  like that..  But i have used workers equal to number of cores.
>>> Each
>>> > vm with 8 corea.
>>> >
>>> > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
>>> > >> > > wrote:
>>> >
>>> > more than one worker on same node ? Did u use custom schedule r?
>>> > because by default, the workers would be spread throughout the
>>> cluster
>>> >
>>>
>>>
>>
>> --
>> Rudraneel Chakraborty
>> Carleton University Real Time and Distributed Systems Reserach
>>
>>
>
> --
> Rudraneel Chakraborty
> Carleton University Real Time and Distributed Systems Reserach
>
>

Re: Need Help regarding topology with numWorker>1

2016-02-10 Thread anshu shukla

Not  like that..  But i have used workers equal to number of cores. Each vm
with 8 corea.
On 11 Feb 2016 9:07 am, "Rudraneel chakraborty" <
rudraneel.chakrabo...@gmail.com> wrote:

> more than one worker on same node ? Did u use custom schedule r? because
> by default, the workers would be spread throughout the cluster
>

Need Help regarding topology with numWorker>1

2016-02-10 Thread Rudraneel chakraborty

Hello Good People,

I desperately need an example of a topology which performs better with more
than one worker process compared to a single worker. Could anyone help

Re: Need Help regarding topology with numWorker>1

2016-02-10 Thread anshu shukla

Since  every worker  process have only  one thread for transfer of message
from  network to executor queue so more often it becomes a bottkeneck when
input rate is high..  That forces us to have more than 1 worker on same
node.
I dont think other than this there is any logic dependent topo case where
we are forced to use more than 1 worker process.
On 11 Feb 2016 8:57 am, "Rudraneel chakraborty" <
rudraneel.chakrabo...@gmail.com> wrote:

> Hello Good People,
>
> I desperately need an example of a topology which performs better with
> more than one worker process compared to a single worker. Could anyone help
>
>
>

Re: Need Help regarding topology with numWorker>1

2016-02-10 Thread Rudraneel chakraborty

more than one worker on same node ? Did u use custom schedule r? because by
default, the workers would be spread throughout the cluster

Fwd: Need help on integration of Apache Storm with WebSphere MQ

2016-01-21 Thread Sameer Kirange

Hi,

I'm quite new to world of IBM Websphere MQ and need advise on one of the
Integration scenario.

In this scenario, we need to integrate MQ with Apache Storm, We need to
read the MQ in Apache Storm spout so that message can be processed in bolt.

Has anybody worked on this integration and is able to successfully receive
the messages from IBM MQ into Storm for processing ? If yes can some
pointers be provided on how to achieve it ?



I understand from the various mailing lists and the blog pages from
hortwonworks that that IBM MQ does not support the integration with Apache
Storm. It states following reasons for the same.

"We ran into issues with IBM MQ-Series with respect to how messages
are acknowledged by IBM
MQ. IBM-MQ requires the thread that receives the message be the same
thread that acks it.
Storm's framework cannot support this requirement, as the receiving
and acking thread by
design are different threads to achieve higher throughput.

However when I searched further i came to know from the reply of Nathan
Marz that

That's not accurate. A spout task's nextTuple, ack, and fail methods
are all called by the
exact same thread. You're confusing Storm acks with acks that go back
to the source message
queue. Storm acks are about detecting tuple DAG completion and have
nothing to do with the
ack to the source message queue

 Are there any guidelines available to achieve the integration of IBM MQ
and Apache Storm  ?

Thanks and Regards,

Sameer

Re: Need help on integration of Apache Storm with WebSphere MQ

2016-01-21 Thread Don Frascuchon

Hi Sameer,

I worked with MQSeries and storm using the MQSeries JMS API implementation
and  https://github.com/ptgoetz/storm-jms components

Best regards!

El jue., 21 ene. 2016 a las 13:26, Sameer Kirange ()
escribió:

> Hi,
>
> I'm quite new to world of IBM Websphere MQ and need advise on one of the
> Integration scenario.
>
> In this scenario, we need to integrate MQ with Apache Storm, We need to
> read the MQ in Apache Storm spout so that message can be processed in bolt.
>
> Has anybody worked on this integration and is able to successfully receive
> the messages from IBM MQ into Storm for processing ? If yes can some
> pointers be provided on how to achieve it ?
>
>
>
> I understand from the various mailing lists and the blog pages from
> hortwonworks that that IBM MQ does not support the integration with Apache
> Storm. It states following reasons for the same.
>
> "We ran into issues with IBM MQ-Series with respect to how messages are 
> acknowledged by IBM
> MQ. IBM-MQ requires the thread that receives the message be the same thread 
> that acks it.
> Storm's framework cannot support this requirement, as the receiving and 
> acking thread by
> design are different threads to achieve higher throughput.
>
> However when I searched further i came to know from the reply of Nathan
> Marz that
>
> That's not accurate. A spout task's nextTuple, ack, and fail methods are all 
> called by the
> exact same thread. You're confusing Storm acks with acks that go back to the 
> source message
> queue. Storm acks are about detecting tuple DAG completion and have nothing 
> to do with the
> ack to the source message queue
>
>  Are there any guidelines available to achieve the integration of IBM MQ
> and Apache Storm  ?
>
> Thanks and Regards,
>
> Sameer
>
>

Re: Need help on integration of Apache Storm with WebSphere MQ

2016-01-21 Thread Seungtack Baek

We do have both MQ and Storm in-house. However, we actually have another java 
process running to read from MQ (polling and stopping every x seconds when the 
queue is empty) and push message to kafka. We thought it was easier to use 
KafkaSpout then creating another spout. Not saying this is the right way to do 
it, but we got by with this :)

Thanks,
Baek

> On Jan 21, 2016, at 6:26 AM, Sameer Kirange  wrote:
> 
> Hi,
> I'm quite new to world of IBM Websphere MQ and need advise on one of the 
> Integration scenario.
> In this scenario, we need to integrate MQ with Apache Storm, We need to read 
> the MQ in Apache Storm spout so that message can be processed in bolt.
> Has anybody worked on this integration and is able to successfully receive 
> the messages from IBM MQ into Storm for processing ? If yes can some pointers 
> be provided on how to achieve it ?
>  
> I understand from the various mailing lists and the blog pages from 
> hortwonworks that that IBM MQ does not support the integration with Apache 
> Storm. It states following reasons for the same.
> "We ran into issues with IBM MQ-Series with respect to how messages are 
> acknowledged by IBM
> MQ. IBM-MQ requires the thread that receives the message be the same thread 
> that acks it.
> Storm's framework cannot support this requirement, as the receiving and 
> acking thread by
> design are different threads to achieve higher throughput.
> However when I searched further i came to know from the reply of Nathan Marz 
> that 
> That's not accurate. A spout task's nextTuple, ack, and fail methods are all 
> called by the
> exact same thread. You're confusing Storm acks with acks that go back to the 
> source message
> queue. Storm acks are about detecting tuple DAG completion and have nothing 
> to do with the
> ack to the source message queue
>  Are there any guidelines available to achieve the integration of IBM MQ and 
> Apache Storm  ?
> Thanks and Regards,
> Sameer
>

Re: Need help in this problem " Address already in use(0x62)"

2015-11-07 Thread researcher cs

i upgraded the version to storm -0.9.5 and got in supervisor log file that
it hasn't start still but i executed tha commend that launch between
supervisor and worker and got this
[ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization") at
backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325)
[storm-core-0.9.5.jar:0.9.5] at clojure.lang.RestFn.invoke(RestFn.java:423)
[clojure-1.5.1.jar:na] at
backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391)
[storm-core-0.9.5.jar:0.9.5] at clojure.lang.RestFn.invoke(RestFn.java:512)
[clojure-1.5.1.jar:na] at
backtype.storm.daemon.worker$_main.invoke(worker.clj:502)
[storm-core-0.9.5.jar:0.9.5] at
clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na] at
clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at
backtype.storm.daemon.worker.main(Unknown Source)
[storm-core-0.9.5.jar:0.9.5]

On Tue, Nov 3, 2015 at 8:36 AM, researcher cs 
wrote:

> i tried this command  ps -ef|grep 6703  and got this
>
> st 2991 2595 41 14:51 pts/3 00:05:19 java -server -Xmx768m
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/st/storm-0.8.2
> -Dlog4j.configuration=storm.log.properties -cp
> /home/st/storm-0.8.2/storm-0.8.2.jar:/home/st/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/st/storm-0.8.2/lib/zookeeper-3.4.5.jar:/home/st/storm-0.8.2/lib/objenesis-1.2.jar:/home/st/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/st/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/st/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/st/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/st/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/st/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/st/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/st/storm-0.8.2/lib/guava-13.0.jar:/home/st/storm-0.8.2/lib/junit-3.8.1.jar:/home/st/storm-0.8.2/lib/servlet-api-2.5.jar:/home/st/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/st/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/st/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/st/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/st/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/st/storm-0.8.2/lib/clojure-1.4.0.jar:/home/st/storm-0.8.2/lib/clout-1.0.1.jar:/home/st/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/st/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/st/storm-0.8.2/lib/asm-4.0.jar:/home/st/storm-0.8.2/lib/commons-io-1.4.jar:/home/st/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/st/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/st/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/st/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/st/storm-0.8.2/lib/commons-exec-1.1.jar:/home/st/storm-0.8.2/lib/minlog-1.2.jar:/home/st/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/st/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/st/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/st/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/st/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/st/storm-0.8.2/lib/commons-codec-1.4.jar:/home/st/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/st/storm-0.8.2/lib/jetty-6.1.26.jar:/home/st/storm-0.8.2/lib/jline-0.9.94.jar:/home/st/storm-0.8.2/lib/json-simple-1.1.jar:/home/st/storm-0.8.2/lib/kryo-2.17.jar:/home/st/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/st/storm-0.8.2/lib/compojure-1.1.3.jar:/home/st/storm-0.8.2/lib/httpcore-4.1.jar:/home/st/storm-0.8.2/lib/commons-lang-2.5.jar:/home/st/storm-0.8.2/lib/log4j-1.2.16.jar:/home/st/storm-0.8.2/lib/joda-time-2.0.jar:/home/st/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/st/storm-0.8.2/log4j:/home/st/storm-0.8.2/conf:/tmp/storm/supervisor/stormdist/fsd-1-1446555065/stormjar.jar
> backtype.storm.daemon.worker fsd-1-1446555065
> 3e3092ff-2722-400a-8aa9-362c9144a6fe 6703
> 0f68434a-d2f8-41c3-8d1b-9b0d0f2fcb6a st 3434 3217 2 15:04 pts/8 00:00:00
> grep --color=auto 6703
>
> and tried to change ip_local_port_range as you said but problem still
> so i'll try to upgrade the version but is there any other solution ?
>
> On Mon, Nov 2, 2015 at 2:40 PM, researcher cs 
> wrote:
>
>> many many many thanks for replying and helping , i'll try what you said ,
>> But the coder of the project that i used it , he used old version 0.8.2 so
>> i'm trying first to use it then upgrade the version
>>
>> On Mon, Nov 2, 2015 at 1:59 PM, Feng Longda  wrote:
>>
>>> I have resolved this problem several times.
>>>
>>> There are two root cause.
>>> (1) local temporary network ports are conflict with storm's ports.
>>> (2) the old worker failed to be killed when kill topology.
>>>
>>> Firstly, please do "ps -ef|grep 67xx" to check whether it is due to the
>>> second problem. if there is no worker using the port. it would be temporary
>>> port conflict.
>>> please do the following action with root privilege
>>>
>>> echo 'net.ipv4.ip_local_port_range = 10240 65535' >> /etc/sysctl.conf
>>> /sbin/sysctl -p
>>>
>>> why occur this, due to in every network connection,  client

Re: Need help in this problem " Address already in use(0x62)"

2015-11-03 Thread researcher cs

i tried this command  ps -ef|grep 6703  and got this

st 2991 2595 41 14:51 pts/3 00:05:19 java -server -Xmx768m
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dlogfile.name=worker-6703.log -Dstorm.home=/home/st/storm-0.8.2
-Dlog4j.configuration=storm.log.properties -cp
/home/st/storm-0.8.2/storm-0.8.2.jar:/home/st/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/st/storm-0.8.2/lib/zookeeper-3.4.5.jar:/home/st/storm-0.8.2/lib/objenesis-1.2.jar:/home/st/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/st/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/st/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/st/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/st/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/st/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/st/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/st/storm-0.8.2/lib/guava-13.0.jar:/home/st/storm-0.8.2/lib/junit-3.8.1.jar:/home/st/storm-0.8.2/lib/servlet-api-2.5.jar:/home/st/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/st/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/st/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/st/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/st/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/st/storm-0.8.2/lib/clojure-1.4.0.jar:/home/st/storm-0.8.2/lib/clout-1.0.1.jar:/home/st/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/st/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/st/storm-0.8.2/lib/asm-4.0.jar:/home/st/storm-0.8.2/lib/commons-io-1.4.jar:/home/st/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/st/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/st/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/st/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/st/storm-0.8.2/lib/commons-exec-1.1.jar:/home/st/storm-0.8.2/lib/minlog-1.2.jar:/home/st/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/st/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/st/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/st/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/st/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/st/storm-0.8.2/lib/commons-codec-1.4.jar:/home/st/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/st/storm-0.8.2/lib/jetty-6.1.26.jar:/home/st/storm-0.8.2/lib/jline-0.9.94.jar:/home/st/storm-0.8.2/lib/json-simple-1.1.jar:/home/st/storm-0.8.2/lib/kryo-2.17.jar:/home/st/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/st/storm-0.8.2/lib/compojure-1.1.3.jar:/home/st/storm-0.8.2/lib/httpcore-4.1.jar:/home/st/storm-0.8.2/lib/commons-lang-2.5.jar:/home/st/storm-0.8.2/lib/log4j-1.2.16.jar:/home/st/storm-0.8.2/lib/joda-time-2.0.jar:/home/st/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/st/storm-0.8.2/log4j:/home/st/storm-0.8.2/conf:/tmp/storm/supervisor/stormdist/fsd-1-1446555065/stormjar.jar
backtype.storm.daemon.worker fsd-1-1446555065
3e3092ff-2722-400a-8aa9-362c9144a6fe 6703
0f68434a-d2f8-41c3-8d1b-9b0d0f2fcb6a st 3434 3217 2 15:04 pts/8 00:00:00
grep --color=auto 6703

and tried to change ip_local_port_range as you said but problem still
so i'll try to upgrade the version but is there any other solution ?

On Mon, Nov 2, 2015 at 2:40 PM, researcher cs 
wrote:

> many many many thanks for replying and helping , i'll try what you said ,
> But the coder of the project that i used it , he used old version 0.8.2 so
> i'm trying first to use it then upgrade the version
>
> On Mon, Nov 2, 2015 at 1:59 PM, Feng Longda  wrote:
>
>> I have resolved this problem several times.
>>
>> There are two root cause.
>> (1) local temporary network ports are conflict with storm's ports.
>> (2) the old worker failed to be killed when kill topology.
>>
>> Firstly, please do "ps -ef|grep 67xx" to check whether it is due to the
>> second problem. if there is no worker using the port. it would be temporary
>> port conflict.
>> please do the following action with root privilege
>>
>> echo 'net.ipv4.ip_local_port_range = 10240 65535' >> /etc/sysctl.conf
>> /sbin/sysctl -p
>>
>> why occur this, due to in every network connection,  client will bind one
>> temporary port, if the linux default temporary range is from 1024 to 65535,
>> so it is still likely to conflict with storm's port.
>>
>>
>> if there are some alive worker using the port, it is due to storm fail to
>> kill worker when kill topology, when the OS's cpu usage is pretty high,
>> this phenomenon will occur, but generally this case is pretty rare. By the
>> way, in the jstorm (https://github.com/alibaba/jstorm), it resolves this
>> issue. Fortunately, jstorm has been donated to storm, you will see  the
>> solution in a short time.
>>
>>
>> By the way, I found you are using an old version storm, why don't try the
>> storm 0.9.x, whose performance has been improved much.
>>
>>
>>
>>
>>
>>
>>
>>
>> 2015-11-01 17:26 GMT+08:00 researcher cs :
>>
>>> after submitting topology
>>>
>>> supervisor log file
>>>
>>> 2015-11-01 09:59:48 executor [INFO] Loading executor b-1:[3 3]
>>> 2015-11-01 09:59:50 executor [INFO] Loaded executor tasks b-1:[3 3][INFO]
>>> Launching worker with assignment
>>>

Re: Need help in this problem " Address already in use(0x62)"

2015-11-02 Thread Feng Longda

I have resolved this problem several times.

There are two root cause.
(1) local temporary network ports are conflict with storm's ports.
(2) the old worker failed to be killed when kill topology.

Firstly, please do "ps -ef|grep 67xx" to check whether it is due to the
second problem. if there is no worker using the port. it would be temporary
port conflict.
please do the following action with root privilege

echo 'net.ipv4.ip_local_port_range = 10240 65535' >> /etc/sysctl.conf
/sbin/sysctl -p

why occur this, due to in every network connection,  client will bind one
temporary port, if the linux default temporary range is from 1024 to 65535,
so it is still likely to conflict with storm's port.


if there are some alive worker using the port, it is due to storm fail to
kill worker when kill topology, when the OS's cpu usage is pretty high,
this phenomenon will occur, but generally this case is pretty rare. By the
way, in the jstorm (https://github.com/alibaba/jstorm), it resolves this
issue. Fortunately, jstorm has been donated to storm, you will see  the
solution in a short time.


By the way, I found you are using an old version storm, why don't try the
storm 0.9.x, whose performance has been improved much.








2015-11-01 17:26 GMT+08:00 researcher cs :

> after submitting topology
>
> supervisor log file
>
> 2015-11-01 09:59:48 executor [INFO] Loading executor b-1:[3 3] 2015-11-01
> 09:59:50 executor [INFO] Loaded executor tasks b-1:[3 3][INFO] Launching
> worker with assignment
> #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
> "df-1-1446364738", :executors ([2 2] [35 35] [5 5] [38 38] [8 8] [41 41]
> [11 11] [44 44] [14 14] [47 47] [17 17] [50 50] [20 20] [53 53] [23 23] [56
> 56] [26 26] [29 29] [32 32])} for this supervisor
> fdd1e16a-650e-4d12-90f4-cb87336f29c3 on port 6702 with id
> eb224d37-bc81-43ec-bbb9-8e5897c203fa
> after
> 2015-11-01 09:59:02 supervisor [INFO] 0af4b32b-e14d-4f64-ba03-d61d79fa6405
> still hasn't started 2015-11-01 09:59:03 supervisor [INFO]
> 0af4b32b-e14d-4f64-ba03-d61d79fa6405 still hasn't started
> !!
>
> but worker has data like
>
> 2015-11-01 09:59:48 executor [INFO] Loading executor b-1:[3 3] 2015-11-01
> 09:59:50 executor [INFO] Loaded executor tasks b-1:[3 3]
>
> then i executed the command which launch worker with supervisor to know
> where is the error exactly  found this
> in the worker log file
> [ERROR] Async loop died! org.zeromq.ZMQException: Address already in
> use(0x62) at org.zeromq.ZMQ$Socket.bind(Native Method) at 
> zilch.mq$bind.invoke(mq.clj:69)
> at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57) at
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375) at
> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown Source)
> 2015-11-01 10:03:19 util [INFO] Halting process
>
>
> i'm on this error more than 3 weeks !! hope really can find someone help
>

Need help in this problem " Address already in use(0x62)"

2015-11-01 Thread researcher cs

 after submitting topology

supervisor log file

2015-11-01 09:59:48 executor [INFO] Loading executor b-1:[3 3] 2015-11-01
09:59:50 executor [INFO] Loaded executor tasks b-1:[3 3][INFO] Launching
worker with assignment
#backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"df-1-1446364738", :executors ([2 2] [35 35] [5 5] [38 38] [8 8] [41 41]
[11 11] [44 44] [14 14] [47 47] [17 17] [50 50] [20 20] [53 53] [23 23] [56
56] [26 26] [29 29] [32 32])} for this supervisor
fdd1e16a-650e-4d12-90f4-cb87336f29c3 on port 6702 with id
eb224d37-bc81-43ec-bbb9-8e5897c203fa
after
2015-11-01 09:59:02 supervisor [INFO] 0af4b32b-e14d-4f64-ba03-d61d79fa6405
still hasn't started 2015-11-01 09:59:03 supervisor [INFO]
0af4b32b-e14d-4f64-ba03-d61d79fa6405 still hasn't started
!!

but worker has data like

2015-11-01 09:59:48 executor [INFO] Loading executor b-1:[3 3] 2015-11-01
09:59:50 executor [INFO] Loaded executor tasks b-1:[3 3]

then i executed the command which launch worker with supervisor to know
where is the error exactly  found this
in the worker log file
[ERROR] Async loop died! org.zeromq.ZMQException: Address already in
use(0x62) at org.zeromq.ZMQ$Socket.bind(Native Method) at
zilch.mq$bind.invoke(mq.clj:69)
at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57) at
backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375) at
clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown Source)
2015-11-01 10:03:19 util [INFO] Halting process


i'm on this error more than 3 weeks !! hope really can find someone help

Re: need help in Storm.yaml

2015-09-21 Thread Javier Gonzalez

This is all the configuration options you can set in the storm.yaml file.
Of interest to you are the drpc.* keys:

https://storm.apache.org/javadoc/apidocs/constant-values.html#backtype.storm.Config

Regards,
Javier

On Mon, Sep 21, 2015 at 7:42 PM, researcher cs 
wrote:

> i'm new in storm how can i fill storm.yaml with appropriate configuration
> here what i wrote
> storm.zookeeper.servers:
> - "127.0.0.1"
>
> nimbus.host: "127.0.0.1"
> storm.local.dir: /tmp/storm
> but if i'll use drpc should i write it here or not and how ?
>
>
>


-- 
Javier González Nicolini

need help in Storm.yaml

2015-09-21 Thread researcher cs

i'm new in storm how can i fill storm.yaml with appropriate configuration
here what i wrote
storm.zookeeper.servers:
- "127.0.0.1"

nimbus.host: "127.0.0.1"
storm.local.dir: /tmp/storm
but if i'll use drpc should i write it here or not and how ?

Re: Need help

2015-05-15 Thread Jeffery Maass

See the 0.9.4 release codebase @ https://github.com/apache/storm/tree/v0.9.4
There is a project called Storm Starter @
https://github.com/apache/storm/tree/v0.9.4/examples/storm-starter


Thank you for your time!

+
Jeff Maass maas...@gmail.com
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+


On Fri, May 15, 2015 at 2:43 AM, Asif Ihsan asifihsan.ih...@gmail.com
wrote:

 I am new to Storm. I studied that Storm is reliable and gives message
 delivery guaranty. I tried to see how it works, but I couldn't do it. can
 anyone help me with the single spout and single bolt. Every message sent by
 the spout to bolt should be acknowledged by the bolt to the spout. And
 display message that stream acknowledged. Thank you.

 --
 

 Regards
 Muhammad Asif Ihsan

RE: Need help

2015-05-15 Thread Rajesh_Kalluri

Dell - Internal Use - Confidential
If you learn better by reading a book try http://www.manning.com/sallen/.

It starts with basics and ramps up to reliable message processing  and beyond.

From: Jeffery Maass [mailto:maas...@gmail.com]
Sent: Friday, May 15, 2015 10:18 AM
To: user@storm.apache.org
Subject: Re: Need help

Totally my bad.  I did not actually go look at the spouts to see if they were 
implemented as reliable spouts or not.

If you haven't already read these, I would read them now:
https://storm.apache.org/documentation/Concepts.html
https://storm.apache.org/documentation/Guaranteeing-message-processing.html
I learned about reliable spouts by dissecting the Kafka spout:
https://github.com/apache/storm/blob/v0.9.4/external/storm-kafka/src/jvm/storm/kafka/KafkaSpout.java

Sorry, I don't have a simpler example on hand.

The essentials are this:

* Your spout handles :
** Gathering records from its datasource
** Sending records to the rest of the topology ( emit )
** what to do when a tuple successfully completes the topology ( ack )
** what to do when a tuple fails somewhere in the topology ( fail )
* Storm handles :
** delivery to all bolts in a topology of an individual tuple
** guaranteed reporting of an individual tuple's success or failure back to the 
originating spout
* To make use of guaranteed message processing:
** When your spout emits a tuple, it must emit a unique message id.  This 
message id is returned to the spout whenever a report is made on the tuple ( 
ack or fail )
** When your bolt is finished with a tuple, it must notify Storm using either 
ack(tuple) or fail(tuple)

Thank you for your time!

+
Jeff Maassmailto:maas...@gmail.com
linkedin.com/in/jeffmaasshttp://linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassqlhttp://stackoverflow.com/users/373418/maassql
+

On Fri, May 15, 2015 at 9:51 AM, Asif Ihsan 
asifihsan.ih...@gmail.commailto:asifihsan.ih...@gmail.com wrote:
That project doesn't do anything about message delivery. I have to make sure 
guaranteed processing of the message sent by the spout to bolt.

On Fri, May 15, 2015 at 5:33 PM, Jeffery Maass 
maas...@gmail.commailto:maas...@gmail.com wrote:
See the 0.9.4 release codebase @ https://github.com/apache/storm/tree/v0.9.4
There is a project called Storm Starter @ 
https://github.com/apache/storm/tree/v0.9.4/examples/storm-starter

Thank you for your time!

+
Jeff Maassmailto:maas...@gmail.com
linkedin.com/in/jeffmaasshttp://linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassqlhttp://stackoverflow.com/users/373418/maassql
+

On Fri, May 15, 2015 at 2:43 AM, Asif Ihsan 
asifihsan.ih...@gmail.commailto:asifihsan.ih...@gmail.com wrote:
I am new to Storm. I studied that Storm is reliable and gives message delivery 
guaranty. I tried to see how it works, but I couldn't do it. can anyone help me 
with the single spout and single bolt. Every message sent by the spout to bolt 
should be acknowledged by the bolt to the spout. And display message that 
stream acknowledged. Thank you.

--

Regards
Muhammad Asif Ihsan

--
Regards
Muhammad Asif Ihsan

Re: Need help

2015-05-15 Thread Asif Ihsan

That project doesn't do anything about message delivery. I have to make
sure guaranteed processing of the message sent by the spout to bolt.

On Fri, May 15, 2015 at 5:33 PM, Jeffery Maass maas...@gmail.com wrote:

 See the 0.9.4 release codebase @
 https://github.com/apache/storm/tree/v0.9.4
 There is a project called Storm Starter @
 https://github.com/apache/storm/tree/v0.9.4/examples/storm-starter


 Thank you for your time!

 +
 Jeff Maass maas...@gmail.com
 linkedin.com/in/jeffmaass
 stackoverflow.com/users/373418/maassql
 +


 On Fri, May 15, 2015 at 2:43 AM, Asif Ihsan asifihsan.ih...@gmail.com
 wrote:

 I am new to Storm. I studied that Storm is reliable and gives message
 delivery guaranty. I tried to see how it works, but I couldn't do it. can
 anyone help me with the single spout and single bolt. Every message sent by
 the spout to bolt should be acknowledged by the bolt to the spout. And
 display message that stream acknowledged. Thank you.

 --
 

 Regards
 Muhammad Asif Ihsan





-- 
Regards
Muhammad Asif Ihsan

Re: Need help

2015-05-15 Thread Enno Shioji

Try this?
https://github.com/wurstmeister/storm-kafka-0.8-plus-test/blob/master/src/main/java/storm/kafka/trident/SentenceAggregationTopology.java

On Fri, May 15, 2015 at 3:51 PM, Asif Ihsan asifihsan.ih...@gmail.com
wrote:

 That project doesn't do anything about message delivery. I have to make
 sure guaranteed processing of the message sent by the spout to bolt.

 On Fri, May 15, 2015 at 5:33 PM, Jeffery Maass maas...@gmail.com wrote:

 See the 0.9.4 release codebase @
 https://github.com/apache/storm/tree/v0.9.4
 There is a project called Storm Starter @
 https://github.com/apache/storm/tree/v0.9.4/examples/storm-starter


 Thank you for your time!

 +
 Jeff Maass maas...@gmail.com
 linkedin.com/in/jeffmaass
 stackoverflow.com/users/373418/maassql
 +


 On Fri, May 15, 2015 at 2:43 AM, Asif Ihsan asifihsan.ih...@gmail.com
 wrote:

 I am new to Storm. I studied that Storm is reliable and gives message
 delivery guaranty. I tried to see how it works, but I couldn't do it. can
 anyone help me with the single spout and single bolt. Every message sent by
 the spout to bolt should be acknowledged by the bolt to the spout. And
 display message that stream acknowledged. Thank you.

 --
 

 Regards
 Muhammad Asif Ihsan





 --
 Regards
 Muhammad Asif Ihsan

Re: Need help

2015-05-15 Thread Asif Ihsan

Thank you all.

On Fri, May 15, 2015 at 8:42 PM, rajesh_kall...@dellteam.com wrote:

*Dell - Internal Use - Confidential *

If you learn better by reading a book try http://www.manning.com/sallen/.

It starts with basics and ramps up to reliable message processing and
beyond.

*From:* Jeffery Maass [mailto:maas...@gmail.com]
*Sent:* Friday, May 15, 2015 10:18 AM
*To:* user@storm.apache.org
*Subject:* Re: Need help

Totally my bad. I did not actually go look at the spouts to see if they
were implemented as reliable spouts or not.

If you haven't already read these, I would read them now:
https://storm.apache.org/documentation/Concepts.html
https://storm.apache.org/documentation/Guaranteeing-message-processing.html

I learned about reliable spouts by dissecting the Kafka spout:

https://github.com/apache/storm/blob/v0.9.4/external/storm-kafka/src/jvm/storm/kafka/KafkaSpout.java

Sorry, I don't have a simpler example on hand.

The essentials are this:

* Your spout handles :

** Gathering records from its datasource

** Sending records to the rest of the topology ( emit )

** what to do when a tuple successfully completes the topology ( ack )

** what to do when a tuple fails somewhere in the topology ( fail )

* Storm handles :
** delivery to all bolts in a topology of an individual tuple

** guaranteed reporting of an individual tuple's success or failure back
to the originating spout

* To make use of guaranteed message processing:
** When your spout emits a tuple, it must emit a unique message id. This
message id is returned to the spout whenever a report is made on the tuple
( ack or fail )

** When your bolt is finished with a tuple, it must notify Storm using
either ack(tuple) or fail(tuple)

Thank you for your time!

+
Jeff Maass maas...@gmail.com
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+

On Fri, May 15, 2015 at 9:51 AM, Asif Ihsan asifihsan.ih...@gmail.com
wrote:

That project doesn't do anything about message delivery. I have to make
sure guaranteed processing of the message sent by the spout to bolt.

On Fri, May 15, 2015 at 5:33 PM, Jeffery Maass maas...@gmail.com wrote:

See the 0.9.4 release codebase @
https://github.com/apache/storm/tree/v0.9.4

There is a project called Storm Starter @
https://github.com/apache/storm/tree/v0.9.4/examples/storm-starter

Thank you for your time!

+
Jeff Maass maas...@gmail.com
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+

On Fri, May 15, 2015 at 2:43 AM, Asif Ihsan asifihsan.ih...@gmail.com
wrote:

I am new to Storm. I studied that Storm is reliable and gives message
delivery guaranty. I tried to see how it works, but I couldn't do it. can
anyone help me with the single spout and single bolt. Every message sent by
the spout to bolt should be acknowledged by the bolt to the spout. And
display message that stream acknowledged. Thank you.

Regards

Muhammad Asif Ihsan

Regards

Muhammad Asif Ihsan

--
Regards
Muhammad Asif Ihsan

Re: Need help with troubleshooting a worker not processing messages

2015-03-20 Thread Girish Joshi

Thanks Taylor for your response.

In my case, I have seen that 4 of my 15 kafka executors do not process any
data; I will check what the kafka # of partitions is but looks like it may
be just 11 in which case I should reduce the number of kafka executors.

around 50 of the 550 mapperBoltExecutors I have do not process anything and
I am now guessing that is because if my maxSpoutPending (500) is low and so
there are not enough tuples to be processed in 550 mapperBoltExecutors.

Do you know if maxSpoutPending is the maximum number of unacked tuples from
a single spout executor or from all spout executors combined? If it is the
later, then my guess makes sense since if there are only 500 tuples
unacked, they will need only 500 more bolts to process them.

kafkaSpoutExecutors: 15
mapperBoltExecutors: 550
workers: 9
maxSpoutPending: 500



On Thu, Mar 19, 2015 at 8:55 PM, P. Taylor Goetz ptgo...@gmail.com wrote:

 More information about your topology would help, but..

 I’ll assume you’re using a core API topology (spouts/bolts).

 On the kafka spout side, does the spout parallelism == the # of kafka
 partitions? (It should.)

  On the bolt side, are you using fields groupings at all, and if so, what
 does the distribution of those fields look like?

 To changel the logging level, edit the log back config files in
 .//storm/logback, if running locally, add or edit a logback config file in
 your project.

 -Taylor

 On Mar 19, 2015, at 7:11 PM, Girish Joshi gjo...@groupon.com wrote:

 I am trying to troubleshoot an issue with our storm cluster where a worker
 process on one of the machines in the cluster does not perform any work.
 All the counts(emitted/transferred/executed) for all executors in that
 worker are 0 as shown below. Even if I restart the worker, storm supervisor
 starts a new one and that does not process any work either.

 [120-120]26m 17sstorm6-prod6702
 http://watson-storm6-prod.lup1:8000/log?file=worker-6702.log000.000
 0.0.0

 Supervisor logs shows that the worker is started and the worker log just
 has a bunch of zookeeper messages printed every minute.

 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Refreshing partition manager
 connections
 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Deleted partition managers: []
 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] New partition managers: []
 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Finished refreshing

 I am looking for some debugging help and have following questions. If you
 have any suggestions , I will appreciate that.

 - From the storm UI, it looks like the worker process is up and running
 and is assigned to executing tasks from all bolts and spouts in the
 topology. But it does not get any messages to work on. Is there a way I can
 find out why is storm infrastructure routing any messages to any of the
 bolts running in that process? For spouts, since they are reading from
 kafka, I could understand that there are no partitions left for this worker
 to read from and so it does not have anything to read. But I would expect
 messages from other kafka spouts to be routed to bolts in this worker
 process.

 - Is there a way I can enable debug logging for storm which can tell me
 why a particular worker process is not getting any messages/tuples to
 execute?

 Thanks,

 Girish.

Need help with troubleshooting a worker not processing messages

2015-03-19 Thread Girish Joshi

I am trying to troubleshoot an issue with our storm cluster where a worker
process on one of the machines in the cluster does not perform any work.
All the counts(emitted/transferred/executed) for all executors in that
worker are 0 as shown below. Even if I restart the worker, storm supervisor
starts a new one and that does not process any work either.

[120-120]26m 17sstorm6-prod6702
http://watson-storm6-prod.lup1:8000/log?file=worker-6702.log000.000
0.0.0

Supervisor logs shows that the worker is started and the worker log just
has a bunch of zookeeper messages printed every minute.

2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Refreshing partition manager
connections
2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Deleted partition managers: []
2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] New partition managers: []
2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Finished refreshing

I am looking for some debugging help and have following questions. If you
have any suggestions , I will appreciate that.

- From the storm UI, it looks like the worker process is up and running and
is assigned to executing tasks from all bolts and spouts in the topology.
But it does not get any messages to work on. Is there a way I can find out
why is storm infrastructure routing any messages to any of the bolts
running in that process? For spouts, since they are reading from kafka, I
could understand that there are no partitions left for this worker to read
from and so it does not have anything to read. But I would expect messages
from other kafka spouts to be routed to bolts in this worker process.

- Is there a way I can enable debug logging for storm which can tell me why
a particular worker process is not getting any messages/tuples to execute?

Thanks,

Girish.

Re: Need help with troubleshooting a worker not processing messages

2015-03-19 Thread P. Taylor Goetz

More information about your topology would help, but..

I’ll assume you’re using a core API topology (spouts/bolts).

On the kafka spout side, does the spout parallelism == the # of kafka 
partitions? (It should.)

 On the bolt side, are you using fields groupings at all, and if so, what does 
the distribution of those fields look like?

To changel the logging level, edit the log back config files in 
.//storm/logback, if running locally, add or edit a logback config file in your 
project.

-Taylor
On Mar 19, 2015, at 7:11 PM, Girish Joshi gjo...@groupon.com wrote:

 I am trying to troubleshoot an issue with our storm cluster where a worker 
 process on one of the machines in the cluster does not perform any work. All 
 the counts(emitted/transferred/executed) for all executors in that worker are 
 0 as shown below. Even if I restart the worker, storm supervisor starts a new 
 one and that does not process any work either. 
 
 [120-120]26m 17sstorm6-prod6702000..0.0
 
 Supervisor logs shows that the worker is started and the worker log just has 
 a bunch of zookeeper messages printed every minute. 
 
 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Refreshing partition manager 
 connections
 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Deleted partition managers: []
 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] New partition managers: []
 2015-03-19 22:25:07 s.k.ZkCoordinator [INFO] Finished refreshing
 
 I am looking for some debugging help and have following questions. If you 
 have any suggestions , I will appreciate that.
 
 - From the storm UI, it looks like the worker process is up and running and 
 is assigned to executing tasks from all bolts and spouts in the topology. But 
 it does not get any messages to work on. Is there a way I can find out why is 
 storm infrastructure routing any messages to any of the bolts running in that 
 process? For spouts, since they are reading from kafka, I could understand 
 that there are no partitions left for this worker to read from and so it does 
 not have anything to read. But I would expect messages from other kafka 
 spouts to be routed to bolts in this worker process.
 
 - Is there a way I can enable debug logging for storm which can tell me why a 
 particular worker process is not getting any messages/tuples to execute?
 
 Thanks,
 
 Girish.



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Need help on kafka-storm

2014-12-01 Thread Madabhattula Rajesh Kumar

Hello,

Could any one help me on above mail query?

Regards,
Rajesh

On Sat, Nov 29, 2014 at 10:30 PM, Madabhattula Rajesh Kumar 
mrajaf...@gmail.com wrote:

 Hello,

 I'm new to Storm and Kafka. I have tried Strom-Kafka integration example
 program. Now I'm able to send message from Kafka and receive those messages
 in storm topology.

 I have observed one thing in storm topology, same messages are processing
 continuously

 *I have sent three messages (First Message, Second Message, Third Message
 ). These 3 messages processing continuously, please find below console log
 file*


 *Could you please help me on below query*

-
 *How to make sure that storm topology process messages one time
successfully(Not multiple times). *
- *What configurations I need to do *

 *Below is my code :*











 * BrokerHosts zk = new ZkHosts(localhost:2181);SpoutConfig
 spoutConf = new SpoutConfig(zk, test-topic, /kafkastorm,
 discovery);spoutConf.scheme = new SchemeAsMultiScheme(new
 StringScheme());KafkaSpout spout = new KafkaSpout(spoutConf);
 TopologyBuilder builder = new TopologyBuilder();
 builder.setSpout(spout, spout, 1);builder.setBolt(printerbolt,
 new PrintBolt()) .shuffleGrouping(spout);Config config = new
 Config();config.setDebug(true);LocalCluster cluster = new
 LocalCluster();cluster.submitTopology(kafka, config,
 builder.createTopology());*

 *Log file :*

 25526 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [First Message]
 25528 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-5148901491748001310=-1334200518948214946}, [First Message]
 *message [First Message]*
 25538 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-5148901491748001310 -1334200518948214946 3]
 25539 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-5148901491748001310 -1334200518948214946 3]
 33530 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [Second Message]
 33531 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-8623931148894813393=4843611232629293066}, [Second Message]
 *message [Second Message]*
 33531 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-8623931148894813393 4843611232629293066 3]
 33532 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-8623931148894813393 4843611232629293066 3]
 38532 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [Thrid Message]
 38536 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-7749553958395790620=-1739211867328620785}, [Thrid Message]
 *message [Thrid Message]*
 38537 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-7749553958395790620 -1739211867328620785 3]
 38537 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-7749553958395790620 -1739211867328620785 3]
 46201 [Thread-10-spout] INFO  backtype.storm.daemon.executor - Processing
 received message source: __system:-1, stream: __tick, id: {}, [30]
 76155 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: __system:-1, stream: __metrics_tick,
 id: {}, [60]
 76159 [Thread-8-printerbolt] INFO  backtype.storm.daemon.task - Emitting:
 printerbolt __metrics [#TaskInfo
 backtype.storm.metric.api.IMetricsConsumer$TaskInfo@780b0702
 [#DataPoint [__ack-count = {}] #DataPoint [__sendqueue = {write_pos=-1,
 read_pos=-1, capacity=1024, population=0}] #DataPoint [__receive =
 {write_pos=4, read_pos=3, capacity=1024, population=1}] #DataPoint
 [__process-latency = {}] #DataPoint [__transfer-count = {}] #DataPoint
 [__execute-latency = {}] #DataPoint [__fail-count = {}] #DataPoint
 [__emit-count = {}] #DataPoint [__execute-count = {}]]]
 76202 [Thread-10-spout] INFO  backtype.storm.daemon.executor - Processing
 received message source: __system:-1, stream: __tick, id: {}, [30]
 76206 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [First Message]
 76206 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {956790162864404846=494501721511970112}, [First Message]
 *message [First Message]*
 76207 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [956790162864404846 494501721511970112 3]
 76207 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream:

Re: Need help on kafka-storm

2014-12-01 Thread Harsha


Does your printer bolt ack the messages it received from KafkaSpout.


On Mon, Dec 1, 2014, at 06:38 PM, Madabhattula Rajesh Kumar wrote:
 Hello,

 Could any one help me on above mail query?

 Regards, Rajesh

 On Sat, Nov 29, 2014 at 10:30 PM, Madabhattula Rajesh Kumar
 mrajaf...@gmail.com wrote:
 Hello,

 I'm new to Storm and Kafka. I have tried Strom-Kafka integration
 example program. Now I'm able to send message from Kafka and receive
 those messages in storm topology.

 I have observed one thing in storm topology, same messages are
 processing continuously

 *I have sent three messages (First Message, Second Message, Third
 Message ). These 3 messages processing continuously, please find
 below console log file*

 *Could you please help me on below query*
  * *How to make sure that storm topology process messages one time
successfully(Not multiple times). *
  * *What configurations I need to do * *Below is my code :*

 * BrokerHosts zk = new ZkHosts(localhost:2181);* * SpoutConfig
   spoutConf = new SpoutConfig(zk, test-topic, /kafkastorm,
   discovery);* * spoutConf.scheme = new SchemeAsMultiScheme(new
   StringScheme());* * KafkaSpout spout = new KafkaSpout(spoutConf);*
   * TopologyBuilder builder = new TopologyBuilder();* *
   builder.setSpout(spout, spout, 1);* *
   builder.setBolt(printerbolt, new PrintBolt())
   .shuffleGrouping(spout);* * Config config = new Config();* *
   config.setDebug(true);* * LocalCluster cluster = new
   LocalCluster();* * cluster.submitTopology(kafka, config,
   builder.createTopology());*

 *Log file :*

 25526 [Thread-10-spout] INFO backtype.storm.daemon.task - Emitting:
 spout default [First Message] 25528 [Thread-8-printerbolt] INFO
 backtype.storm.daemon.executor - Processing received message source:
 spout:3, stream: default, id:
 {-5148901491748001310=-1334200518948214946}, [First Message] *message
 [First Message]* 25538 [Thread-10-spout] INFO
 backtype.storm.daemon.task - Emitting: spout __ack_init
 [-5148901491748001310 -1334200518948214946 3] 25539
 [Thread-14-__acker] INFO backtype.storm.daemon.executor - Processing
 received message source: spout:3, stream: __ack_init, id: {},
 [-5148901491748001310 -1334200518948214946 3] 33530 [Thread-10-spout]
 INFO backtype.storm.daemon.task - Emitting: spout default [Second
 Message] 33531 [Thread-8-printerbolt] INFO
 backtype.storm.daemon.executor - Processing received message source:
 spout:3, stream: default, id:
 {-8623931148894813393=4843611232629293066}, [Second Message] *message
 [Second Message]* 33531 [Thread-10-spout] INFO
 backtype.storm.daemon.task - Emitting: spout __ack_init
 [-8623931148894813393 4843611232629293066 3] 33532
 [Thread-14-__acker] INFO backtype.storm.daemon.executor - Processing
 received message source: spout:3, stream: __ack_init, id: {},
 [-8623931148894813393 4843611232629293066 3] 38532 [Thread-10-spout]
 INFO backtype.storm.daemon.task - Emitting: spout default [Thrid
 Message] 38536 [Thread-8-printerbolt] INFO
 backtype.storm.daemon.executor - Processing received message source:
 spout:3, stream: default, id:
 {-7749553958395790620=-1739211867328620785}, [Thrid Message] *message
 [Thrid Message]* 38537 [Thread-10-spout] INFO
 backtype.storm.daemon.task - Emitting: spout __ack_init
 [-7749553958395790620 -1739211867328620785 3] 38537
 [Thread-14-__acker] INFO backtype.storm.daemon.executor - Processing
 received message source: spout:3, stream: __ack_init, id: {},
 [-7749553958395790620 -1739211867328620785 3] 46201 [Thread-10-spout]
 INFO backtype.storm.daemon.executor - Processing received message
 source: __system:-1, stream: __tick, id: {}, [30] 76155
 [Thread-8-printerbolt] INFO backtype.storm.daemon.executor -
 Processing received message source: __system:-1, stream:
 __metrics_tick, id: {}, [60] 76159 [Thread-8-printerbolt] INFO
 backtype.storm.daemon.task - Emitting: printerbolt __metrics
 [#TaskInfo
 backtype.storm.metric.api.IMetricsConsumer$TaskInfo@780b0702
 [#DataPoint [__ack-count = {}] #DataPoint [__sendqueue =
 {write_pos=-1, read_pos=-1, capacity=1024, population=0}]
 #DataPoint [__receive = {write_pos=4, read_pos=3, capacity=1024,
 population=1}] #DataPoint [__process-latency = {}] #DataPoint
 [__transfer-count = {}] #DataPoint [__execute-latency = {}]
 #DataPoint [__fail-count = {}] #DataPoint [__emit-count = {}]
 #DataPoint [__execute-count = {}]]] 76202 [Thread-10-spout] INFO
 backtype.storm.daemon.executor - Processing received message source:
 __system:-1, stream: __tick, id: {}, [30] 76206 [Thread-10-spout]
 INFO backtype.storm.daemon.task - Emitting: spout default [First
 Message] 76206 [Thread-8-printerbolt] INFO
 backtype.storm.daemon.executor - Processing received message source:
 spout:3, stream: default, id:
 {956790162864404846=494501721511970112}, [First Message] *message
 [First Message]* 76207 [Thread-10-spout] INFO
 backtype.storm.daemon.task - Emitting: spout __ack_init
 [956790162864404846 494501721511970112 3] 76207 [Thread-14-__acker]

Re: Need help on kafka-storm

2014-12-01 Thread Madabhattula Rajesh Kumar

Thank you Harsha for your response.

I'm just printing the messages in printer bolt.

Please find below printer blot code












*public class PrintBolt extends BaseRichBolt   {private static final
long serialVersionUID = 1L;public void execute(Tuple tuple)
{ System.out.println(message  + tuple.getValues());}   public
void prepare(Map arg0, TopologyContext arg1, OutputCollector arg2) {   }
public void declareOutputFields(OutputFieldsDeclarer arg0) {  }}*

Regards,
Rajesh


On Tue, Dec 2, 2014 at 8:19 AM, Harsha st...@harsha.io wrote:

  Does your printer bolt ack the messages it received from KafkaSpout.


 On Mon, Dec 1, 2014, at 06:38 PM, Madabhattula Rajesh Kumar wrote:

 Hello,

 Could any one help me on above mail query?

 Regards,
 Rajesh

 On Sat, Nov 29, 2014 at 10:30 PM, Madabhattula Rajesh Kumar 
 mrajaf...@gmail.com wrote:

 Hello,

 I'm new to Storm and Kafka. I have tried Strom-Kafka integration example
 program. Now I'm able to send message from Kafka and receive those messages
 in storm topology.

  I have observed one thing in storm topology, same messages are
 processing continuously

  *I have sent three messages (First Message, Second Message, Third
 Message ). These 3 messages processing continuously, please find below
 console log file*

 *Could you please help me on below query*

- *How to make sure that storm topology process messages one time
successfully(Not multiple times). *
- *What configurations I need to do *

 *Below is my code :*

  * BrokerHosts zk = new ZkHosts(localhost:2181);*
 *SpoutConfig spoutConf = new SpoutConfig(zk, test-topic,
 /kafkastorm, discovery);*
 *spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());*
 *KafkaSpout spout = new KafkaSpout(spoutConf);*
 *TopologyBuilder builder = new TopologyBuilder();*
 *builder.setSpout(spout, spout, 1);*
 *builder.setBolt(printerbolt, new PrintBolt())
 .shuffleGrouping(spout);*
 *Config config = new Config();*
 *config.setDebug(true);*
 *LocalCluster cluster = new LocalCluster();*
 *cluster.submitTopology(kafka, config,
 builder.createTopology());*

  *Log file :*

 25526 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [First Message]
 25528 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-5148901491748001310=-1334200518948214946}, [First Message]
 *message [First Message]*
 25538 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-5148901491748001310 -1334200518948214946 3]
 25539 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-5148901491748001310 -1334200518948214946 3]
 33530 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [Second Message]
 33531 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-8623931148894813393=4843611232629293066}, [Second Message]
 *message [Second Message]*
 33531 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-8623931148894813393 4843611232629293066 3]
 33532 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-8623931148894813393 4843611232629293066 3]
 38532 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [Thrid Message]
 38536 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-7749553958395790620=-1739211867328620785}, [Thrid Message]
 *message [Thrid Message]*
 38537 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-7749553958395790620 -1739211867328620785 3]
 38537 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-7749553958395790620 -1739211867328620785 3]
 46201 [Thread-10-spout] INFO  backtype.storm.daemon.executor - Processing
 received message source: __system:-1, stream: __tick, id: {}, [30]
 76155 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: __system:-1, stream: __metrics_tick,
 id: {}, [60]
 76159 [Thread-8-printerbolt] INFO  backtype.storm.daemon.task - Emitting:
 printerbolt __metrics [#TaskInfo
 backtype.storm.metric.api.IMetricsConsumer$TaskInfo@780b0702
 [#DataPoint [__ack-count = {}] #DataPoint [__sendqueue = {write_pos=-1,
 read_pos=-1, capacity=1024, population=0}] #DataPoint [__receive =
 {write_pos=4, read_pos=3, capacity=1024, population=1}] #DataPoint
 [__process-latency = {}] #DataPoint [__transfer-count = {}] #DataPoint
 [__execute-latency = {}] #DataPoint [__fail-count = {}] #DataPoint
 [__emit-count = {}]

Re: Need help on kafka-storm

2014-12-01 Thread Madabhattula Rajesh Kumar

Thank you very much Harsha

Regards,
Rajesh

On Tue, Dec 2, 2014 at 8:50 AM, Harsha st...@harsha.io wrote:

  Ok from the earlier logs it looks like your tuples are being timed out
 and getting replayed.
 In your PrintBolt.execute do collector.ack(tuple)
 public class PrintBolt extends BaseRichBolt
 {
 private static final long serialVersionUID = 1L;
 private OutputCollector collector;

 public void execute(Tuple tuple)
 {
  System.out.println(message  + tuple.getValues());
  collector.ack(tuple);
 }
public void prepare(Map arg0, TopologyContext arg1, OutputCollector
 arg2) {
 this.collector = arg2;
}
   public void declareOutputFields(OutputFieldsDeclarer arg0) {
   }
 }


 On Mon, Dec 1, 2014, at 07:10 PM, Madabhattula Rajesh Kumar wrote:

 Thank you Harsha for your response.

 I'm just printing the messages in printer bolt.

 Please find below printer blot code

 *public class PrintBolt extends BaseRichBolt   *
 *{*
 *private static final long serialVersionUID = 1L;*
 *public void execute(Tuple tuple) *
 *{*
 * System.out.println(message  + tuple.getValues());*
 *}*
 *   public void prepare(Map arg0, TopologyContext arg1, OutputCollector
 arg2) {*
 *   }*
 *  public void declareOutputFields(OutputFieldsDeclarer arg0) {*
 *  }*
 *}*

  Regards,
 Rajesh


 On Tue, Dec 2, 2014 at 8:19 AM, Harsha st...@harsha.io wrote:


 Does your printer bolt ack the messages it received from KafkaSpout.


 On Mon, Dec 1, 2014, at 06:38 PM, Madabhattula Rajesh Kumar wrote:

 Hello,

 Could any one help me on above mail query?

 Regards,
 Rajesh

 On Sat, Nov 29, 2014 at 10:30 PM, Madabhattula Rajesh Kumar 
 mrajaf...@gmail.com wrote:

 Hello,

 I'm new to Storm and Kafka. I have tried Strom-Kafka integration example
 program. Now I'm able to send message from Kafka and receive those messages
 in storm topology.

  I have observed one thing in storm topology, same messages are
 processing continuously

  *I have sent three messages (First Message, Second Message, Third
 Message ). These 3 messages processing continuously, please find below
 console log file*

 *Could you please help me on below query*

- *How to make sure that storm topology process messages one time
successfully(Not multiple times). *
- *What configurations I need to do *

 *Below is my code :*

 * BrokerHosts zk = new ZkHosts(localhost:2181);*
 *SpoutConfig spoutConf = new SpoutConfig(zk, test-topic,
 /kafkastorm, discovery);*
 *spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());*
 *KafkaSpout spout = new KafkaSpout(spoutConf);*
 *TopologyBuilder builder = new TopologyBuilder();*
 *builder.setSpout(spout, spout, 1);*
 *builder.setBolt(printerbolt, new PrintBolt())
 .shuffleGrouping(spout);*
 *Config config = new Config();*
 *config.setDebug(true);*
 *LocalCluster cluster = new LocalCluster();*
 *cluster.submitTopology(kafka, config,
 builder.createTopology());*

  *Log file :*

 25526 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [First Message]
 25528 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-5148901491748001310=-1334200518948214946}, [First Message]
 *message [First Message]*
 25538 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-5148901491748001310 -1334200518948214946 3]
 25539 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-5148901491748001310 -1334200518948214946 3]
 33530 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [Second Message]
 33531 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-8623931148894813393=4843611232629293066}, [Second Message]
 *message [Second Message]*
 33531 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-8623931148894813393 4843611232629293066 3]
 33532 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-8623931148894813393 4843611232629293066 3]
 38532 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 default [Thrid Message]
 38536 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: default, id:
 {-7749553958395790620=-1739211867328620785}, [Thrid Message]
 *message [Thrid Message]*
 38537 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
 __ack_init [-7749553958395790620 -1739211867328620785 3]
 38537 [Thread-14-__acker] INFO  backtype.storm.daemon.executor -
 Processing received message source: spout:3, stream: __ack_init, id: {},
 [-7749553958395790620 -1739211867328620785 3]

Need help on kafka-storm

2014-11-29 Thread Madabhattula Rajesh Kumar

Hello,

I'm new to Storm and Kafka. I have tried Strom-Kafka integration example
program. Now I'm able to send message from Kafka and receive those messages
in storm topology.

I have observed one thing in storm topology, same messages are processing
continuously

*I have sent three messages (First Message, Second Message, Third Message
). These 3 messages processing continuously, please find below console log
file*


*Could you please help me on below query*

   -
*How to make sure that storm topology process messages one time
   successfully(Not multiple times). *
   - *What configurations I need to do *

*Below is my code :*











* BrokerHosts zk = new ZkHosts(localhost:2181);SpoutConfig
spoutConf = new SpoutConfig(zk, test-topic, /kafkastorm,
discovery);spoutConf.scheme = new SchemeAsMultiScheme(new
StringScheme());KafkaSpout spout = new KafkaSpout(spoutConf);
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(spout, spout, 1);builder.setBolt(printerbolt,
new PrintBolt()) .shuffleGrouping(spout);Config config = new
Config();config.setDebug(true);LocalCluster cluster = new
LocalCluster();cluster.submitTopology(kafka, config,
builder.createTopology());*

*Log file :*

25526 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
default [First Message]
25528 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
Processing received message source: spout:3, stream: default, id:
{-5148901491748001310=-1334200518948214946}, [First Message]
*message [First Message]*
25538 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
__ack_init [-5148901491748001310 -1334200518948214946 3]
25539 [Thread-14-__acker] INFO  backtype.storm.daemon.executor - Processing
received message source: spout:3, stream: __ack_init, id: {},
[-5148901491748001310 -1334200518948214946 3]
33530 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
default [Second Message]
33531 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
Processing received message source: spout:3, stream: default, id:
{-8623931148894813393=4843611232629293066}, [Second Message]
*message [Second Message]*
33531 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
__ack_init [-8623931148894813393 4843611232629293066 3]
33532 [Thread-14-__acker] INFO  backtype.storm.daemon.executor - Processing
received message source: spout:3, stream: __ack_init, id: {},
[-8623931148894813393 4843611232629293066 3]
38532 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
default [Thrid Message]
38536 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
Processing received message source: spout:3, stream: default, id:
{-7749553958395790620=-1739211867328620785}, [Thrid Message]
*message [Thrid Message]*
38537 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
__ack_init [-7749553958395790620 -1739211867328620785 3]
38537 [Thread-14-__acker] INFO  backtype.storm.daemon.executor - Processing
received message source: spout:3, stream: __ack_init, id: {},
[-7749553958395790620 -1739211867328620785 3]
46201 [Thread-10-spout] INFO  backtype.storm.daemon.executor - Processing
received message source: __system:-1, stream: __tick, id: {}, [30]
76155 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
Processing received message source: __system:-1, stream: __metrics_tick,
id: {}, [60]
76159 [Thread-8-printerbolt] INFO  backtype.storm.daemon.task - Emitting:
printerbolt __metrics [#TaskInfo
backtype.storm.metric.api.IMetricsConsumer$TaskInfo@780b0702 [#DataPoint
[__ack-count = {}] #DataPoint [__sendqueue = {write_pos=-1, read_pos=-1,
capacity=1024, population=0}] #DataPoint [__receive = {write_pos=4,
read_pos=3, capacity=1024, population=1}] #DataPoint [__process-latency =
{}] #DataPoint [__transfer-count = {}] #DataPoint [__execute-latency =
{}] #DataPoint [__fail-count = {}] #DataPoint [__emit-count = {}]
#DataPoint [__execute-count = {}]]]
76202 [Thread-10-spout] INFO  backtype.storm.daemon.executor - Processing
received message source: __system:-1, stream: __tick, id: {}, [30]
76206 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
default [First Message]
76206 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
Processing received message source: spout:3, stream: default, id:
{956790162864404846=494501721511970112}, [First Message]
*message [First Message]*
76207 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
__ack_init [956790162864404846 494501721511970112 3]
76207 [Thread-14-__acker] INFO  backtype.storm.daemon.executor - Processing
received message source: spout:3, stream: __ack_init, id: {},
[956790162864404846 494501721511970112 3]
76207 [Thread-10-spout] INFO  backtype.storm.daemon.task - Emitting: spout
default [Second Message]
76208 [Thread-8-printerbolt] INFO  backtype.storm.daemon.executor -
Processing

48 matches

Mail list logo