Re: Storm not processing topology without logs
I am getting following error when trying to run the command for worker
directly on console
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread main-SendThread(hdp.ambari:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-2
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-12-bolt1
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-10-bolt2
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-8-bolt3
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-14-spout
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(localhost:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(localhost:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(hdp.ambari:2181)
As one of the possible bug situations, I looked for multiple netty jars as
suggested in other mail thread, it didn't work. Can anyone help me out
where should I look next to resolve the issue.
On Tue, Aug 26, 2014 at 2:20 PM, Vikas Agarwal vi...@infoobjects.com
wrote:
However, now my topology is failing to start worker process again. :(
This time is not showing me any good clue to resolve it. Running the
command manually on console causes Address already in use error for
supervisor ports (6700,6701). So, it is not letting me move forward to see
what actually the error is while running the worker.
On Mon, Aug 25, 2014 at 9:00 PM, Vikas Agarwal vi...@infoobjects.com
wrote:
Yes, I was able to see the topology in Storm UI and nothing was logged
into worker logs. However, as I mentioned, I am able to resolve it by
finding an hint in supervisor.log file this time.
On Mon, Aug 25, 2014 at 8:58 PM, Georgy Abraham itsmegeo...@gmail.com
wrote:
Are you able to see the topology in storm UI or with storm list command
?? And worker mentioned in the UI doesn't have any log ??
--
From: Vikas Agarwal
Sent: 25-08-2014 PM 05:25
To: user@storm.incubator.apache.org
Subject: Storm not processing topology without logs
Hi,
I have started to explore the Storm for distributed processing for our
use case which we were earlier fulfilling by JMS based MQ system. Topology
worked after some efforts. It has one spout (KafkaSpout from kafka-storm
project) and 3 bolts. First bolt sets context for other two bolts which in
turn do some processing on the tuples and persist the analyzed results in
some DB (Mongo, Solr, HBase etc).
Recently the topology stopped working. I am able to submit the topology
and it does not throw any error in submitting the topology, however,
nimbus.log or worker-6701.log files are not showing any progress and
eventually topology does not consume any message. I don't have doubt on
KafkaSpout because if it was the culprit, at least some initialization logs
of spout and bolts should have been there in nimbus.log or worker-.log.
Isn't it?
Here is the snippet of nimbus.log after uploading the jar to cluster
Uploading file from client to
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file from
client:
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Received topology submission for
aleads with conf {topology.max.task.parallelism nil,
topology.acker.executors nil, topology.kryo.register nil,
topology.kryo.decorators (), topology.name aleads, storm.id
aleads-3-1408964869, modelId ut, topology.workers 1,
topology.debug true}
2014-08-25 07:07:50 b.s.d.nimbus [INFO] Activating aleads:
aleads-3-1408964869
2014-08-25 07:07:50 b.s.s.EvenScheduler [INFO] Available slots:
([e56c2cc7-d35a-4355-9906-506618ff70c5 6701]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6700])
2014-08-25 07:07:50 b.s.d.nimbus [INFO] Setting new assignment for
topology id aleads-3-1408964869:
#backtype.storm.daemon.common.Assignment{:master-code-dir
/hadoop/storm/nimbus/stormdist/aleads-3-1408964869, :node-host
{e56c2cc7-d35a-4355-9906-506618ff70c5 hdp.ambari}, :executor-node+port
{[2 2] [e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [3 3]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [4 4]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [5 5]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [6 6]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [7 7]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [8 8]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [9 9]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [1 1]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701]},
Re: Storm not processing topology without logs
Vikas,
Are you able to get past this error Running the
command manually on console causes Address already in use
error for supervisor ports (6700,6701). Did you check if there
are any processes running on that port.
-Harsha
On Thu, Aug 28, 2014, at 01:58 AM, Vikas Agarwal wrote:
I am getting following error when trying to run the command for
worker directly on console
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
main-SendThread(hdp.ambari:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-2
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-12-bolt1
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-10-bolt2
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-8-bolt3
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-14-spout
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(localhost:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(localhost:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(hdp.ambari:2181)
As one of the possible bug situations, I looked for multiple
netty jars as suggested in other mail thread, it didn't work.
Can anyone help me out where should I look next to resolve the
issue.
On Tue, Aug 26, 2014 at 2:20 PM, Vikas Agarwal
[1]vi...@infoobjects.com wrote:
However, now my topology is failing to start worker process
again. :(
This time is not showing me any good clue to resolve it.
Running the command manually on console causes Address already
in use error for supervisor ports (6700,6701). So, it is not
letting me move forward to see what actually the error is while
running the worker.
On Mon, Aug 25, 2014 at 9:00 PM, Vikas Agarwal
[2]vi...@infoobjects.com wrote:
Yes, I was able to see the topology in Storm UI and nothing was
logged into worker logs. However, as I mentioned, I am able to
resolve it by finding an hint in supervisor.log file this time.
On Mon, Aug 25, 2014 at 8:58 PM, Georgy Abraham
[3]itsmegeo...@gmail.com wrote:
Are you able to see the topology in storm UI or with storm list
command ?? And worker mentioned in the UI doesn't have any log
??
__
From: Vikas Agarwal
Sent: 25-08-2014 PM 05:25
To: [4]user@storm.incubator.apache.org
Subject: Storm not processing topology without logs
Hi,
I have started to explore the Storm for distributed processing
for our use case which we were earlier fulfilling by JMS based
MQ system. Topology worked after some efforts. It has one spout
(KafkaSpout from kafka-storm project) and 3 bolts. First bolt
sets context for other two bolts which in turn do some
processing on the tuples and persist the analyzed results in
some DB (Mongo, Solr, HBase etc).
Recently the topology stopped working. I am able to submit the
topology and it does not throw any error in submitting the
topology, however, nimbus.log or worker-6701.log files are not
showing any progress and eventually topology does not consume
any message. I don't have doubt on KafkaSpout because if it was
the culprit, at least some initialization logs of spout and
bolts should have been there in nimbus.log or worker-.log.
Isn't it?
Here is the snippet of nimbus.log after uploading the jar to
cluster
Uploading file from client to
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe1
3c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file
from client:
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe1
3c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Received topology
submission for aleads with conf
{topology.max.task.parallelism nil,
topology.acker.executors nil, topology.kryo.register nil,
topology.kryo.decorators (), [5]topology.name aleads,
[6]storm.id aleads-3-1408964869, modelId ut,
topology.workers 1, topology.debug true}
2014-08-25 07:07:50 b.s.d.nimbus [INFO] Activating aleads:
aleads-3-1408964869
2014-08-25 07:07:50 b.s.s.EvenScheduler [INFO] Available slots:
([e56c2cc7-d35a-4355-9906-506618ff70c5 6701]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6700])
2014-08-25 07:07:50 b.s.d.nimbus [INFO] Setting new assignment
for topology id aleads-3-1408964869:
#backtype.storm.daemon.common.Assignment{:master-code-dir
/hadoop/storm/nimbus/stormdist/aleads-3-1408964869,
:node-host {e56c2cc7-d35a-4355-9906-506618ff70c5
hdp.ambari}, :executor-node+port {[2 2]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [3 3]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [4 4]
[e56c2cc7-d35a-4355-9906-506618ff70c5 6701], [5 5]
Re: Storm not processing topology without logs
Yes, I am through it. I have killed the processes created by main
supervisor processes for 6700 and 6701 ports and then started process for
one of these ports.
After that I faced issues due to multiple versions of same library in storm
lib e.g. netty and servlet-api
After that I faced this stack overflow issue. Now, I am even able to fix
it. Multiple slf4j-log4j implementations was the issue behind stack
overflow.
Now, I am back to the same state where the process just don't start. Now
running the worker command manually is even not showing any log except this:
JMXetricAgent instrumented JVM, see https://github.com/ganglia/jmxetric
Aug 28, 2014 10:28:39 AM info.ganglia.gmetric4j.GMonitor start
INFO: Setting up 1 samplers
And then process get killed.
On Thu, Aug 28, 2014 at 7:22 PM, Harsha st...@harsha.io wrote:
Vikas,
Are you able to get past this error Running the command manually
on console causes Address already in use error for supervisor ports
(6700,6701). Did you check if there are any processes running on that
port.
-Harsha
On Thu, Aug 28, 2014, at 01:58 AM, Vikas Agarwal wrote:
I am getting following error when trying to run the command for worker
directly on console
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread main-SendThread(hdp.ambari:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-2
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-12-bolt1
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-10-bolt2
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-8-bolt3
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread Thread-14-spout
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(localhost:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(localhost:2181)
Exception: java.lang.StackOverflowError thrown from the
UncaughtExceptionHandler in thread
Thread-14-feed-stream-SendThread(hdp.ambari:2181)
As one of the possible bug situations, I looked for multiple netty jars as
suggested in other mail thread, it didn't work. Can anyone help me out
where should I look next to resolve the issue.
On Tue, Aug 26, 2014 at 2:20 PM, Vikas Agarwal vi...@infoobjects.com
wrote:
However, now my topology is failing to start worker process again. :(
This time is not showing me any good clue to resolve it. Running the
command manually on console causes Address already in use error for
supervisor ports (6700,6701). So, it is not letting me move forward to see
what actually the error is while running the worker.
On Mon, Aug 25, 2014 at 9:00 PM, Vikas Agarwal vi...@infoobjects.com
wrote:
Yes, I was able to see the topology in Storm UI and nothing was logged
into worker logs. However, as I mentioned, I am able to resolve it by
finding an hint in supervisor.log file this time.
On Mon, Aug 25, 2014 at 8:58 PM, Georgy Abraham itsmegeo...@gmail.com
wrote:
Are you able to see the topology in storm UI or with storm list command ??
And worker mentioned in the UI doesn't have any log ??
--
*From: *Vikas Agarwal
*Sent: *25-08-2014 PM 05:25
*To: *user@storm.incubator.apache.org
*Subject: *Storm not processing topology without logs
Hi,
I have started to explore the Storm for distributed processing for our use
case which we were earlier fulfilling by JMS based MQ system. Topology
worked after some efforts. It has one spout (KafkaSpout from kafka-storm
project) and 3 bolts. First bolt sets context for other two bolts which in
turn do some processing on the tuples and persist the analyzed results in
some DB (Mongo, Solr, HBase etc).
Recently the topology stopped working. I am able to submit the topology
and it does not throw any error in submitting the topology, however,
nimbus.log or worker-6701.log files are not showing any progress and
eventually topology does not consume any message. I don't have doubt on
KafkaSpout because if it was the culprit, at least some initialization logs
of spout and bolts should have been there in nimbus.log or worker-.log.
Isn't it?
Here is the snippet of nimbus.log after uploading the jar to cluster
Uploading file from client to
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file from
client:
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Received topology submission for
aleads with conf {topology.max.task.parallelism nil,
topology.acker.executors nil,
Re: Storm not processing topology without logs
If possible can you post some logs from supervisor.log. Interested in looking at the log when your supervisor starts. -Harsha On Thu, Aug 28, 2014, at 07:29 AM, Vikas Agarwal wrote: Yes, I am through it. I have killed the processes created by main supervisor processes for 6700 and 6701 ports and then started process for one of these ports. After that I faced issues due to multiple versions of same library in storm lib e.g. netty and servlet-api After that I faced this stack overflow issue. Now, I am even able to fix it. Multiple slf4j-log4j implementations was the issue behind stack overflow. Now, I am back to the same state where the process just don't start. Now running the worker command manually is even not showing any log except this: JMXetricAgent instrumented JVM, see [1]https://github.com/ganglia/jmxetric Aug 28, 2014 10:28:39 AM info.ganglia.gmetric4j.GMonitor start INFO: Setting up 1 samplers And then process get killed. On Thu, Aug 28, 2014 at 7:22 PM, Harsha [2]st...@harsha.io wrote: Vikas, Are you able to get past this error Running the command manually on console causes Address already in use error for supervisor ports (6700,6701). Did you check if there are any processes running on that port. -Harsha On Thu, Aug 28, 2014, at 01:58 AM, Vikas Agarwal wrote: I am getting following error when trying to run the command for worker directly on console Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread main-SendThread(hdp.ambari:2181) Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-2 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-12-bolt1 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-10-bolt2 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-8-bolt3 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-spout Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-feed-stream-SendThread(localhost:2181) Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-feed-stream-SendThread(localhost:2181) Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-feed-stream-SendThread(hdp.ambari:2181) As one of the possible bug situations, I looked for multiple netty jars as suggested in other mail thread, it didn't work. Can anyone help me out where should I look next to resolve the issue. On Tue, Aug 26, 2014 at 2:20 PM, Vikas Agarwal [3]vi...@infoobjects.com wrote: However, now my topology is failing to start worker process again. :( This time is not showing me any good clue to resolve it. Running the command manually on console causes Address already in use error for supervisor ports (6700,6701). So, it is not letting me move forward to see what actually the error is while running the worker. On Mon, Aug 25, 2014 at 9:00 PM, Vikas Agarwal [4]vi...@infoobjects.com wrote: Yes, I was able to see the topology in Storm UI and nothing was logged into worker logs. However, as I mentioned, I am able to resolve it by finding an hint in supervisor.log file this time. On Mon, Aug 25, 2014 at 8:58 PM, Georgy Abraham [5]itsmegeo...@gmail.com wrote: Are you able to see the topology in storm UI or with storm list command ?? And worker mentioned in the UI doesn't have any log ?? __ From: Vikas Agarwal Sent: 25-08-2014 PM 05:25 To: [6]user@storm.incubator.apache.org Subject: Storm not processing topology without logs Hi, I have started to explore the Storm for distributed processing for our use case which we were earlier fulfilling by JMS based MQ system. Topology worked after some efforts. It has one spout (KafkaSpout from kafka-storm project) and 3 bolts. First bolt sets context for other two bolts which in turn do some processing on the tuples and persist the analyzed results in some DB (Mongo, Solr, HBase etc). Recently the topology stopped working. I am able to submit the topology and it does not throw any error in submitting the topology, however, nimbus.log or worker-6701.log files are not showing any progress and eventually topology does not consume any message. I don't have doubt on KafkaSpout because if it was the culprit, at least some initialization logs of spout and bolts should have been there in nimbus.log or worker-.log. Isn't it? Here is the snippet of nimbus.log after uploading the jar to cluster Uploading file from client to /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe1 3c706b2ab.jar 2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file from client: /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe1 3c706b2ab.jar 2014-08-25
Re: Storm not processing topology without logs
JMXetricAgent instrumented JVM, see https://github.com/ganglia/jmxetric Aug 28, 2014 10:28:39 AM info.ganglia.gmetric4j.GMonitor start INFO: Setting up 1 samplers This is the only log now when I start it manually and supervisor is still saying the same still hasn't started nothing more than that. It seems to me that it is some issue of inconsistent state because of the past errors. So, I am restarting the machine it self to check if it works after that. On Thu, Aug 28, 2014 at 8:27 PM, Harsha st...@harsha.io wrote: If possible can you post some logs from supervisor.log. Interested in looking at the log when your supervisor starts. -Harsha On Thu, Aug 28, 2014, at 07:29 AM, Vikas Agarwal wrote: Yes, I am through it. I have killed the processes created by main supervisor processes for 6700 and 6701 ports and then started process for one of these ports. After that I faced issues due to multiple versions of same library in storm lib e.g. netty and servlet-api After that I faced this stack overflow issue. Now, I am even able to fix it. Multiple slf4j-log4j implementations was the issue behind stack overflow. Now, I am back to the same state where the process just don't start. Now running the worker command manually is even not showing any log except this: JMXetricAgent instrumented JVM, see https://github.com/ganglia/jmxetric Aug 28, 2014 10:28:39 AM info.ganglia.gmetric4j.GMonitor start INFO: Setting up 1 samplers And then process get killed. On Thu, Aug 28, 2014 at 7:22 PM, Harsha st...@harsha.io wrote: Vikas, Are you able to get past this error Running the command manually on console causes Address already in use error for supervisor ports (6700,6701). Did you check if there are any processes running on that port. -Harsha On Thu, Aug 28, 2014, at 01:58 AM, Vikas Agarwal wrote: I am getting following error when trying to run the command for worker directly on console Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread main-SendThread(hdp.ambari:2181) Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-2 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-12-bolt1 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-10-bolt2 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-8-bolt3 Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-spout Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-feed-stream-SendThread(localhost:2181) Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-feed-stream-SendThread(localhost:2181) Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread Thread-14-feed-stream-SendThread(hdp.ambari:2181) As one of the possible bug situations, I looked for multiple netty jars as suggested in other mail thread, it didn't work. Can anyone help me out where should I look next to resolve the issue. On Tue, Aug 26, 2014 at 2:20 PM, Vikas Agarwal vi...@infoobjects.com wrote: However, now my topology is failing to start worker process again. :( This time is not showing me any good clue to resolve it. Running the command manually on console causes Address already in use error for supervisor ports (6700,6701). So, it is not letting me move forward to see what actually the error is while running the worker. On Mon, Aug 25, 2014 at 9:00 PM, Vikas Agarwal vi...@infoobjects.com wrote: Yes, I was able to see the topology in Storm UI and nothing was logged into worker logs. However, as I mentioned, I am able to resolve it by finding an hint in supervisor.log file this time. On Mon, Aug 25, 2014 at 8:58 PM, Georgy Abraham itsmegeo...@gmail.com wrote: Are you able to see the topology in storm UI or with storm list command ?? And worker mentioned in the UI doesn't have any log ?? -- *From: *Vikas Agarwal *Sent: *25-08-2014 PM 05:25 *To: *user@storm.incubator.apache.org *Subject: *Storm not processing topology without logs Hi, I have started to explore the Storm for distributed processing for our use case which we were earlier fulfilling by JMS based MQ system. Topology worked after some efforts. It has one spout (KafkaSpout from kafka-storm project) and 3 bolts. First bolt sets context for other two bolts which in turn do some processing on the tuples and persist the analyzed results in some DB (Mongo, Solr, HBase etc). Recently the topology stopped working. I am able to submit the topology and it does not throw any error in submitting the topology, however, nimbus.log or worker-6701.log files are not showing any progress and eventually
Re: adding bolt to TridentTopology
You can not use storm core bolts to process trident tuples. You could use core spouts in trident topologies but not bolts. You will have to write your trident states that may do identical stuff as your bolt implementation in batch mode. You can refer to this doc to understand how you can write your own trident state https://storm.incubator.apache.org/documentation/Trident-tutorial.html Hope this helps. Thanks Parth On Aug 27, 2014, at 10:05 PM, Naga Vij nvbuc...@gmail.com wrote: Hello, How can I add a bolt to TridentTopology? I realize the use of function on the stream (kafka stream in my case) ... parsedStream.each(parsedStream.getOutputFields(), new SomeFunction(), new Fields()); But I want to do subsequent processing by using bolts after the function. How can I do that? Tried looking into the API, but appears I need some help from others who might have tried it already. Thanks in advance. Naga -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[DISCUSS] Apache Storm Release 0.9.3/0.10.0
I’d like to gather community feedback for the next two releases of Apache Storm. 0.9.3-incubating will be our next release. Please indicate (by JIRA ticket ID) which bug fixes and/or new features you would like to be considered for inclusion in the next release. If there is not an existing for a particular issue or feature, please consider adding one. For the next and subsequent releases, we will be using a slightly different approach than what we did in the past. Instead of voting right away on a build, we will make one or more “unofficial” release candidate builds available prior to voting on an official release. This will give the Apache Storm community more time to discuss, evaluate, identify and fix potential issues before the official release. This should enable us to ensure the final release is as bug free as possible. Apache Storm 0.10.0 (STORM-216) As some of you are aware, the engineering team at Yahoo! has done a lot of work to bring security and multi-tenancy to Storm, and has contributed that work back to the community. Over the past few months we have been in the process of enhancing and syncing that work with the master branch in a separate branch labeled “security.” That work is now nearing completion, and I would like us to consider merging it into master after the 0.9.3 release. Since the security work includes a large number of changes and enhancements, I propose we bump the version number to 0.10.0 for the first release to include those features. More information about the security branch can be found in this pull request [1], as well as the SECURITY.md file in the security branch [2]. I also discussed it in a blog post [3] on the Hortonworks website. Please feel free to direct any comments or questions about the security branch to the mailing list. Similar to the process we’ll follow for 0.9.3, we plan to make several unofficial “development” builds available for those who would like to help with testing the new security features. -Taylor [1] https://github.com/apache/incubator-storm/pull/121 [2] https://github.com/apache/incubator-storm/blob/security/SECURITY.md [3] http://hortonworks.com/blog/the-future-of-apache-storm/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Reading config.priperties file
Thanks Anand, I finally get to play around with this again. I used the backtype.storm.Config class to pass arounds my properties. static variables in clustered environment does not work as it does in a single VM environment. Thanks to you both once again. -- Kushan Maskey 817.403.7500 On Thu, Aug 21, 2014 at 4:48 AM, Georgy Abraham itsmegeo...@gmail.com wrote: If you want the config file not to be packed with the code/jar and need to be in a location outside , the storm doesn't have a filesystem associated with it unlike hadoop which has its own distributed file system . If your storm cluster doubles as a hadoop cluster too which I have seen in many cases , I can use hdfs location to store the property file. -- From: Anand Nalya Sent: 21-08-2014 PM 12:00 To: user@storm.incubator.apache.org Subject: Re: Reading config.priperties file Hi Kushan, A slight variation on second approach suggested by Parth is to read the properties file in your driver class on the gateway and copy all the properties to the backtype.storm.Config object. In this way, the properties will be available in the prepare/open method of your bolt/spout. Regards, Anand On 21 August 2014 03:51, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Are you packing the config file in the jar? Does the config file get loaded on the gateway , where you run storm command, or its suppose to be loaded as part of spout’s/bolt's prepare method? In the former case you need to ensure that your property file is part of your jar file. If you are using maven you can do so by adding the following to your build target: resources resource directorysrc/resource/directory /resource /resources and then in your code you can get a handle on the config file by SomeClass.class.getClassLoader().getResourceAsStream(“yourconfig.properties) The other way would be to just read the config file at the gateway, read the properties and set the property value as instance variables in the appropriate bolt and spout object. Ensure that the instance variables are not marked as transient. Thanks Parth If the config file is read and loaded at the gateway then are you storing On Aug 20, 2014, at 2:50 PM, Kushan Maskey kushan.mas...@mmillerassociates.com wrote: I pass the config file as an argument to the Topology. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [DISCUSS] Apache Storm Release 0.9.3/0.10.0
I am supportive. I think it makes sense to move to 0.10.0 because of the significance of the changes. -- Derek On 8/28/14, 15:34, P.Taylor Goetz wrote: I’d like to gather community feedback for the next two releases of Apache Storm. 0.9.3-incubating will be our next release. Please indicate (by JIRA ticket ID) which bug fixes and/or new features you would like to be considered for inclusion in the next release. If there is not an existing for a particular issue or feature, please consider adding one. For the next and subsequent releases, we will be using a slightly different approach than what we did in the past. Instead of voting right away on a build, we will make one or more “unofficial” release candidate builds available prior to voting on an official release. This will give the Apache Storm community more time to discuss, evaluate, identify and fix potential issues before the official release. This should enable us to ensure the final release is as bug free as possible. Apache Storm 0.10.0 (STORM-216) As some of you are aware, the engineering team at Yahoo! has done a lot of work to bring security and multi-tenancy to Storm, and has contributed that work back to the community. Over the past few months we have been in the process of enhancing and syncing that work with the master branch in a separate branch labeled “security.” That work is now nearing completion, and I would like us to consider merging it into master after the 0.9.3 release. Since the security work includes a large number of changes and enhancements, I propose we bump the version number to 0.10.0 for the first release to include those features. More information about the security branch can be found in this pull request [1], as well as the SECURITY.md file in the security branch [2]. I also discussed it in a blog post [3] on the Hortonworks website. Please feel free to direct any comments or questions about the security branch to the mailing list. Similar to the process we’ll follow for 0.9.3, we plan to make several unofficial “development” builds available for those who would like to help with testing the new security features. -Taylor [1] https://github.com/apache/incubator-storm/pull/121 [2] https://github.com/apache/incubator-storm/blob/security/SECURITY.md [3] http://hortonworks.com/blog/the-future-of-apache-storm/
Re: [DISCUSS] Apache Storm Release 0.9.3/0.10.0
I agree with the version bump and also with the strategy to have a beta release. Thanks Parth — Sent from Mailbox On Thu, Aug 28, 2014 at 2:59 PM, Derek Dagit der...@yahoo-inc.com wrote: I am supportive. I think it makes sense to move to 0.10.0 because of the significance of the changes. -- Derek On 8/28/14, 15:34, P.Taylor Goetz wrote: I’d like to gather community feedback for the next two releases of Apache Storm. 0.9.3-incubating will be our next release. Please indicate (by JIRA ticket ID) which bug fixes and/or new features you would like to be considered for inclusion in the next release. If there is not an existing for a particular issue or feature, please consider adding one. For the next and subsequent releases, we will be using a slightly different approach than what we did in the past. Instead of voting right away on a build, we will make one or more “unofficial” release candidate builds available prior to voting on an official release. This will give the Apache Storm community more time to discuss, evaluate, identify and fix potential issues before the official release. This should enable us to ensure the final release is as bug free as possible. Apache Storm 0.10.0 (STORM-216) As some of you are aware, the engineering team at Yahoo! has done a lot of work to bring security and multi-tenancy to Storm, and has contributed that work back to the community. Over the past few months we have been in the process of enhancing and syncing that work with the master branch in a separate branch labeled “security.” That work is now nearing completion, and I would like us to consider merging it into master after the 0.9.3 release. Since the security work includes a large number of changes and enhancements, I propose we bump the version number to 0.10.0 for the first release to include those features. More information about the security branch can be found in this pull request [1], as well as the SECURITY.md file in the security branch [2]. I also discussed it in a blog post [3] on the Hortonworks website. Please feel free to direct any comments or questions about the security branch to the mailing list. Similar to the process we’ll follow for 0.9.3, we plan to make several unofficial “development” builds available for those who would like to help with testing the new security features. -Taylor [1] https://github.com/apache/incubator-storm/pull/121 [2] https://github.com/apache/incubator-storm/blob/security/SECURITY.md [3] http://hortonworks.com/blog/the-future-of-apache-storm/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
