Trident Metrics Consumer

I've been following the tutorials here ( http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to) to create metrics in Storm. However I am using Trident which abstracts bolts away from the user. How can I go about creating metrics in trident ? Thanks -- Raphael Hsieh

RE: nette reconnects

We see exactly the same thing in our worker logs. I don't know if this is "correct" behavior, but just acknowledging that we see the same thing. Richard Gunderson Mobile: (612) 860-1676 -Original Message- From: Tyson Norris [mailto:tnor...@adobe.com] Sent: Friday, September 26, 2014 1:0

Re: secure storm UI

This is available in the security branch. See https://github.com/apache/storm/blob/security/SECURITY.md You do not need to enable all of the security features to get UI auth. For authentication, look at ui.filter and ui.filter.params. For authorization, nimbus.admins, ui.users, logs.users, an

secure storm UI

Is there a way to secure the storm UI page. Like enable logging to access the page so only authorized people can only access it. -- Kushan Maskey 817.403.7500 M. Miller & Associates kushan.mas...@mmillerassociates.com

Re: nette reconnects

@varun - I still see workers waiting, reconnecting, closing connections, and dying, when using a longer max_wait_ms and shorter worker.start timeout @derek - based on that bug, I will try to see if using a single worker per node (currently 4 workers per node) makes a difference. Thanks Tyson

Re: nette reconnects

This could be https://issues.apache.org/jira/browse/STORM-510 The send thread is blocked on a connection attempt, and so no messages get sent out until the connection is re-established or it times out. -- Derek On 9/26/14 13:47, Varun Vijayaraghavan wrote: I first tried increasing the max_re

Re: nette reconnects

I first tried increasing the max_retries to a much higher number (300) but that did not make a difference. On Fri, Sep 26, 2014 at 2:46 PM, Varun Vijayaraghavan wrote: > Hey, > > I've been facing the same issues in my topologies. It seems like a crash > in a single worker would trigger a reconne

Re: nette reconnects

Hey, I've been facing the same issues in my topologies. It seems like a crash in a single worker would trigger a reconnect from other workers for x amount of time (30 x 10s = ~300 seconds in your case) before crashing themselves - thus leading to a catastrophic failure in the topology. There is a

nette reconnects

Hi - We are seeing workers dying and restarting quite a bit, apparently from netty connection issues. For example, the log below shows: * Reconnect for worker at 121:6700 * connection established to 121:6700 * closing connection to 121:6700 * Reconnect started to 121:6700 all within 1 second.

Re: LoggingMetricsConsumer

I have not used Trident, but I think the metrics should be handled the same for trident bolts as normal rich bolts. You can confirm that the metrics bolt is there by looking in the storm ui. Click on the topology and go down to the bottom of the page and click on "Show System Stats" and it will s

possible memory leak in supervisor? storm 0.9.0

We are running storm 0.9.0 and we see a gradual increase in consumption of the supervisor process memory. It is growing approximately at 4KB every minute or so. Our machines show a gradual decrease in available memory over 7 days till the memory spikes back up and we see supervisors restart. It

Re: What's the best way to guarantee external delivery of messages with Storm

Related to redelivery, it depends on the granularity of what you are comfortable replaying. If the whole process can easily be replayed, you can allow the http timeout/failure to fail the tuple, and track that tuple in your spout for replaying later. If you don’t want the whole process to repl

Re: LoggingMetricsConsumer

Thanks for your response John, Could you explain to me how this would work when I am using Trident? Since with Trident bolts are abstracted away from the user, how might I configure my own MetricsConsumerBolt / debug it to figure out why it isn't calling "handleDataPoints()" ? my metricsConsumer's

Trident Kafka Spout - Ack count increasing even though no messages are processed

Hi, I deployed a basic trident topology in HDP storm cluster. I am using TransactionalTridentKafkaSpout to process messages from Kafka Queue . For some reason the ack count in Storm GUI in spout and bolt increases by 20 every second even if there is no messages in Kafka queue . Is there any reaso

Re: What's the best way to guarantee external delivery of messages with Storm

If we don't care about how many times the message is delivered (at least once) then we can use some error handling in HTTP to achieve a guarantee. You can use a request/response in HTTP and until you get a HTTP 200/202 you can retry the delivery. To get a exactly once guarantees we may need to go t

Re: What's the best way to guarantee external delivery of messages with Storm

Will the HTTP event sink respond with some acknowledgement that it received whatever was sent? If so, could this be as simple as telling your bolt not to ack the tuple until this response is received from the HTTP service? -- Derek On 9/26/14 10:10, Peter Neumark wrote: Thanks for the quick

Re: What's the best way to guarantee external delivery of messages with Storm

Thanks for the quick response! Unfortunately, we're forced to use HTTP. Any ideas? On Fri, Sep 26, 2014 at 5:07 PM, Supun Kamburugamuva wrote: > On Fri, Sep 26, 2014 at 10:49 AM, Peter Neumark > wrote: > >> Hi all, >> >> We want to replace a legacy custom app with storm, but -being storm >> new

Re: What's the best way to guarantee external delivery of messages with Storm

On Fri, Sep 26, 2014 at 10:49 AM, Peter Neumark wrote: > Hi all, > > We want to replace a legacy custom app with storm, but -being storm > newbies- we're not sure what's the best way to solve the following problem: > > An HTTP endpoint returns the list of events which occurred between two > times

What's the best way to guarantee external delivery of messages with Storm

Hi all, We want to replace a legacy custom app with storm, but -being storm newbies- we're not sure what's the best way to solve the following problem: An HTTP endpoint returns the list of events which occurred between two timestamps. The task is to continuously poll this event source for new eve

RE: java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/SOIMDB

Hi Team Restarting apache storm helped us here.thanks -Original Message- From: Pavan Jakati G Sent: Fri 9/26/2014 12:45 PM To: user@storm.incubator.apache.org Subject: java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/SOIMDB Hi Team, We are facing so

java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/SOIMDB

Hi Team, We are facing some weird issue when try running topology in remote mode .For some reason apache storm is unable to find the class in the driver . We have placed mysql-connector jar file under /lib directory of apache storm and is getting identified when run storm classpath . The co