Re: Using Lambda with Storm Flux

2020-01-15 Thread Roshan Naik
I am not sure if this can be achieved via flux or not although Storm 2.0 introduced support for lambda expressions. https://issues.apache.org/jira/browse/STORM-2490 I think, Taylor is the original author of flux. If there is a way to access this feature via flux... he might be able to tell. -

Re: Using Lambda with Storm Flux

2020-01-15 Thread joe smith
Anyone have info/tips? On Wednesday, December 4, 2019, 11:08:36 AM EST, joe smith wrote: Hi, Would like to know if there is a way to write a lambda expression in storm flux so it can be used to create/define a component. For example:   - id: kafkaSpoutFields     className: org.ap

Re: question on deactivate and kill

2020-01-15 Thread Rui Abreu
To change the number of executors to a particular bolt, you can execute this on a Nimbus: storm rebalance exclamation-topology -e even-digit-bolt=3 But again, this will have a practical effect if you have spare tasks that those executors might work on. On Wed, 15 Jan 2020 at 17:32, Zainal Arifin (

Re: question on deactivate and kill

2020-01-15 Thread Zainal Arifin (BLOOMBERG/ 731 LEX)
Thanks Rui for the answer. The link mentioned that we can use Storm Web UI to re-balance the topology, but can I increase the number of parallelism of particular bolt only? I guess not. Btw, we are using streamparse, https://streamparse.readthedocs.io/en/stable/topologies.html#parallelism-and-w

Re: question on deactivate and kill

2020-01-15 Thread Rui Abreu
Deactivating a topology means the spouts stop consuming, but everything else continues to work as expected, meaning your bolts will process all in flight tuples. You can change the number of workers and executors in runtime. In order to scale up the latter, in you need to have previously submitted

question on deactivate and kill

2020-01-15 Thread Zainal Arifin (BLOOMBERG/ 731 LEX)
Hi, Right now we have a bolt task that is in back log, i.e. ~10 hours behind processing the data. There was a bug and we have the fix which will speed up drastically. I think if I kill the topology (to deploy the fix), I will definitely lose those events been processed by the bolt, is that correc