Thanks a lot Arun, exactly what I needed.
I created a ticket (https://issues.apache.org/jira/browse/STORM-1873)
and a pull request with a slightly different implementation to move the
discussion forward.
--
Balázs
On 05/30/2016 09:41 PM, Arun Mahadevan wrote:
Hi Balázs,
The idea sounds good.
Only whitelisted configs can be overridden at component level. The java configs
are converted to clojure variables by replacing _ (underscore) with - (hyphen)
and you need to add and entry like [1] in executor.clj to get it working.
- Arun
[1]
https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/executor.clj#L114
On 5/30/16, 11:26 PM, "Balázs Kossovics" <balazs.kossov...@s4m.io> wrote:
Hey,
I'm trying to implement an alterative behaviour in case of late tuples
for windowing. Currently late tuples are just logged (and acknowledged
in the coming 1.0.2), but in my usecase it would be desirable to emit
them onto a user defined stream. One could define a bolt with a stream
for late tuples like this:
new MyWindowedBolt()
.withTimestampField("timestamp")
* .withLateTupleStream("late_tuples")
.withWindow(
new BaseWindowedBolt.Duration(1, TimeUnit.MINUTES),
new BaseWindowedBolt.Duration(1, TimeUnit.SECONDS)
)
I made a quick patch
(https://github.com/kosii/storm/commit/216c991da3c5b6c6cac1b25182b86507c3fb5e9e)
to test the idea, where the indended behaviour would be something like this:
1, the withLateTupleStream builder method puts a new key into the
windowConfiguration Map,
2, in WindowedBoltExecutor. declareOutputFields a new stream gets declared,
3, in WindowedBoltExecutor.prepare we store the stream's name in a
private variable,
4, if this variable is not null, then in the execute method we emit each
late tuple onto the new stream.
The problem is in the 3rd step
(https://github.com/kosii/storm/commit/216c991da3c5b6c6cac1b25182b86507c3fb5e9e#diff-32eff130ad25f4b7b6c069e8d42245acR170),
where the stormConf map doesn't contain anymore my key, which was still
present in the 2nd step. I'm out of ideas, so I'd really appreciate if
someone could explain me what's happening here.
I'm also interested in your ideas concerning the feature, and what would
you like to eventually see in the upstream.
Best regards,
Balazs
