However this gets done, I think it is going to be a common problem. I think we should definitely figure this out somehow.
To me this sounds like a "streaming join" problem, rather then just simply wanting to sleep. Storm has some minimal advise on doing a streaming join; http://storm.apache.org/releases/1.0.1/Common-patterns.html. This also relates to how we implemented threat intelligence enrichment. We currently load threat intelligence data in a batch manner and then enrich the streaming data with the batch data. If you can load one side in batch, then the problem goes away. Understand that you cannot always load one-side in batch. On Fri, Nov 4, 2016 at 10:29 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > The reason I say spout is the naive thought that it is better to leave the > ‘backup’/caching to kafka than to add it in somewhere else > > > On November 4, 2016 at 10:28:53, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > So spout orchestration/gating? > > Spout checks for external state flag > > if CURRENT - process > if UPDATING - wait > > With the ingesting agent sets flag to updating when running? > > > On November 4, 2016 at 09:29:16, zeo...@gmail.com (zeo...@gmail.com) > wrote: > > Is there a good method (i.e. something using Stellar/ZK) to implement an > intentional processing delay to all tuples in a specific topology? I plan > to do some custom enrichments, but the data used to do the enrichment *may* > be > ingested at roughly the same time the data to be enriched is (it also may > not ever be sent). So I'd like to add a delay in my cluster that applies > to certain parser topologies. > > I took a look around in the documentation and in JIRA and didn't find > anything available or being worked on, but I did see that this may conflict > with METRON-322. Essentially what I'm considering is a {sleep,delay,wait} > stellar function, but it could also be a delay in a parser's kafka spout > (much less of a fan of the second option). > > I'm looking for feedback on the best way to approach this, and I'd be happy > to do the work myself (if necessary) when it gets to that point. I did > consider implementing this delay upstream (in the sensor itself), but after > looking in more detail it doesn't seem as feasible. > > Jon > -- > > Jon > -- Nick Allen <n...@nickallen.org>
