Re: Enriching data from external source with cache

2017-10-02 Thread Derek VerLee
Thanks Timo, watching the video now. I did try out the method with iteration in a simple prototype and it works.  But you are right, combining it with the other requirements into a single process function has so far resulted in more complexity than I'd like, and

Re: Enriching data from external source with cache

2017-10-02 Thread Timo Walther
Hi Derek, maybe the following talk can inspire you, how to do this with joins and async IO: https://www.youtube.com/watch?v=Do7C4UJyWCM (around the 17th min). Basically, you split the stream and wait for an Async IO result in a downstream operator. But I think having a transient guava cache

Enriching data from external source with cache

2017-09-29 Thread Derek VerLee
My basic problem will sound familiar I think, I need to enrich incoming data using a REST call to an external system for slowly evolving metadata. and some cache based lag is acceptable, so to reduce load on the external system and to process more efficiently, I would