Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-15 Thread Jiawei Wu
Thanks Arvid and Kurt. That's very helpful discussion. Currently we will continue this with Lambda, but I'll definitely do a A-A test between Lambda and Flink for this case. Regards, Jiawei On Wed, Mar 11, 2020 at 5:40 PM Kurt Young wrote: > > The second reason is this query need to scan the

Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-11 Thread Kurt Young
> The second reason is this query need to scan the whole table. I think we can do better :-) Not necessarily, you said all the changes will trigger a DDB stream, you can use Flink to consume such stream incrementally. For the 1st problem, I think you can use DataStream API and register a timer

Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-11 Thread Arvid Heise
> > About the problem, we have 2 choices. The first one is using Flink as > described in this email thread. The second one is using AWS Lambda > triggered by CDC stream and compute the latest 15 days record, which is a > walk-around solution and looks not as elegant as Flink to me. > > Currently

Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-11 Thread Jiawei Wu
Hi Kurt, What you said is the 1st reason. The second reason is this query need to scan the whole table. I think we can do better :-) Best, Jiawei On Wed, Mar 11, 2020 at 10:52 AM Kurt Young wrote: > Hi Jiawai, > > Sorry I still didn't fully get your question. What's wrong with your > proposed

Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-10 Thread Kurt Young
Hi Jiawai, Sorry I still didn't fully get your question. What's wrong with your proposed SQL? > select vendorId, sum(inventory units) > from dynamodb > where today's time - inbound time > 15 > group by vendorId My guess is that such query would only trigger calculations by new event. So if a

Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-10 Thread Jiawei Wu
Hi Robert, Your answer really helps. About the problem, we have 2 choices. The first one is using Flink as described in this email thread. The second one is using AWS Lambda triggered by CDC stream and compute the latest 15 days record, which is a walk-around solution and looks not as elegant as

Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-09 Thread Robert Metzger
Hey Jiawei, I'm sorry that you haven't received an answer yet. So you basically have a stream of dynamodb table updates (let's call id CDC stream), and you would like to maintain the inventory of the last 15 days for each vendor. Whenever there's an update in the inventory data (a new event

Use flink to calculate sum of the inventory under certain conditions

2020-03-02 Thread Jiawei Wu
Hi flink users, We have a problem and think flink may be a good solution for that. But I'm new to flink and hope can get some insights from flink community :) Here is the problem. Suppose we have a DynamoDB table which store the inventory data, the schema is like: * vendorId (primary key) *