Re: Bootstrapping

Hi Gregory, I have similar issue when dealing with historical data. We choose Lambda and figure out use case specific hand off protocol. Unless storage side can support replay logs within a time range, Streaming application authors still needs to carry extra work to implement batching layer

Bootstrapping

Hi group, I want to bootstrap some aggregates based on historic data in S3 and then keep them updated based on a stream. To do this I was thinking of doing something like processing all of the historic data, doing a save point, then restoring my program from that save point but with a stream

Re: Kafka Producer timeout causing data loss

The reorder issue can be resolved by setting MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to 1 if we talking pure kafka producer configs ( and I believe they port over to flink kafka connecter ). This does limit the concurrency ( at the TCP level ) when kafka is back up an issue which is not very

Re: Send ACK when all records of file are processed

Hi, If an operator has multiple inputs, it’s watermark will be the minimum of all of the inputs. Thus your hypothetical “ACK Operator” will get Watermark(Long.MAX_VALUE) only when of the preceding operators report Watermark(Long.MAX_VALUE). Yes, instead of simply adding sink, you would have

Re: Send ACK when all records of file are processed

Hi Piotrek, Thank you for your detailed answer. Yes, I want to generate the ack when all the records of the file are written to DB. So to understand what you are saying , we will receive a single EOF watermark value at the ack operator when all the downstream operator process all the records of

Re: Trying to understand why a job is spilling despite of huge memory provided

Hi Konstantin, Flink's sort algorithm works as follows: - Each memory-consuming task (such as sort or combine) has a memory budget which depends on the number of operators in the plan, the TM managed memory, and the number of slots of the TM. Each TM slot has the same fraction of the over all TM

Re: Failing to recover once checkpoint fails

To add to this, we are assuming that the default configuration will fail a pipeline if a checkpoint fails and will hit the recover loop only and only if the retry limit is not reached On Thu, Jan 25, 2018 at 7:00 AM, Vishal Santoshi wrote: > Sorry. > > There are 2

Re: Data exchange between tasks (operators/sources) at streaming api runtime

Yes, makes sense, I think consider one of those better options. Thanks! Ishwara Sent from my iPhone > On Jan 25, 2018, at 7:12 AM, Piotr Nowojski wrote: > > If you want to go this way, you could: > - as you proposed use some busy waiting with reading some file from a

Re: Timer & Window Memory Consumption

You can connect to the TaskManagers with a tool such as jvisualvm to observe where the objects are created. It doesn't sound normal that there are millions of these objects if only a couple thousand elements come in. > On 25. Jan 2018, at 14:59, Fabian Hueske wrote: > >

Re: Timer & Window Memory Consumption

Aljoscha (in CC), do you have an idea about this issue? Thanks, Fabian 2018-01-24 7:06 GMT+01:00 Navneeth Krishnan : > Thanks Fabian but for 1.5k messages per second per TM there are several > million Internal & TimerWindow objects created within a period of 5 >

Re: Does web ui hang for 1.4 for job submission ?

In our case we do not see that happening...when the web ui hangs and we refresh the page it does not effect TMs etc. Is there something splecific that you do that triggers this other issue ? On Jan 25, 2018 7:15 AM, "Chesnay Schepler" wrote: > That's a separate issue, and I

Re: Data exchange between tasks (operators/sources) at streaming api runtime

If you want to go this way, you could: - as you proposed use some busy waiting with reading some file from a distributed file system - wait for some network message (opening your own socket) - use some other external system for this purpose: Kafka? Zookeeper? Although all of them seems hacky

Re: Data exchange between tasks (operators/sources) at streaming api runtime

The FLIP-17 is promising. Until it’s available I’m planning to do this: extend Kafka consumer and add logic to hold consuming until other source (fixed set) completes sending and those messages are processed by the application. However the question is to how to let the Kafka consumer know that

Re: Does web ui hang for 1.4 for job submission ?

That's a separate issue, and I would think deserves a JIRA as it sounds rather serious. On 25.01.2018 12:43, Lasse Nedergaard wrote: Great news  Does it also cover protection against loosing all your task managers? Med venlig hilsen / Best regards Lasse Nedergaard Den 25. jan. 2018 kl.

Re: Failing to recover once checkpoint fails

Sorry. There are 2 scenerios * Idempotent Sinks Use Case where we would want to restore from the latest valid checkpoint. If I understand the code correctly we try to retrieve all completed checkpoints for all handles in ZK and abort ( throw an exception ) if there are handles but no

Re: Does web ui hang for 1.4 for job submission ?

Great news  Does it also cover protection against loosing all your task managers? Med venlig hilsen / Best regards Lasse Nedergaard > Den 25. jan. 2018 kl. 12.00 skrev Chesnay Schepler : > > In 1.4, jobs submitted through the WebUI that never actually execute a job >

Re: Does web ui hang for 1.4 for job submission ?

In 1.4, jobs submitted through the WebUI that never actually execute a job will cause the UI to hang, with the only option of resolving that being to reload the page. This will be fixed for 1.4.1 and 1.5 . On 24.01.2018 19:19, Vishal Santoshi wrote: Exactly, the same context and issue. We too

Re: Send ACK when all records of file are processed

Hi, As you figured out, some dummy EOF record is one solution, however you might try to achieve it also by wrapping an existing CSV function. Your wrapper could emit this dummy EOF record. Another (probably better) idea is to use Watermark(Long.MAX_VALUE) for the EOF marker. Stream source

Re: Avoiding deadlock with iterations

Hi, This is a known problem and I don’t think there is an easy solution to this. Please refer to the: http://mail-archives.apache.org/mod_mbox/flink-user/201704.mbox/%3c5486a7fd-41c3-4131-5100-272825088...@gaborhermann.com%3E

Re: Data exchange between tasks (operators/sources) at streaming api runtime

Hi, As far as I know there is currently no simple way to do this: Join stream with static data in https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API and