Re: Distribute Parallelism/Tasks within RichOutputFormat?

I'm not an expert of the streaming APIs but you could try to do something like this: DataStream ds = null; DataStream ds1 = ds.filter(...).setParallelism(3); DataStream ds2 = ds.filter(...).setParallelism(7); Could it fit your needs? Best, Flavio On Wed, Dec 23, 2020 at 3:54 AM Hailu, Andreas [

Re: Distribute Parallelism/Tasks within RichOutputFormat?

Essentially I see 2 options here: a) split your output format such that each format is it's own sink, and then follow Flavio's suggestion to filter the stream and apply each sink to one of the streams, with the respective parallelism. This would be the recommended approach. b) modify your (cust

Re: Flink SQL continuous join checkpointing

Hi Leonard, Thank you for answer, in fact I used regular join because my interval condition was based on wrong column. I extended my join with attribute column condition and it solved the problem: ... FROM table_fx fx LEFT JOIN table_v v ON v.active = fx.instrument_active_id

Re: How to implement a WindowableTask(similar to samza) in Apache flink?

Please note that I responded to this question on Stack Overflow: https://stackoverflow.com/questions/65414125/how-to-implement-a-windowabletask-similar-to-samza-in-apache-flink Regards, David On Wed, Dec 23, 2020 at 7:08 AM Debraj Manna wrote: > I am new to flink and this is my first post in th

How does Flink handle shorted lived keyed streams

Hi, Belos is the use case. Have a stream of transaction events, success/failure of a transaction can be determined by those events. Partitioning stream by transaction id and applying CEP to determine the success/failure of a transaction. Each transaction keyed stream is valid only until the final

Issue in WordCount Example with DataStream API in BATCH RuntimeExecutionMode

Hi team, Recently I am trying to explore the new features of Flink 1.12 with Batch Execution. I locally wrote a classic WordCount program to read from text file and count the words (almost same as the one in Flink Github repo https://github.com/apache/flink/blob/master/flink-examples/flink-exampl

Re: Issue in WordCount Example with DataStream API in BATCH RuntimeExecutionMode

I did a little experiment, and I was able to reproduce this if I use the sum aggregator on KeyedStream to do the counting. However, if I implement my own counting in a KeyedProcessFunction, or if I use the Table API, I get correct results with RuntimeExecutionMode.BATCH -- though the results are p

Re: Re: checkpoint delay consume message

Checkpoint can be done synchronously and asynchronously, the latter is the default . If you chooese the synchronous way , it may cause this problem. nick toker 于2020年12月23日周三 下午3:53写道： > Hi Yun, > > Sorry but we didn't understand your questions. > The delay we are experiencing is on the *read

RE: Distribute Parallelism/Tasks within RichOutputFormat?

Thanks Chesnay, Flavio – I believe Flavio’s first recommendation will work well enough. I agree that the second approach may be a bit finicky to use long-term. Cheers. // ah From: Chesnay Schepler Sent: Wednesday, December 23, 2020 4:07 AM To: Flavio Pompermaier ; Hailu, Andreas [Engineering]

Re: Queryable state on task managers that are not running the job

Hi Martin, What kind of deploy mode you choose? If you use per-job mode [1] to launch jobs, there might exist only idle slots instead of idle taskmanagers. Currently, queryable state is bounded to specific job and if the idle taskmanager is not registered in the target's resource manager, no qu

Just published connect-flink-with-kinesis-kinesalite-using-scala

Hi , After stumbling a little with connecting to kinesis/kinesalite I just published connect-flink-with-kinesis-kinesalite-using-scala hopefully it will assist someone. would love to get your inputtes