Hiring Flink developers

2020-09-26 Thread Dan Hill
I'm looking to hire Flink developers (full time or contractors) to work on a specialized user event logging system. Besides for the usual developer hiring websites, what are good hiring sources for Flink developers? Thanks! - Dan

Re: Checkpoint dir is not cleaned up after cancel the job with monitoring API

2020-09-26 Thread Congxian Qiu
Hi Eleanore What the `CheckpointRetentionPolicy`[1] did you set for your job? if `ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION` is set, then the checkpoint will be kept when canceling a job. PS the image did not show [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/stat

Re: Flink 1.10 classpath

2020-09-26 Thread Yang Wang
It is an interesting requirement. Usually, we suggest our users to add these dependencies in lib directory or ship via --yarnship cli option. If you really do not want to do this, I think you could ship an updated yarn-site.xml, set "yarn.application.classpath" including the mapreduce jars. Then a

Checkpoint dir is not cleaned up after cancel the job with monitoring API

2020-09-26 Thread Eleanore Jin
Hi experts, I am running flink 1.10.2 on kubernetes as per job cluster. Checkpoint is enabled, with interval 3s, minimumPause 1s, timeout 10s. I'm using FsStateBackend, snapshots are persisted to azure blob storage (Microsoft cloud storage service). Checkpointed state is just source kafka topic o

Re: Back pressure with multiple joins

2020-09-26 Thread Dan Hill
I can't reproduce the issue now. I'm using the same commits so I'm guessing another environment factor is at play. The explains are pretty similar (there's an extra LogicalProject when there is an extra view). On Fri, Sep 25, 2020 at 12:55 AM Timo Walther wrote: > Hi Dan, > > could you share t

Dynamic Kafka Source

2020-09-26 Thread Prasanna kumar
Hi, My requirement has been captured by the following stack overflow question. https://stackoverflow.com/questions/61876849/custom-kafka-source-on-apache-flink Could anyone take a shot at it ? Thanks, Prasanna.