Re: Retaining uploaded job jars on Flink HA restarts on Kubernetes

The jar directory is automatically deleted when a JobManager shuts down. In other words, there is no way to retain uploaded jars if a JobManager dies, and no way to point a JobManager to a pre-existing directory. On 07.05.2018 08:18, Chirag Dewan wrote: I think you are looking for *jobmanager

Re: Retaining uploaded job jars on Flink HA restarts on Kubernetes

Hey Chirag, I tried adding both the configs as per the documentation, and I can see the jars getting uploaded to the specified paths, but on JobManager restarts the JARS are actually *deleted* *from* the `jobmanager.web.upload.dir` path. Anything else that I am missing? Thanks. - Rohil On Mon,

Re: Retaining uploaded job jars on Flink HA restarts on Kubernetes

I think you are looking for jobmanager.web.tmpdir along with upload.dir  >From the documentation : - jobmanager.web.tmpdir: This configuration parameter allows defining the Flink web directory to be used by the web interface. The web interface will copy its static files into the dire

Re: Apache Flink - Flink Forward SF 2018 - Scaling stream data pipelines (source code)

Hi, you can find the Flink code here [1] and the Pravega connector code here [2]. Let me know if you run into any problems. [1] https://github.com/tillrohrmann/flink/commits/rescalingPolicy [2] https://github.com/tillrohrmann/flink-connectors/ tree/connectorWithRescaling Cheers, Till On Wed, M

Re: Dynamically deleting kafka topics does not remove partitions from kafkaConsumer

Ah, correct, sorry for the incorrect link. Thanks Ted! On 7 May 2018 at 11:43:12 AM, Ted Yu (yuzhih...@gmail.com) wrote: It seems the correct JIRA should be FLINK-9303 On Sun, May 6, 2018 at 8:29 PM, Tzu-Li (Gordon) Tai wrote: Hi Edward, Thanks for brining this up, and I think your suggestion

Re: Dynamically deleting kafka topics does not remove partitions from kafkaConsumer

It seems the correct JIRA should be FLINK-9303 On Sun, May 6, 2018 at 8:29 PM, Tzu-Li (Gordon) Tai wrote: > Hi Edward, > > Thanks for brining this up, and I think your suggestion makes sense. > The problem is that the Kafka consumer has no notio

Re: Question regarding refreshing KafkaConsumer in FlinkKafkaConnector

Hi, AFAIK, there is no built-in feature for scheduling / orchestrating submission of Flink jobs. However, you should be able to easily use tools like cron jobs to do that. It should work by just taking a savepoint of your running job, and then resuming for that, and you do this periodically. Chee

Re: Assign JIRA issue permission

Hi Sampath, Do you already have a target JIRA that you would like to work on? Once you have one, let us know the JIRA issue ID and your JIRA account ID, then we'll assign you contributor permissions. With that, you can pick up unassigned JIRA issues to work on by yourself in the future. Cheers,

Re: Dynamically deleting kafka topics does not remove partitions from kafkaConsumer

Hi Edward, Thanks for brining this up, and I think your suggestion makes sense. The problem is that the Kafka consumer has no notion of "closed" partitions at the moment, so statically assigned partitions to the Kafka client is never removed and is always continuously requested for records. For e

Standalone HA Cluster using Shared Zookeeper

Hello All, I am planning on setting up a Standalone HA Cluster using the same Zookeeper cluster I've that is managing a Kafka Cluster. The Zookeeper cluster for Kafka has a /kafka chroot specified and for flink the chroot would be /flink (so they are isolated). Are there any downsides to this app

Re: Stashing key with AggregateFunction

Hi Ken, Thanks for the bug report! Fabian 2018-05-05 0:46 GMT+02:00 Ken Krugler : > Hi Fabian & Stefan, > > Thanks, and yes that does work more like what I’d expect. > > Regards, > > — Ken > > PS - Just FYI the Java code examples in the documentation referenced below > have a number of bugs, se

Re: Use of AggregateFunction's merge() method

Hi Ken, You are right. The merge() method combines partial aggregates, similar to a combinable reducer. The only situation when merge() is called in a DataStream job (that I am aware of) is when session windows get merged. For example when you define a session window with 30 minute gap and you re