Trace jar file name from jobId, is that possible?

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app. I want to use savepoints to resume a job after each deployment. As you know I need jar file name and path to savepoints to resume a task. Currently `flink list` command only gives me job ids, not jar file names. And

non-shared TaskManager-specific config file

Hi team, I am wondering how can I create a non-shared config file and let Flink read it. Can I use include in the config? Or I have to prepare a different config for each TM? https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/config.html - taskmanager.hostname: The

Re: Checkpoint expired before completing

Here is the checkpoint config. no concurrent checkpoints with 2 minute checkpoint interval and timeout. Problem is gone after redeployment. I will try if I can reproduce the issue [image: Inline image 1] On Fri, Dec 1, 2017 at 6:17 AM, Nico Kruber wrote: > Hi Steven, >

aggregate does not allow RichAggregateFunction ?

I have a simple Aggregation with one caveat. For some reason I have to keep a large amount of state till the window is GCed. The state is within the Accumulator ( ACC ). I am hitting a memory bottleneck and would like to offload the state to the states backend ( ROCKSDB), keeping the between

Re: Incremental RocksDB checkpointing

Ok, I think all I can comment about this case was already in the previous email. Incremental checkpoints are designed with large state in mind and you cannot extrapolate this observation to e.g. 1 million keys, so I think everything is working just fine. > Am 01.12.2017 um 18:22 schrieb

Re: Incremental RocksDB checkpointing

I observed the job for 18 hrs, it went from 118kb to 1.10MB. I am using version 1.3.0 flink On Fri, Dec 1, 2017 at 11:39 AM, Stefan Richter wrote: > Maybe one more question: is the size always increasing, or will it also > reduce eventually? Over what period of

Re: Incremental RocksDB checkpointing

Maybe one more question: is the size always increasing, or will it also reduce eventually? Over what period of time did you observe growth? From the way how RocksDB works, it does persist updates in a way that is sometimes closer to a log than in-place updates. So it is perfectly possible that

Re: flink local & interactive development

That is basically the same thing, only that sbt will start a default scala shell but put my personal code directly on the class path and requiring to manually start flink. However, this doesn't fully seem to work. Fabian Hueske schrieb am Fr. 1. Dez. 2017 um 10:11: > Hi

Incremental RocksDB checkpointing

I have simple event time window aggregate count function with incremental checkpointing enabled. The checkpoint size keeps increasing over a period of time, even though my input data has a single key and data is flowing at a constant rate. When i turn off incremental checkpointing, checkpoint

Re: Non-intrusive way to detect which type is using kryo ?

Generally, you don't need to run it on the cluster. Types are available in the so-called pre-flight phase (before calling env.execute()). So you can just print the types in your IDE while developing: "println(stream.getType)" Regards, Timo Am 12/1/17 um 1:42 PM schrieb Kien Truong: Hi

Re: Checkpoint expired before completing

Hi Steven, by default, checkpoints time out after 10 minutes if you haven't used CheckpointConfig#setCheckpointTimeout() to change this timeout. Depending on your checkpoint interval, and your number of concurrent checkpoints, there may already be some other checkpoint processes running while you

Re: ElasticSearch Connector for version 6.x and scala 2.11

Hi A workmate of mine tried to migrate the existing flink connector to ElasticSearch 6 but we had problems with netty dependencies that clashed (Flink uses 4.0.27 and ES is on 4.1). You can change the flink-connector-elasticsearch5 connector to ES 5.6.4, but then you have to do some adaptions to

Re: Kafka consumer to sync topics by event time?

Thanks for the answers, I still don't understand why I can see the offsets being quickly committed to Kafka for the "small topic"? Are they committed to Kafka before their watermark has passed on Flink's side? That would be quite confusing.. Indeed when Flink handles the state/offsets internally,

Re: Non-intrusive way to detect which type is using kryo ?

Hi Timo, Having a utility like that would be great. It'd be even better if it's can be executed without having to actually run a cluster. Best regards, Kien On 11/28/2017 3:39 PM, Timo Walther wrote: Hi Kien, at the moment I'm working on some improvements to the type system that will

Re: flink local & interactive development

Hi Georg, I have no experience with SBT's console mode, so I cannot comment on that, but Flink provides a Scala REPL that might be useful [1]. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/scala_shell.html 2017-11-30 23:09 GMT+01:00 Georg Heiler

Re: ElasticSearch Connector for version 6.x and scala 2.11

Hi Rahul, Flink does not provide a connector for ElasticSearch 6 yet. There is this JIRA issue to track the development progress [1]. Best, Fabian [1] https://issues.apache.org/jira/browse/FLINK-8101 2017-12-01 7:22 GMT+01:00 Rahul Raj : > Hi All, > > Is there a Flink