Flink Version 1.11 job savepoint failures

Hi Flink Community, Our flink jobs are in version 1.11 and we use this to trigger savepoint. $ bin/flink savepoint :jobId [:targetDirectory] We can get trigger Id with savepoint path successfully. But we saw these errors by querying savepoint endpoint: https://ci.apache.org/projects/flink/flink-d

Re: Use State query to dump state into datalake

I think you'd be better off using the State Processor API [1] instead. The State Processor API has cleaner semantics -- as you'll be seeing a self-consistent snapshot of all the state -- and it's also much more performant. Note also that the Queryable State API is "approaching end of life" [2]. Th

Scale up REGEX pipeline

Hi community, I'm working in a pipeline which needs to apply several REGEX expressions for matching. I have around 10K Regex expressions but, depending on some metadata in the message, I only need to apply 5-10 for that specific message. I've being doing some research

Re: Flink Version 1.11 job savepoint failures

Hi Rainie, the savepoint creation failed due to some tasks already being finished. It looks like you ran into an issue that was (partially as FLINK-21066 [1] is only a subtask of a bigger issue?) addressed in Flink 1.13 (see FLINK-21066). I'm pulling Yun Gao into this thread. Let's see whether Yun

java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.Reque

Hi Team, I have Flink 1.8.1 and hadoop open source 3.2.0 . My flink jobs run without issues on HDP 2.5.3 version. when run on hadoop open source 3.2.0 encountering the below mentioned exception . I have set hadoop export HADOOP_CONF_DIR=/etc/hadoop/conf export HADOOP_CLASSPATH=`hadoop classpath`

Re: Setup of Scala/Flink project using Bazel

Hi Salva, unfortunately, I have no experience with Bazel. Just by looking at the code you shared I cannot come up with an answer either. Have you checked out the ML thread in [1]? It provides two other examples where users used Bazel in the context of Flink. This might give you some hints on where

Re: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.R

Hi Ragini, this is a dependency version issue. Flink 1.8.x does not support Hadoop 3, yet. The support for Apache Hadoop 3.x was added in Flink 1.11 [1] through FLINK-11086 [2]. You would need to upgrade to a more recent Flink version. Best, Matthias [1] https://flink.apache.org/news/2020/07/06/r

[ANNOUNCE] Apache Flink 1.13.0 released

|The Apache Flink community is very happy to announce the release of Apache Flink 1.13.0.|   |Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications.|   |The release is available for download at:| |htt

Re: Protobuf support with Flink SQL and Kafka Connector

Hi Shipeng, it looks like there is an open Jira issue FLINK-18202 [1] addressing this topic. You might want to follow up on that one. I'm adding Timo and Jark to this thread. They might have more insights. Best, Matthias [1] https://issues.apache.org/jira/browse/FLINK-18202 On Sat, May 1, 2021 a

Re: remote task manager netty exception

Hi, I see that JM and TM failures are different (from TM, it's actually a warning). Could you please share the ERROR message from TM? Have you tried increasing taskmanager.network.retries [1]? [1] https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#taskmanager-network-r

Re: [ANNOUNCE] Apache Flink 1.13.0 released

This is great news. Thanks a lot for being our release managers Dawid and Guowei! And also thanks to everyone who has made this release possible :-) Cheers, Till On Mon, May 3, 2021 at 5:46 PM vishalovercome wrote: > This is a very big release! Many thanks to the flink developers for their > co

Re: Flink(1.12.2/scala 2.11) HA with Zk in kubernetes standalone mode deployment is not working

Hi Bhagi, Thanks for reaching out to the Flink community. The error the UI is showing is normal during an ongoing leader election. Additionally, the connection refused warnings seem to be normal according to other mailing list threads. Are you referring to the UI error as the issue you are facing?

Re: Setup of Scala/Flink project using Bazel

Hi Matthias, Thanks a lot for your reply. I am already aware of that reference, but it's not exactly what I need. What I'd like to have is the typical word count (hello world) app migrated from sbt to bazel, in order to use it as a template for my Flink/Scala apps. -- Sent from: http://apache

Re: Flink(1.12.2/scala 2.11) HA with Zk in kubernetes standalone mode deployment is not working

Somewhere the system retrieves the address x.x.x.x:43092 which cannot be connected to. Can you check that this points towards a valid Flink process? Maybe it is some leftover information in the ZooKeeper from a previous run? Maybe you can check what's written in the Znodes for /leader/resource_mana

Re: Zookeeper or Kubernetes for HA?

Hi Vishal, Do I understand you correctly that you're wondering whether you should stick to ZooKeeper HA on Kubernetes vs Kubernetes HA? You could argue that ZooKeeper might be better since it's already supported for longer and, therefore, better tested. The Kubernetes HA implementation left the exp

Re: Questions about implementing a flink source

Hi Evan, 1) You are absolutely correct that we would urge users to add new sources as FLIP-27 and new sinks as FLIP-143. I can provide guidance in both cases. For FLIP-27 sources, I'd recommend using KafkaSource [1] and FileSource [2] as a starting point. Especially basing the reader implementatio

Is keyed state supported in PyFlink?

Hi, Is keyed state [1] supported by PyFlink yet? I can see some code for it in the Flink master branch, but there's no mention of it in the 1.12 Python documentation. Thanks, Sumeet [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/state/state.html

Interactive Programming in Flink (Cache operation)

Hello, Does the new release of Flink 1.13.0 includes the cache operation feature (https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink). Thank you, Iacovos

Re: Flink Version 1.11 job savepoint failures

It helps. Thanks Matthias. Best regards Rainie On Mon, May 3, 2021 at 4:25 AM Matthias Pohl wrote: > Hi Rainie, > the savepoint creation failed due to some tasks already being finished. It > looks like you ran into an issue that was (partially as FLINK-21066 [1] is > only a subtask of a bigger

Re: Setup of Scala/Flink project using Bazel

Hey Salva, This appears to be a bug in the `bazel-deps` tool, caused by mixing scala and Java dependencies. The tool seems to use the same target name for both, and thus produces duplicate targets (one for scala and one for java). If you look at the dict lines that are reported as conflicting, yo

Re: [ANNOUNCE] Apache Flink 1.13.0 released

This is a very big release! Many thanks to the flink developers for their contributions to making Flink as good a framework that it is! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Zookeeper or Kubernetes for HA?

Flink docs provide details on setting up HA but doesn't provide any recommendations as such. For jobs running in kubernetes and having a zookeeper deployment, which high availability option would be more desirable? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.co

Re: StopWithSavepoint() method doesn't work in Java based flink native k8s operator

Hello, Update: I think stopWithSavepoint() only suspend the job. It doesn’t actually terminate (./bin/flink cancel) the job. I switched to cancelWithSavepoint() and it works here. Maybe stopWithSavepoint() should only be used to update the configurations like parallelism? For updating the imag

org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time,

Hi , One of my flink applications needs to get and put records from HBASE for every event while processing in real time . When there are less events the application process without any issues. when the number of events increases we start hitting with the below mentioned exception .Can these excepti

savepoint command in code

Hello, I am trying to use the savepoint command (./bin/flink savepoint jobid) in the code instead of doing it manually in the terminal. The jobid can get using getjobid(). The problem is to define the path ./bin/flink — it can not be shown as a directory (probably because of a unix executable fi

Re: Setup of Scala/Flink project using Bazel

Hey Austin, There was no special reason for vendoring using `bazel-deps`, really. I just took another project as a reference for mine and that project was already using `bazel-deps`. I am going to give `rules_jvm_external` a try, and hopefully I can make it work! Regards, Salva -- Sent from:

Re: StopWithSavepoint() method doesn't work in Java based flink native k8s operator

Hi Fuyao, sorry for not replying earlier. The stop-with-savepoint operation shouldn't only suspend but terminate the job. Is it that you might have a larger state that makes creating the savepoint take longer? Even though, considering that you don't experience this behavior with your 2nd solution,

Re: savepoint command in code

Hi Abdullah, is there a reason you're not considering triggering the stop-with-savepoint operation through the REST API [1]? I'm not entirely sure whether I understand you correctly: ./bin/flink is an executable. Why Would you assume it to be shown as a directory? You would need to provide FLINK_HO

Re: Interactive Programming in Flink (Cache operation)

Hi Iacovos, unfortunately, it doesn't as the related FLINK-19343 [1] is not resolved, yet. The release notes for Flink 1.13 can be found in [2]. Best, Matthias [1] https://issues.apache.org/jira/browse/FLINK-19343 [2] https://flink.apache.org/news/2021/05/03/release-1.13.0.html On Mon, May 3, 20

Re: Interactive Programming in Flink (Cache operation)

Hi Matthias, Thank you for your reply. Are you going to include it in future versions? Best, Iacovos On 4/5/21 9:10 π.μ., Matthias Pohl wrote: Hi Iacovos, unfortunately, it doesn't as the related FLINK-19343 [1] is not resolved, yet. The release notes for Flink 1.13 can be found in [2]. Bes

Re: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.R

Thank you for the clarification. On Mon, May 3, 2021 at 6:57 PM Matthias Pohl wrote: > Hi Ragini, > this is a dependency version issue. Flink 1.8.x does not support Hadoop 3, > yet. The support for Apache Hadoop 3.x was added in Flink 1.11 [1] through > FLINK-11086 [2]. You would need to upgrade

Re: Interactive Programming in Flink (Cache operation)

It is planned for some future release as the corresponding vote [1] succeeded. I guess the effort is stalled for some time, unfortunately. I'm gonna add Timo to this conversation. He might have more insights on the plans to proceed with it. [1] http://apache-flink-mailing-list-archive.1008284.n3.n

Re: [ANNOUNCE] Apache Flink 1.13.0 released

Thank you Dawid and Guowei! Great job everyone :) On Mon, May 3, 2021 at 7:11 PM Till Rohrmann wrote: > This is great news. Thanks a lot for being our release managers Dawid and > Guowei! And also thanks to everyone who has made this release possible :-) > > Cheers, > Till > > On Mon, May 3, 202