Flink HA on Kubernetes - RPC port

2023-01-20 Thread bastien dine
Hello, We are migrating our HA setup from ZK to K8S, and we have a question regarding the RPC port. Previously with ZK, the RPC connection config was the : high-availability.jobmanager.port We were expecting that the config will be the same with K8S HA, as the doc says : "The port (range) used b

DuplicateJobSubmissionException on restart after taskmanagers crash

2023-01-20 Thread Javier Vegas
I have a Flink app (Flink 1.16.0, deployed to Kubernetes via operator 1.3.1 and using Kubernetes HighAvailaibilty with storage in S3) that depends on multiple Thrift services for data queries. When one of those services is down (or throws exceptions) the Flink job managers end up crashing and only

Re: UDFs classloading changes in 1.16

2023-01-20 Thread Charles Tan
Hi everyone, I wanted to bring some attention to this issue again. It seems that the suggested approach to creating UDFs in https://issues.apache.org/jira/browse/FLINK-29890 breaks when checkpointing is enabled. https://issues.apache.org/jira/browse/FLINK-28897 reports a very similar issue. I've u

Fwd: UDFs classloading changes in 1.16

2023-01-20 Thread Martijn Visser
I'm forwarding this issue to the Dev mailing list since it should be discussed there, moving User ML to BCC. Also adding Dong since he was involved in the discussion Thanks for reporting this Charles. I've re-opened https://issues.apache.org/jira/browse/FLINK-28897 but it could be that this ultima

Re: HybridSource: how to specify different watermark strategies for underlying sources

2023-01-20 Thread Martijn Visser
Hi David, That's good to know, perhaps it would be nice to see if we can move it all forward into a new version. The externalization is nearly finished (I pushed a release candidate out for the version that's currently bundled with Flink 1.16), so let's see if we can move it forward. I'm also loop

Custom catalog implementation - getting table schema for computed columns

2023-01-20 Thread Krzysztof Chmielewski
Hi, I'm implementing a custom Catalog where for "create table" I need to get tables schema, both column names and types from DDL. Now the Catalog's createTable method has "CatalogBaseTable table" argument. The CatalogBaseTable has a deprecated "getSchema" and suggest to use getUnresolvedSchema in

Re: Custom catalog implementation - getting table schema for computed columns

2023-01-20 Thread Krzysztof Chmielewski
Ok, so now I see that runtime type of "table" parameter is ResolvedCatalogTable that has method getResolvedSchema. So I guess my question is, can I assume that ResolvedCatalogTable will be always a runtime type? pt., 20 sty 2023 o 19:27 Krzysztof Chmielewski < krzysiek.chmielew...@gmail.com> napi

Blob server connection problem

2023-01-20 Thread Yang Liu
Hello, Is anyone familiar with the "blob server connection"? We have constantly been seeing the "Error while executing Blob connection" error, which sometimes causes a job stuck in the middle of a run if there are too many connection errors and eventually causes a failure, though most of the time

Re: DuplicateJobSubmissionException on restart after taskmanagers crash

2023-01-20 Thread Javier Vegas
My issue is described in https://issues.apache.org/jira/browse/FLINK-21928 where it says was fixed in 1.14, but I am still seeing the problem. Although there it says: "Additionally, it is still required that the user cleans up the corresponding HA entries for the running jobs registry because thes