Re: How to open a Prometheus metrics port on the rest service when using the Kubernetes operator?

What I would need is to set ports: - name: metrics port: protocol: TCP in the generated YAML fir the appname-rest service which properly aggregates the metrics from the pods, but I can't not figure out either from the job deployment file or modifying the operator templat

Re: flink ci build run longer than the maximum time of 310 minutes.

Usually, it would be more helpful to provide a link to the PR to get a better picture of the problem. I'm not 100% sure whether I grasp what's wrong. It looks like your branch is based on apache/flink:release-1.15 [1]. Therefore, you should fetch the most recent version from upstream and then do a

[NOTICE] Switch docker image base to Eclipse Temurin

On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images will switch bases FROM openjdk:8/11-jar (Debian-based) TO eclipse-temurin:8/11-jre-jammy (Ubuntu-based) due to the deprecation of the OpenJDK images. Users that customized the images are advised to check for breaking changes.

Re: Deploying Jobmanager on k8s as a Deployment

You can use deployments of course , the operator and native k8s integration does exactly that. Even then job updates can be tricky so I believe you are much better off with the operator. Gyula On Sun, 4 Sep 2022 at 11:11, marco andreas wrote: > Hello, > > Thanks for the response, I will take a

Re: [NOTICE] Switch docker image base to Eclipse Temurin

* September 7th On 05/09/2022 11:27, Chesnay Schepler wrote: On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images will switch bases FROM openjdk:8/11-jar (Debian-based) TO eclipse-temurin:8/11-jre-jammy (Ubuntu-based) due to the deprecation of the OpenJDK images. Users that cus

Re: [E] Re: Kubernetes operator expose UI rest service as NodePort instead of default clusterIP

I think we have no concrete reason to always override the "REST_SERVICE_EXPOSED_TYPE" to "ClusterIP". It was introduced to fix the default value for releases before 1.15. And I believe we need to respect the user configured values. Best, Yang Vignesh Kumar Kathiresan 于2022年9月3日周六 05:07写道： > Jac

Re: How to open a Prometheus metrics port on the rest service when using the Kubernetes operator?

I do not think we could add an additional port to the rest service since it is created by Flink internally. Actually, I do not suggest scrapping the metrics from rest service. Instead, the port in the pod needs to be used. Because the metrics might not work correctly if multiple JobManagers are ru

Re: [Flink 1.15.1 - Application mode native k8s Exception] - Exception occurred while acquiring lock 'ConfigMapLock

Could you please check whether the "kubernetes.config.file" is configured to /opt/flink/.kube/config in the Flink configmap? It should be removed before creating the Flink configmap. Best, Yang Tamir Sagi 于2022年9月4日周日 18:08写道： > Hey All, > > We recently updated to Flink 1.15.1. We deploy stream

Flink kafka producer using the same transactional id twice?

Hi all, i am quite new to flink and kafka, so i might mix something up here. The situation is that we do have a flink application (1.14.5 with scala 2.12) running for a few hours to days and suddenly it stops working and can't publish to kafka anymore. I then noticed this message showing up twice

Re: Deploying Jobmanager on k8s as a Deployment

Hey Marco, Unfortunately there is no built in k8s API that models an application mode JM exactly but Deployments should be fine, in general. As Gyula notes, where they can be difficult is during application upgrades as Deployments never let their pods exit, even if successful, so there is no way t

Re: [Flink 1.15.1 - Application mode native k8s Exception] - Exception occurred while acquiring lock 'ConfigMapLock

Hey Yang, The flink-conf.yaml submitted to the cluster does not contain "kubernetes.config.file" at all. In addition, I verified flink config maps under cluster's namespace do not contain "kubernetes.config.file". In addition, we also noticed the following exception (appears to happen sporadic

Slow Tests in Flink 1.15

Hi, we were going to upgrade our application from Flink 1.14.4 to Flink 1.15.2, when we noticed, that all our job tests, using a MiniClusterWithClientResource, are multiple times slower in 1.15 than before in 1.14. I, unfortunately, have not found mentions in that regard in the changelog or doc

?????? flink ci build run longer than the maximum time of 310 minutes.

Hi,Matthias I have solved this problem as you say.The link to the PR [1] .thank you. [1]https://github.com/apache/flink/pull/20671 Best, Hjw   --  -- ??:

Where will the state be stored in the taskmanager when using rocksdbstatebend??

The EmbeddedRocksDBStateBackend holds in-flight data in a RocksDB database that is (per default) stored in the TaskManager local data directories. Which path does local data directories store RocksDB database in TaskManager point to in operating system? If the job state is very large, I think I

Re: Where will the state be stored in the taskmanager when using rocksdbstatebend？

https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/#state-backend-rocksdb-localdir Make sure to use a local SSD disk (not NFS/EBS). Best, Alexander Fedulov On Mon, Sep 5, 2022 at 7:24 PM hjw <1010445...@qq.com> wrote: > The EmbeddedRocksDBStateBackend holds in-flig

Re: Flink kafka producer using the same transactional id twice?

> > I then noticed this message showing up twice and thought "this does not > look right": That's fine, this is how the sink works (see the comment here: KafkaWriter.java#L294-L301

Re: Deploying Jobmanager on k8s as a Deployment

Hi! The operator supports both Flink native and standalone deployment modes and in both cases the JM is deployed as k8s Deployment. During upgrade Flink/operator deletes the deployment after savepoint and waits for termination before it creates a new one with the updated spec. Cheers, Gyula On

Out of memory in heap memory when working with state

Hi team,Currently, I was facing the OutOfMemoryError: Java heap space. This was some how due to the fact that I was storing the state on FileSystem. With the FsStateBackend, the working state for each task manager is in memory (on the JVM heap), and state backups (checkpoints) go to a distributed f

?????? Where will the state be stored in the taskmanager when using rocksdbstatebend??

Hi,Alexander When Flink job deployed on Native k8s, taskmanager is a Pod.The data directory size of a single container is limited in our company.Are there any idea to deal with this ? Best, Hjw   --  -- ??:

Re: Out of memory in heap memory when working with state

Hi, lan. I guess you are using the old version of flink. You could use RocksDBStateBackend[1] in the new version. It will put the state into disk when the state is large which could avoid using too much memory. BTW, In the current internal mechanism, the state on the external storage like s3 is jus