Re: Task manager errors with Flink ZooKeeper High Availability

2022-02-17 Thread Koffman, Noa (Nokia - IL/Kfar Sava)
? Is there any way to achieve this? Thanks Noa From: Chesnay Schepler Date: Thursday, 17 February 2022 at 15:00 To: Koffman, Noa (Nokia - IL/Kfar Sava) , Yun Gao , user Subject: Re: Task manager errors with Flink ZooKeeper High Availability Everything is fine. The TM tries to retrieve the jar

Re: Task manager errors with Flink ZooKeeper High Availability

2022-02-17 Thread Koffman, Noa (Nokia - IL/Kfar Sava)
, 17 February 2022 at 14:04 To: Koffman, Noa (Nokia - IL/Kfar Sava) , user Subject: Re: Task manager errors with Flink ZooKeeper High Availability Hi Koffman, >From TM side the only possible usage come to me is that or components like >BlobCache, which is used to transfer jars or larg

Task manager errors with Flink ZooKeeper High Availability

2022-02-16 Thread Koffman, Noa (Nokia - IL/Kfar Sava)
Hi, We are currently running flink in session deployment on k8s cluster, with 1 job-manager and 3 task-managers To support recovery from job-manager failure, following a different mail thread, We have enabled zookeeper high availability using a k8s Persistent Volume To achieve this, we’ve added

Re: Flink High-Availability and Job-Manager recovery

2022-02-14 Thread Koffman, Noa (Nokia - IL/Kfar Sava)
, is there a way to make zookeeper hold and manage the job states, instead of using the shared storage? Thank Noa From: bastien dine Date: Friday, 4 February 2022 at 10:56 To: Koffman, Noa (Nokia - IL/Kfar Sava) Cc: user@flink.apache.org Subject: Re: Flink High-Availability and Job-Manager

Flink High-Availability and Job-Manager recovery

2022-02-03 Thread Koffman, Noa (Nokia - IL/Kfar Sava)
Hi all, We are currently deploying flink on k8s 3 nodes cluster - with 1 job-manager and 3 task managers We are trying to understand the recommendation for deployment, more specifically for recovery from job-manager failure, and have some questions about that: 1. If we use flink HA

Remove stackTrace from error response

2021-12-27 Thread Koffman, Noa (Nokia - IL/Kfar Sava)
Hi, as part of security improvement we need to make sure APIs do not return stack traces as part of the error response, The JobManager REST API does currently return the stack trace on error, For example: ❯ curl http://localhost:8081/jobs/123

Re: pipeline are not started sporadically

2021-11-14 Thread Koffman, Noa (Nokia - IL/Kfar Sava)
Hi, thanks for the quick response, The pipelines we are running are: One pipeline doing simple conversions from read kafak and send to kafka Two pipelines that are working with DB (one write and one read) Two pipelines that are working with BigQuery.IO (both writing data) The ones we usually have