Re: Kubernetes operator listing jobs TimeoutException

2023-06-07 Thread Evgeniy Lyutikov
Hi, thanks for the reply. These errors occur on jobs that have already been successfully deployed and are running. When such an error occurs, the operator begins to consider that the job is in the DEPLOYING or DEPLOYED_NOT_READY status, but all this time the job is in the RUNNING state and no a

RE: Parquet decoding exception - Flink 1.16.x

2023-06-07 Thread Kamal Mittal via user
Hello, Can you please share view that for “file system sources”, how to create custom metrices e.g. no. of corrupt records count? Using Flink file source API as in below mail and decoding parquet formatted data. Able to count corrupt records but how to give it to Flink? Rgds, Kamal From: Kama

Re: Kubernetes operator listing jobs TimeoutException

2023-06-07 Thread Shammon FY
Hi Evgeniy, >From the following exception message: at org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:123) at org.apache.flink.runtime.rest.RestClient.submitRequest(RestClient.java:469) at org.apache.flink.runtime.rest.RestClient.sendRequ

Re: [ANNOUNCE] Apache flink-connector-pulsar v3.0.1 released

2023-06-07 Thread Neng Lu
Thank you very much for coordinating this! I think we also need to release 4.0.1 to fix the pulsar-client OAuth2 issue. On Wed, Jun 7, 2023 at 1:48 AM Leonard Xu wrote: > The Apache Flink community is very happy to announce the release of Apache > flink-connector-pulsar v3.0.1. > This release i

Kubernetes operator listing jobs TimeoutException

2023-06-07 Thread Evgeniy Lyutikov
Hello. We use Kubernetes operator 1.4.0, operator serves about 50 jobs, but sometimes there are errors in the logs that are reflected in the metrics (FlinkDeployment.JmDeploymentStatus.READY.Count). What is the reason for such errors? 2023-06-07 15:28:27,601 o.a.f.k.o.c.FlinkDeploymentControll

PyFlink Error JAR files

2023-06-07 Thread Kadiyala, Ruthvik via user
Hi, Please find below the code I have been using to consume a Kafka Stream that is hosted on confluent. It returns an error regarding the jar files. Please find the error below the code snippet. Let me know what I am doing wrong. I am running this on Docker with Flink Version: 1.7.1. Code: fr

RE: Parquet decoding exception - Flink 1.16.x

2023-06-07 Thread Kamal Mittal via user
Hello, Metrices link given in below mail doesn’t give any way to create metrices for source function right? I am using below Flink API to read/decode parquet data, query is where exception can be caught for error like “decoding exception” from internal parquet API like “AvroParquetReader” and

Re: Custom Counter on Flink File Source

2023-06-07 Thread Hang Ruan
Hi, Kirti. We could find these information in the 1.18 release wiki page[1]. Its timeline is as follows. Feature Freeze: July 11, 2023, end of business CEST Release: End of September 2023 Best, Hang [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release#id-1.18Release-Summary Kirti

Re: Raise alarm for corrupt records

2023-06-07 Thread Martijn Visser
Hi Kamal, Documentation on the metrics can be found at https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/ Best regards, Martijn On Wed, Jun 7, 2023 at 10:13 AM Kamal Mittal via user wrote: > Hello, > > > > Thanks for quick reply. > > > > I am using parquet encoder/decoder

[ANNOUNCE] Apache flink-connector-pulsar v3.0.1 released

2023-06-07 Thread Leonard Xu
The Apache Flink community is very happy to announce the release of Apache flink-connector-pulsar v3.0.1. This release is compatible with Flink 1.16.x series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data stream

RE: Raise alarm for corrupt records

2023-06-07 Thread Kamal Mittal via user
Hello, Thanks for quick reply. I am using parquet encoder/decoder and during decoding if any corrupt record comes then need to raise alarm and maintain metrices visible over Flink Metrices GUI. So any custom metrices can be created in Flink? Please give some reference of any such documentatio

RE: Custom Counter on Flink File Source

2023-06-07 Thread Kirti Dhar Upadhyay K via user
Thanks Hang. Any expected date for Flink 1.18.0 release? Regards, Kirti Dhar From: Hang Ruan Sent: 07 June 2023 07:34 To: Kirti Dhar Upadhyay K Cc: user@flink.apache.org Subject: Re: Custom Counter on Flink File Source Hi, Kirti Dhar Upadhyay K. I check the FLIP-274[1]. This issue will be rel

Re: Raise alarm for corrupt records

2023-06-07 Thread Martijn Visser
Hi Kamal, No, but it should be straightforward to create metrics or events for these types of situations and integrate them with your own alerting solution. Best regards, Martijn On Wed, Jun 7, 2023 at 8:25 AM Kamal Mittal via user wrote: > Hello Community, > > > > Is there any way Flink prov