Re: Question of Flink Operator Application Cluster Deployment

You don't have to mount the service account explicitly, this should be auto-mounted for you. Please share your (redacted) yamls for the RBAC configs ( https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-othe

Question of Flink Operator Application Cluster Deployment

Hi Flink Community, First of all, I would like to express my great thankfulness about the flink operator on Kubernetes. It is a new door to help us deploy the Flink application on top of the K8s. Our team is currently doing the Application cluster deployment through the operator. We have set up t

Re: Running in application mode on YARN without fat jar

The usrlib for YARN only works for 1.15.0 and later versions. Refer to the ticket[1] for more information. [1]. https://issues.apache.org/jira/browse/FLINK-24897 Best, Yang Pavel Penkov 于2022年5月16日周一 22:59写道： > I can't manage to run an application on YARN because of classpath issues. > Flink d

Checkpoint declined (Null Pointer exception)

Hello, I have the following situation: We upgraded our application code on a flink 1.11.1 cluster. We use rocksdb as the state backend. The upgrade used a savepoint from the prior app version. We added a few MapStates to an existing KeyedProcessWindowFunction. This function used to have a single va

Re: Kinesis Sink - Data being received with intermittent breaks

Hello Zain, When you say "converting them to chunks of <= 1MB " does this mean you are creating these chunks in a custom Flink operator, or you are relying on the connector to do so? If you are generating your own chunks you can potentially disable Aggregation at the sink. Your throughput is incr

Checkpoint directories not cleared as TaskManagers run

Hello, I'm seeing my Flink deployment's checkpoint storage directories build up and never clear down. When I run from my own IDE, I see the only the latest "chk-x" directory under the job id folder. So the first checkpoint is "chk-1", which is then replaced with "chk-2" etc. However, when

How to distribute DataStream rows parallelly to executors without duplication

Hi, I am reading from GCP bigquery table using dynamic flink connector and getting corresponding Flink Table. And for writing the processed data back to new BigQuery table, I am using : Table.executeInsert(tableSink) and table sink is created by extending AppendStreamTa

Re: Kinesis Sink - Data being received with intermittent breaks

Hi Zain, I'm looping in Danny here, he is probably the most knowledgeable when it comes to the Kinesis connector. Best, Alexander On Mon, May 16, 2022 at 12:13 AM Zain Haider Nemati wrote: > Hi, > Im fetching data from kafka topics converting them to chunks of <= 1MB and > sinking them to a ki

Running in application mode on YARN without fat jar

I can't manage to run an application on YARN because of classpath issues. Flink distribution is unpacked in $HOME/flink-1.14.4 $HOME/flink1.14.4/usrlib contains all the dependency jars excluding the main application jar as flat file structure. The application is started with ./bin/flink run-applic

Does kafka key is supported in kafka sink table

Hi dear engineer, Flink sql supports kafka sink table, not sure whether it supports kafka key in kafka sink table? As I want to specify kafka key when inserting data into kafka sink table. Thanks for your answer in advance. Thanks && Regards, Hunk

Re: Flink on Native K8s jobs turn in to `SUSPENDED` status unexpectedly.

It will help a lot if you could share the logs of JobManager and TaskManager for the unexpected `SUSPENDED` job. Best, Yang Xiaolong Wang 于2022年5月16日周一 13:30写道： > Sorry for the late reply. > > I checked the logs in both jobmanager & taskmanager. > > During that time, there were no more logs th

Re: taskexecutor .out files

Sorry, the command is parsed as reference. The real command is : " > taskmanager.out " Best, Weihua > 2022年5月16日 下午9:52，Weihua Hu 写道： > > Hi, > > Flink redirects stdout to the taskmanager.out when starting TaskManager. > If taskmanager.out is deleted, Flink cannot automatically create > t

Re: taskexecutor .out files

Hi, Flink redirects stdout to the taskmanager.out when starting TaskManager. If taskmanager.out is deleted, Flink cannot automatically create taskmanager.out, which means any subsequent output to stdout will be lost. If you want to clean up the content of taskmanager.out, you can try using:

Re: taskexecutor .out files

Hi, Zain: The taskmanager.out only contains contents outputted by stdout. Sometimes some fatal exceptions, like JVM exit exceptions and so on will be outputted to the .out file. If you don't specify the file path for the gc log, the content of the gc log will be saved into the .out file, too. Howe

Re: Community fork of Flink Scala API for Scala 2.12/2.13/3.x

Great work! Thank you for sharing. Am Do., 12. Mai 2022 um 17:19 Uhr schrieb Jeff Zhang : > That's true scala shell is removed from flink . Fortunately, Apache > Zeppelin has its own scala repl for Flink. So if Flink can support scala > 2.13, I am wondering whether it is possible to integrate it

What is the alternative schema of DirectOutput

Hi,     Could you do me a favor plz? I wonder if the DirectOutput class which works in version 1.11 is removed in version 1.15. If so, Could you tell me why and what is the alternative solution?    Thank you!