RE: Re: Looking for help with Job Initialisation issue

Hi Biao, Thank you for your response. We have tried looking into Thread dumps of Task Managers before but that's not helping our case. We see that even when all the Taskslots of that particular operator are stuck in an INITIALISING state, many of them have already started processing new data. Is th

Re: Flink scheduler keeps trying to schedule the pods indefinitely

Hey Gyula, Thanks for getting back. 1) Yes, some more testing revealed the job was able to start with lower parallelism i.e. lower than the upper bound that was set by the adaptive scheduler. 2) I am limiting the parallelism of any job-vertex by setting pipeline.max-parallelism to a value that ke

Operator/Autoscaler/Autotuner tuning behavior question

Hello. I have some questions about memory autotuning in the Operator. 1. Does the autotuner try to upgrade the job with more memory allocated if it intercepts OutOfMemoryError? Say I initially provided too little memory for TM `resource` - will the job fail and stop on initializing or will the

SQL failed to restore from savepoint after change in default-parallelism

Hello, After bumping 'table.exec.resource.default-parallelism' from 1 to 4, I am observing the following exception on restoring job from savepoint with an unmodified statement set. [ERROR] Could not execute SQL statement. Reason: > java.lang.IllegalStateException: Failed to rollback to > checkpoi

Re: Checkpointing

Hey Jacob, If you understand how the Kafka offset managed in the checkpoint, then you could map this notion to other Flink sources. I would suggest to read the Data Sources[1] document and FLIP-27[5]. Each source should define a `Split`, then it is `SourceReaderBase`[2] class' responsibility to

Re: Flink Kubernetes Operator Application mode multiple jobs

Got it. Thanks for the clarification guys. From: Guozhen Yang Sent: Wednesday, May 8, 2024 9:38 AM To: user@flink.apache.org Subject: RE: Flink Kubernetes Operator Application mode multiple jobs Hi Raihan, We have encountered the same issue though we are using F

Checkpointing

Hello, I'm curious about how Flink checkpointing would aid in recovering data if the data source is not Kafka but another system. I understand that checkpoint snapshots are taken at regular time intervals. What happens to the data that were read after the previous successful checkpoint if the sys

Re: Flink submit to cluster issue

Hi Phil, >>> Is it a library missing from the Flink cluster? What would cause that error? In most cases, it implies the collision of jar dependencies. flink-sql-connector-kafka jar should be enough to run the job and you may need to check your classpath in the jobmanager to see if any other jars t

Flink submit to cluster issue

I am running a Flink job locally using python -m job.py and it runs fine. The job is: calcul_count = t_env.execute_sql(""" SELECT username, COUNT(action) AS a_count FROM kafka_logs GROUP BY username """) with calcul_count.collect() as results: for row in results: print(row) When i try to s

Re: Incremental snapshot based source

Hi. cloud young. The property 'log.mining.strategy' is actually a setting from the oracle debezium connector[1]. Are some exceptions able to be found in your job and which version are you using? Best, Hang [1] https://debezium.io/documentation/reference/1.9/connectors/oracle.html#oracle-property

Incremental snapshot based source

I have read the doc ncremental-snapshot-based-datastream-experimental , and run the example code. It works. but I also need to get

[no subject]

Re: Re: Evolving Flink SQL statement set and restoring from savepoints

Hi Xuyang, Thank you for anticipating my questions and pointing me to the right resources. I've requested JIRA account, will create feature request once approved and then start discussion on dev mailing list. I am interested in contributing to this feature, would appreciate it if you can point m

Re: Exception in Flink 1.18 (Time should be non negative)

Hi. I also see it on my own machine too where I only run with one task manger. Med venlig hilsen / Best regardsLasse NedergaardDen 8. maj 2024 kl. 00.24 skrev Talat Uyarer :Hi Lasse,If there's a significant difference in the system time between Flink TaskManagers, it can lead to negative time calc