请问有没有公司可以提供开源Flink维保服务？

如题

Re: Flink Kubernetes Operator 1.8.0 CRDs

Sorry, let me explain. I currently have the operator deployed and managed via ArgoCD. The CRDs I separated out into a different chart so I can do upgrades on them. I am working on upgrading from version 1.7.0 to 1.8.0 using ArgoCD. What I’ve done is replace the CRDs in the separate chart and

Re: Flink Kubernetes Operator 1.8.0 CRDs

Hey! We have not observed any issue so far, can you please share some error information / log ? Opening a jira ticket would be best Thanks Gyula On Thu, 9 May 2024 at 21:18, Prasad, Neil wrote: > I am writing to report an issue with the Flink Kubernetes Operator version > 1.8.0. The CRD is

Re: Flink Kubernetes Operator 1.8.0 CRDs

Hi, What do you mean exactly by cannot be applied or replaced? What exactly is the issue? Are you installing fresh or trying to upgrade from a previous version? If the latter please follow this:

Flink Kubernetes Operator 1.8.0 CRDs

I am writing to report an issue with the Flink Kubernetes Operator version 1.8.0. The CRD is unable to be applied or replaced in minikube or GKE. However, the CRD works on version 1.7.0 of the operator. I thought it would be helpful to bring this issue to the attention of the community and get

Flink Kubernetes Operator - How can I use a jar that is hosted on a private maven repo for a FlinkSessionJob?

Hello, I am trying to run a Flink Session Job with a jar that is hosted on a maven repository in Google's Artifact Registry. The first thing I tried was to just specify the `jarURI` directly: apiVersion: flink.apache.org/v1beta1 kind: FlinkSessionJob metadata: name: myJobName spec:

Saw a java.lang.ClassNotFoundException: com.facebook.presto.hive.s3.PrestoS3FileSystem$UnrecoverableS3OperationException

Hi, We use S3 as our datastore for checkpoint/savepoints, and following an S3 error we saw that exception: ``` java.io.IOException: GET operation failed: Could not transfer error message at org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:231) at

Re: Apache Flink-Redis Connector Depreciated In New Version | Adsolut Media

Hi Kush Unfortunately there is currently no real Redis connector maintained by the Flink community. I am aware that Bahir's version might be outdated but we are currently working on a community supported connector[1] 1-https://github.com/apache/flink-connector-redis-streams Best Regards Ahmed

Re: 退订

Hi, 退订请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org . Best, yunfeng On Thu, May 9, 2024 at 5:58 PM xpfei0811 wrote: > > 退订 > > 回复的原邮件 > | 发件人 | wangfengyang | > | 发送日期 | 2024年04月23日 18:10 | > | 收件人 | user-zh | > | 主题 | 退订 | > 退订

??????????

| ?? | wangfengyang | | | 2024??04??23?? 18:10 | | ?? | user-zh | | | |

回复：退订

退订 回复的原邮件 | 发件人 | jh...@163.com | | 发送日期 | 2024年04月20日 22:01 | | 收件人 | user-zh | | 主题 | Re: 退订 | | | jhg22 | | jh...@163.com | jh...@163.com 发件人： 冮雪程 发送时间： 2024-04-19 18:01 收件人： user-zh@flink.apache.org 主题： 退订 | | 冮雪程 | | gxc_bigd...@163.com | 回复的原邮件 | 发件人 |

use flink 1.19 JDBC Driver can find jdbc connector

I put flink-connector-jdbc into flink\lib.use flink 1.19 JDBC Driver can not find jdbc connector,but use sql-client is normal.

How to perform stateless restarts with checkpointing enabled?

We have a source/sink mechanism which uses checkpoints for persistence and can operate in a minor data loss scenario. Is there a method to use checkpoints (to enable use of those source/sink operators) while disabling stateful recovery during restarts? Our setup uses Flink 1.16.1 alongside Flink

Re: Re: Looking for help with Job Initialisation issue

Hi Abhi, > We see that even when all the Taskslots of that particular operator are stuck in an INITIALISING state Can you include the stack trace of these threads so that we can understand what the operators are stuck on INITIALISING? Regards Keith On Thu, May 9, 2024 at 6:58 AM Abhi Sagar

RE: Re: Looking for help with Job Initialisation issue

Hi Biao, Thank you for your response. We have tried looking into Thread dumps of Task Managers before but that's not helping our case. We see that even when all the Taskslots of that particular operator are stuck in an INITIALISING state, many of them have already started processing new data. Is

Re: Flink scheduler keeps trying to schedule the pods indefinitely

Hey Gyula, Thanks for getting back. 1) Yes, some more testing revealed the job was able to start with lower parallelism i.e. lower than the upper bound that was set by the adaptive scheduler. 2) I am limiting the parallelism of any job-vertex by setting pipeline.max-parallelism to a value that

Operator/Autoscaler/Autotuner tuning behavior question

Hello. I have some questions about memory autotuning in the Operator. 1. Does the autotuner try to upgrade the job with more memory allocated if it intercepts OutOfMemoryError? Say I initially provided too little memory for TM `resource` - will the job fail and stop on initializing or will the

SQL failed to restore from savepoint after change in default-parallelism

Hello, After bumping 'table.exec.resource.default-parallelism' from 1 to 4, I am observing the following exception on restoring job from savepoint with an unmodified statement set. [ERROR] Could not execute SQL statement. Reason: > java.lang.IllegalStateException: Failed to rollback to >

Re: Checkpointing

Hey Jacob, If you understand how the Kafka offset managed in the checkpoint, then you could map this notion to other Flink sources. I would suggest to read the Data Sources[1] document and FLIP-27[5]. Each source should define a `Split`, then it is `SourceReaderBase`[2] class' responsibility to

Re: Flink Kubernetes Operator Application mode multiple jobs

Got it. Thanks for the clarification guys. From: Guozhen Yang Sent: Wednesday, May 8, 2024 9:38 AM To: user@flink.apache.org Subject: RE: Flink Kubernetes Operator Application mode multiple jobs Hi Raihan, We have encountered the same issue though we are using

Checkpointing

Hello, I'm curious about how Flink checkpointing would aid in recovering data if the data source is not Kafka but another system. I understand that checkpoint snapshots are taken at regular time intervals. What happens to the data that were read after the previous successful checkpoint if the

Re: Flink submit to cluster issue

Hi Phil, >>> Is it a library missing from the Flink cluster? What would cause that error? In most cases, it implies the collision of jar dependencies. flink-sql-connector-kafka jar should be enough to run the job and you may need to check your classpath in the jobmanager to see if any other jars

Flink submit to cluster issue

I am running a Flink job locally using python -m job.py and it runs fine. The job is: calcul_count = t_env.execute_sql(""" SELECT username, COUNT(action) AS a_count FROM kafka_logs GROUP BY username """) with calcul_count.collect() as results: for row in results: print(row) When i try to

Re: Incremental snapshot based source

Hi. cloud young. The property 'log.mining.strategy' is actually a setting from the oracle debezium connector[1]. Are some exceptions able to be found in your job and which version are you using? Best, Hang [1]

Incremental snapshot based source

I have read the doc ncremental-snapshot-based-datastream-experimental , and run the example code. It works. but I also need to get

[no subject]

Re: Re: Evolving Flink SQL statement set and restoring from savepoints

Hi Xuyang, Thank you for anticipating my questions and pointing me to the right resources. I've requested JIRA account, will create feature request once approved and then start discussion on dev mailing list. I am interested in contributing to this feature, would appreciate it if you can point

Re: Exception in Flink 1.18 (Time should be non negative)

Hi. I also see it on my own machine too where I only run with one task manger. Med venlig hilsen / Best regardsLasse NedergaardDen 8. maj 2024 kl. 00.24 skrev Talat Uyarer :Hi Lasse,If there's a significant difference in the system time between Flink TaskManagers, it can lead to negative time

RE: Flink Kubernetes Operator Application mode multiple jobs

Hi Raihan, We have encountered the same issue though we are using Flink Kubernetes Operator 1.6. Biao Geng's explanation is correct. We also have a detailed briefing in this jira ticket but it seems that submitting multiple jobs is not allowed

Re:Is there open requests to support Redis Datastream connector

Hi, Tony. There are some related works in Flink commu to support Redis source / sink connector. You can see more here[1][2][3][4]. However, these JIRAs are still WIP. The good news is that the PR in jira[1] is already available. You can directly download the PR and package it into a

Re:Re: Evolving Flink SQL statement set and restoring from savepoints

Hi, if the processing logic is modified, then the representation of the topology would change. Consequently, the UIDs that are determined by the topological order might change as well, which could potentially cause state recovery to fail. For further details, you can refer to [1]. Currently,

Is there open requests to support Redis Datastream connector

Hi team, I need a Redis sink connector for my Flink app but the best I can find is from Bahir, which is deprecated. I am wondering if someone in the community is

Re: Issue in PrefetchCount

Hi ajay, When you have 3 parallelisms you will have 3 independent clients. If you want to keep prefetch count 3 you need to set setRequestedChannelMax as 1 and setParallelism 3. So All 3 clients can have one connection. Talat On Tue, May 7, 2024 at 5:52 AM ajay pandey wrote: > Hi Flink Team,

Re: Exception in Flink 1.18 (Time should be non negative)

Hi Lasse, If there's a significant difference in the system time between Flink TaskManagers, it can lead to negative time calculations when comparing timestamps from different sources. On Mon, May 6, 2024 at 5:40 AM Lasse Nedergaard < lassenedergaardfl...@gmail.com> wrote: > Hi. > > In Flink

Re: Evolving Flink SQL statement set and restoring from savepoints

Hi Keith, When you add a new insert statement to your EXECUTE STATEMENT you change your job graph with independent two graphs.Unfortunately, Flink doesn't currently provide a way to directly force specific UIDs for operators through configuration or SQL hints. This is primarily due to how Flink's

Re: Flink Kubernetes Operator Application mode multiple jobs

Hi Raihan, I believe the flink k8s operator has some implicit assumption that each application should contain only one job so that it can manage the lifecycle of the job reasonably. Your usage of deploying multiple jobs in a single python file(i.e. in a single flink application) may be not

RE: [EXTERNAL] Re: Coordinator of operator ... does not exist or the job vertex this operator belongs to is not initialized.

Hi Biao, I figured that the error happens only when there is a JOIN in the select. But I will put together a simple example. Thank you, Ed Skhisov Architect | www.intradiem.com 303.588.2518 Mailing Address: 2500 Dallas Hwy Ste 202, Dept #37049 Marietta, GA 30064

Evolving Flink SQL statement set and restoring from savepoints

Hello, I'm running into issues restoring from savepoint after changing SQL statement. [ERROR] Could not execute SQL statement. Reason: > java.lang.IllegalStateException: Failed to rollback to > checkpoint/savepoint > file:/tmp/flink-savepoints/savepoint-52c344-3bedb8204ff0. Cannot map >

Flink Kubernetes Operator Application mode multiple jobs

Hi everyone, I need some help with deploying multiple jobs from a single main function in Application mode using Flink Kubernetes Operator. As per the documentation [1] it should be possible to use multiple "executeAsync()" to deploy multiple jobs from the same file. This indeed is the case

Re: Re: java.io.IOException: Could not load the native RocksDB library

或许您可以尝试参考下[1] 再验证下加载的问题。 BTW，目前看起来是有些依赖库找不到，librocksdbjni-win64.dll 当时是基于 VS2022 编译出来的，您也尝试下在本地安装下VS2022后重试。 [1] https://github.com/facebook/rocksdb/issues/2531#issuecomment-313209314 ha.fen...@aisino.com 于2024年5月7日周二 10:22写道： > > idea工具,win10操作系统 > java.io.IOException: Could not load the

Re: Coordinator of operator ... does not exist or the job vertex this operator belongs to is not initialized.

Hi Ed, Would you mind giving a minimal example to reproduce your case? I tried a pretty simple case like this in a mini cluster: ``` tEnv.createTemporaryView("test", env.fromData(1, 2, 3)); Table table = tEnv.sqlQuery("SELECT * FROM test"); table.execute().collect().next();

Re: java.io.IOException: Could not load the native RocksDB library

请问是什么开发环境呢？ windows吗？ 可以分享一下更详细的报错吗？比如.dll 找不到 ha.fen...@aisino.com 于2024年5月7日周二 09:34写道： > > Configuration config = new Configuration(); > config.set(StateBackendOptions.STATE_BACKEND, "rocksdb"); > config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem"); >

Exception in Flink 1.18 (Time should be non negative)

Hi. In Flink jobs running 1.18 I see the error below sometimes. I Can see the same problem has been reported and fixed I 1.14 Anyone have an idea how to debug and/or work around this problem? java.lang.IllegalArgumentException: Time should be non negative at

Re: Flink scheduler keeps trying to schedule the pods indefinitely

Hey! Let me first answer your questions then provide some actual solution hopefully :) 1. The adaptive scheduler would not reduce the vertex desired parallelism in this case but it should allow the job to start depending on the lower/upper bound resource config. There have been some changes in

Re: Looking for help with Job Initialisation issue

Hi Abhi, If your case can be reproduced steadily, have your ever tried to get the thread dump of the TM which the problematic operator resides in? Maybe we can get more clues with the thread dump to see where the operator is getting stuck. Best, Biao Geng Abhi Sagar Khatri via user

Flink scheduler keeps trying to schedule the pods indefinitely

Hello, I am running a flink job in the application mode on k8s. It's deployed as a FlinkDeployment and its life-cycle is managed by the flink-k8s-operator. The autoscaler is being used with the following config job.autoscaler.enabled: true job.autoscaler.metrics.window: 5m

Re: Coordinator of operator ... does not exist or the job vertex this operator belongs to is not initialized.

Hi Eduard, You may need to set log level = INFO to see if there are any other error messages generated in the JM or TM's log. The current exception message seems to be a result error generated from the JM, but the causing error message should still be lying somewhere in the TM's log. Best

Re: Autoscaling with flink-k8s-operator 1.8.0

Hi Gyula, Thanks for getting back and explaining the difference in the responsibilities of the autoscaler and the operator. I figured out what the issue was. Here is what I was trying to do: the autoscaler had initially down-scaled (2->1) the flinkDeployment so there was

Coordinator of operator ... does not exist or the job vertex this operator belongs to is not initialized.

Update: I figured out that error happens only of the SQL contains JOIN of any kind. If there are no JOINs, everything works fine. Any help? Hello, I am trying to use CloseableIterator, but next() operation reliably generates the following error: java.util.concurrent.ExecutionException:

Re: Autoscaling with flink-k8s-operator 1.8.0

Hi Chetas, The operator logic itself would normally call the rescale api during the upgrade process, not the autoscaler module. The autoscaler module sets the correct config with the parallelism overrides, and then the operator performs the regular upgrade cycle (as when you yourself change

Autoscaling with flink-k8s-operator 1.8.0

Hello, We recently upgraded the operator to 1.8.0 to leverage the new autoscaling features ( https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.8/docs/custom-resource/autoscaler/). The FlinkDeployment (application cluster) is set to flink v1_18 as well. I am able to

Re: [Flink Kubernetes Operator] The "last-state" upgrade mode is only supported in FlinkDeployments

Thanks for answering my questions, Gyula! And your insights are very helpful. Let me take a deeper look at the existing logic and think more. On Tue, Apr 30, 2024 at 12:00 PM Gyula Fóra wrote: > The application mode indeed has a sticky jobId (at least when we are > performing a last-state

Coordinator of operator ... does not exist or the job vertex this operator belongs to is not initialized.

Hello, I am trying to use CloseableIterator, but next() operation reliably generates the following error: java.util.concurrent.ExecutionException: org.apache.flink.util.FlinkException: Coordinator of operator 4596fb32cad14208ec80c1cae8623e11 does not exist or the job vertex this operator

Re: [Flink Kubernetes Operator] The "last-state" upgrade mode is only supported in FlinkDeployments

The application mode indeed has a sticky jobId (at least when we are performing a last-state upgrade, otherwise a new jobId is generated during stateless deployments). But that's only part of the story and arguably the less important bit. The last-state upgrade mechanism for running/failing (but

Re: [Flink Kubernetes Operator] The "last-state" upgrade mode is only supported in FlinkDeployments

Hi Gyula, Thanks for your reply! Good suggestion on JIRA ticket, I created a JIRA ticket for tracking it: https://issues.apache.org/jira/browse/FLINK-35279. We could be interested in working on it because of our own requirement, I will check you and the community again once we have some updates.

Re: Looking for help with Job Initialisation issue

Some more context: Our job graph has 5 different Tasks/operators/flink functions of which we are seeing this issue every time in a particular operator We’re using Unaligned checkpoints. With aligned checkpoint we don’t see this issue but the checkpoint duration in that case is very high and causes

Looking for help with Job Initialisation issue

Hi Flink folks, Our team has been working on a Flink service. After completing the service development, we moved on to the Job Stabilisation exercises at the production load. During high load, we see that if the job restarts (mostly due to the "org.apache.flink.util.FlinkExpectedException: The

Re: Flink sql retract to append

以处理时间为升序，处理结果肯定不会出现回撤的，因为往后的时间不会比当前时间小了，你可以在试试这个去重 On Tue, Apr 30, 2024 at 3:35 PM 焦童 wrote: > 谢谢你的建议 但是top-1也会产生回撤信息 > > > 2024年4月30日 15:27，ha.fen...@aisino.com 写道： > > > > 可以参考这个 > > > https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/dev/table/sql/queries/deduplication/ > >

Re:Re: 在idea中用CliFrontend提交job 报错 java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;

您好，根据您的指点，配置之后，重启了StandaloneSessionClusterEntrypoint和TaskManagerrunner，这个问题解决了，谢谢您 在 2024-04-30 15:45:18，"Biao Geng" 写道： >Hi, > >这个报错一般是JDK版本不一致导致的。建议统一build flink和执行flink作业时的Java版本，（都用JDK8 或者 都用JDK11）。 >用JDK11时没有sun.misc的问题可以试试勾选掉Idea的Settings-> Build, Execution and Deployment

Re:Re: 在idea中用CliFrontend提交job 报错 java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;

您好，按照您的指点，现在运行和编译都用的jdk11，已经没有sun.misc的报错了，但是提交job 还是报相同的错误 D:\software\jdk-11.0.7\bin\java.exe -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:11039,suspend=y,server=n -Dlog.file=./log/flink-client.log -Dlog4j.configuration=./conf/log4j-cli.properties

Re: 在idea中用CliFrontend提交job 报错 java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;

Hi, 这个报错一般是JDK版本不一致导致的。建议统一build flink和执行flink作业时的Java版本，（都用JDK8 或者 都用JDK11）。 用JDK11时没有sun.misc的问题可以试试勾选掉Idea的Settings-> Build, Execution and Deployment -> Compiler-> Java Compiler的Use '--release' option for cross-compilation' 选项。 Best, Biao Geng z_mmG <13520871...@163.com> 于2024年4月30日周二

Re: Flink sql retract to append

谢谢你的建议 但是top-1也会产生回撤信息 > 2024年4月30日 15:27，ha.fen...@aisino.com 写道： > > 可以参考这个 > https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/dev/table/sql/queries/deduplication/ > 1.11版本不知道是不是支持 > > From: 焦童 > Date: 2024-04-30 11:25 > To: user-zh > Subject: Flink sql retract to append

在idea中用CliFrontend提交job 报错 java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;

JDK11 编译的flink1.19的源码 因为他说没有sun.misc，所以启动用的jdk8 已连接到地址为 ''127.0.0.1:8339'，传输: '套接字'' 的目标虚拟机 Job has been submitted with JobID 0975ec264edfd11d236dd190e7708d70 The program finished with the following exception:

Re: [Flink Kubernetes Operator] The "last-state" upgrade mode is only supported in FlinkDeployments

Hi Alan! I think it should be possible to address this gap for most cases. We don't have the same robust way of getting the last-state information for session jobs as we do for applications, so it will be slightly less reliable overall. For session jobs the last checkpoint info has to be queried

Flink sql retract to append

Hello , 我使用Flink 1.11 版本 sql 进行数据去重（通过 group by 形式）但是这会产生回撤流，下游存储不支持回撤流信息仅支持append，在DataStream 中我可以通过状态进行去重，但是在sql中如何做到去重且不产生回撤流呢。谢谢各位

[Flink Kubernetes Operator] The "last-state" upgrade mode is only supported in FlinkDeployments

Hi, We wanted to use the Apache Flink Kubernetes operator to manage the lifecycle of our Flink jobs in Flink session clusters. And we wanted to have the "last-state" upgrade feature for our use cases. However, the latest official doc states the "last-state" upgrade mode is not supported in the

Re: Suggestions for aggregating records to a Kinesis Sink, (or generic Async Sink)?

Hi Ahmed, hi Hong, Thanks for your responses. It sounds like the most promising would be to initially focus on the Global Window with the custom trigger. We don't need to be compatible with the aggregation used by the KPL (actually we would likely combine records in protobuf, and my impression

Re: Suggestions for aggregating records to a Kinesis Sink, (or generic Async Sink)?

Hi Michael, Unfortunately the new `KinesisDataStreamsSink` doesn't support aggregation yet. My suggestion if you want to use native kinesis aggregation is to use the latest connector version that supports KPL as sink for Table API, that would be 1.14.x. you could package the connector of that

Re: Flink SQL checkpoint failed when running on yarn

Hi there, Would you mind sharing the whole JM/TM log? It looks like the error log in the previous email is not the root cause. Best, Biao Geng ou...@139.com 于2024年4月29日周一 16:07写道： > Hi all: >When I ran flink sql datagen source and wrote to jdbc, checkpoint kept > failing with the

Re: Flink SQL Client does not start job with savepoint

Thanks Biao Geng for your response. Indeed, 1.19 documentation uses execution.savepoint.path, restoration works with said configuration name. https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sqlclient/#execute-sql-files Regards Keith From: Biao Geng Date: Friday, 26

Flink SQL checkpoint failed when running on yarn

Hi all: When I ran flink sql datagen source and wrote to jdbc, checkpoint kept failing with the following error log. 2024-04-29 15:46:25,270 ERROR org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler [] - Unhandled exception.

Suggestions for aggregating records to a Kinesis Sink, (or generic Async Sink)?

Hi all, We are currently using Flink 1.18.1 (AWS Managed Flink) and are writing to Kinesis streams in several of our applications using the Table API. In our use case, we would like to be able to aggregate multiple records (rows) together and emit them in a single Kinesis record. As far as I

Re: Strange Problem (0 AvailableTask)

Alright, THanks so much Biao ... On Sun, Apr 28, 2024 at 9:45 AM Biao Geng wrote: > Hi Hemi, > Glad to hear that your problem is solved! > As for deploying a flink cluster, you can check these docs for > more information based on your resource provider: > YARN: >

Re: CSV format and hdfs

Hi Artem, I research on this and open a issue[1] , Rob Young , Alexander Fedulov and I discuss on this. We also think this performance issue can be solved by manual flush. I had opened a pr[2]. You can cherry pick and package on your local, replace the jar in lib folder. I'm willing to hear

Re: Strange Problem (0 AvailableTask)

Hi Hemi, Glad to hear that your problem is solved! As for deploying a flink cluster, you can check these docs for more information based on your resource provider: YARN: https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/ K8s:

Re: [External] Regarding java.lang.IllegalStateException

My guess it’s a major known issue. Need a workaround. https://issues.apache.org/jira/browse/FLINK-32212 /Maxim From: prashant parbhane Date: Tuesday, April 23, 2024 at 11:09 PM To: user@flink.apache.org Subject: [External] Regarding java.lang.IllegalStateException Hello, We have been facing

Re: [External] Exception during autoscaling operation - Flink 1.18/Operator 1.8.0

I was talking about Flink Kubernetes operator and HA , not the individual Flink jobs. But based on your answer it’s probably not the cause Gyula On Fri, 26 Apr 2024 at 21:15, Maxim Senin wrote: > Hi, Gyula. Thanks for the tips. > > All jobs are deployed in a single namespace, “flink”. > >

Re: [External] Exception during autoscaling operation - Flink 1.18/Operator 1.8.0

Here’s more detail on the UnsupportedOperation exception. The job starts, operator collects some stats and then the job dies, apparently on rescaling op: [INFO ][flink/f-d7681d0f-c093-5d8a-b5f5-2b66b4547bf6] >>> Event | Info| JOBSTATUSCHANGED | Job status changed from CREATED to RUNNING

Re: Regarding java.lang.IllegalStateException

We are also seeing something similar: 2024-04-26 16:30:44,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Power Consumption:power_consumption -> Ingest Power Consumption -> PopSysFields -> WindowingWatermarkPreCheck (1/1)

Re: [External] Exception during autoscaling operation - Flink 1.18/Operator 1.8.0

Hi, Gyula. Thanks for the tips. All jobs are deployed in a single namespace, “flink”. Which replicas? The JM replicas are already 1, I tried with TM replicas set to 1, but same exception happens. We have only 1 instance of the operator (replicas=1) in this environment. The only workarounds I

Re: Strange Problem (0 AvailableTask)

Hi Biao, Thanks for your reply, fortunately the problem is solved. All I did was changed the bind-host to 0.0.0.0 (previously it was set to the server's IP). I don't know if it's best practice or not but everything is working fine now. RIght now I am using flink as standalone (I have the

Re: Flink SQL Client does not start job with savepoint

Hi Lee, A quick question: what version of flink are you using for testing execution.state-recovery.path? It looks like this config is only supported in flink 1.20 which is not released yet. Best, Biao Geng Lee, Keith 于2024年4月26日周五 04:51写道：

Re: Strange Problem (0 AvailableTask)

Hi Hemi, How do you start your flink cluster? Are you using standalone cluster or using k8s/yarn as resource providers? Also, it would be very helpful if you can share the full jobmanager log. Best, Biao Geng Hemi Grs 于2024年4月18日周四 15:43写道： > Hello, > > I have several versions of Flink

Re: [External] Exception during autoscaling operation - Flink 1.18/Operator 1.8.0

Hi Maxim! Regarding the status update error, it could be related to a problem that we have discovered recently with the Flink Operator HA. Where during a namespace change both leader and follower instances would start processing. It has been fixed in the current master by updating the JOSDK

Re: Async code inside Flink Sink

Hi Jacob, For your first question, I think it is fine to use Java completableFuture for your case. If we create lots of threads, of course it would consume more CPU and influent the processing of records. But in your case, the close op may not be very heavy. One thing comes to mind is that when

Re: [External] Exception during autoscaling operation - Flink 1.18/Operator 1.8.0

I have also seen this exception: o.a.f.k.o.o.JobStatusObserver [ERROR][flink/f-d7681d0f-c093-5d8a-b5f5-2b66b4547bf6] Job d0ac9da5959d8cc9a82645eeef6751a5 failed with error: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException:

Re: CSV format and hdfs

Hi Artem, I had a debug of Flink 1.17.1 (running CsvFilesystemBatchITCase) and I see the same behaviour. It's the same on master too. Jackson flushes [1] the underlying stream after every `writeValue` call. I experimented with disabling the flush by disabling Jackson's FLUSH_PASSED_TO_STREAM [2]

Re: Flink SQL Client does not start job with savepoint

Apologies, I have included the jobmanager log for 6969725a69ecc967aac2ce3eedcc274a instead of 7881d53d28751f9bbbd3581976d9fe3d, however they looked exactly the same. Can include if necessary. Thanks Keith From: "Lee, Keith" Date: Thursday, 25 April 2024 at 21:41 To: "user@flink.apache.org"

Flink SQL Client does not start job with savepoint

Hi, Referring to https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#start-a-sql-job-from-a-savepoint I’ve followed the instruction however I do not see evidence of the job being started with savepoint. See SQL statements excerpt below: Flink SQL> STOP JOB

Exception during autoscaling operation - Flink 1.18/Operator 1.8.0

Hi. I already asked before but never got an answer. My observation is that the operator, after collecting some stats, is trying to restart one of the deployments. This includes taking a savepoint (`takeSavepointOnUpgrade: true`, `upgradeMode: savepoint`) and “gracefully” shutting down the

Best serialization performance for `Objects`

I know that an `Object` is treated as a generic data type by Flink and hence serialized using Kryo. I wonder if there is anything one can do to improve performance w.r.t. to the Kryo-based serializer or if that is simply an inherent worst case scenario and nothing can be done without actually

Re:Flink 截止到1.18，是否有办法在Table API上添加uid？

Hi, 如果在中间添加了op，或者修改了处理逻辑，那么代表拓扑图会变，那么基于拓扑序所确定的uid也会变，从状态恢复就可能失败。具体可以参考[1] 目前table api应该是没有开放自定义uid的能力，可以在jira[2]上新建一个feature的jira，然后在dev邮件里发起讨论下。 [1]

Flink 截止到1.18，是否有办法在Table API上添加uid？

Hi Team， 我们这边的业务使用 Flink MySQL CDC到 OpenSearch并且使用TABLE API： INSERT INTO t1 SELECT * FROM t2 这种方式。 由于我们这边可能会在运行过程中添加额外的Operator，我们有办法在使用snapshot 恢复后保留之前src和sink operator的状态么？我看到在DataStream API可以通过设定uid。Table API有同样的方法吗？我看到Flink jira：https://issues.apache.org/jira/browse/FLINK-28861

PyFlink Performance

Hi, I am currently evaluating PyFlink in comparison to Java and did some various tests, mainly comparing identical pipelines with focus on throughput. For me it seems, that PyFlink is generally worse for wear and seems to reach its limits in throughput at a point where Java still has resources

Regarding java.lang.IllegalStateException

Hello, We have been facing this weird issue, where we are getting below exception and the job is getting restarted with new task managers. We are using flink 1.17. Same job works fine with lower number of task managers. (<10) java.lang.IllegalStateException: The library registration references a

Re: Flink 1.18: Unable to resume from a savepoint with error InvalidPidMappingException

Hi JM, > why having "transactional.id.expiration.ms" < "transaction.timeout.ms" helps When recover a job from a checkpoint/savepoint which contains Kafka transactions, Flink will try to re-commit those transactions based on transaction ID upon recovery. If those transactions timeout or

Re:处理时间的滚动窗口提前触发

Hi, 我看你使用了System.currentTimeMillis()，有可能是分布式的情况下，多台TM上的机器时间不一致导致的吗？ -- Best！ Xuyang 在 2024-04-20 19:04:14，"hhq" <424028...@qq.com.INVALID> 写道： >我使用了一个基于处理时间的滚动窗口，窗口大小设置为60s，但是我在窗口的处理函数中比较窗口的结束时间和系统时间，偶尔会发现获取到的系统时间早于窗口结束时间（这里的提前量不大，只有几毫秒，但是我不清楚，这是flink窗口本身的原因还是我代码的问题）我没有找到原因，请求帮助

Re: FlinkCEP

Hi, As Zhongqiang said, the CEP API is stable. Besides that, there are some changes worth mentioning： 1. https://issues.apache.org/jira/browse/FLINK-23890 Since flink 1.16.0, the timer creation is optimized which can incredibly reduce the resource usage of cep operator given same workload. 2.

Re: FlinkCEP

Hi, After flink 1.5 , there have been no major changes to the CEP API. Best, Zhongqiang Gong Esa Heikkinen 于2024年4月23日周二 04:19写道： > Hi > > It's been over 5 years since I last did anything with FlinkCEP and Flink. > > Has there been any significant development in FlinkCEP during this time? > >

Re: Why RocksDB metrics cache-usage is larger than cache-capacity

Sorry, it was probably an observation mistake. I export the metrics to Prometheus and query the result on grafana, actually the usage will not exceed the capacity Thanks, Lei On Fri, Apr 19, 2024 at 9:55 AM Hangxiang Yu wrote: > Hi, Lei. > It's indeed a bit confusing. Could you share the

<    1   2   3   4   5   6   7   8   9   10   >