Re: Taskslots usage

Hello Alexandre, Flink does not use TaskSlot per each task by default, but rather task slot will hold a slice of the entire pipeline (up to 1 subtasks of each operator, depending on the operator parallelism) [1]. So if your job parallelism is 1 - only a single task slot will be occupied. If you

Re: Taskslots usage

Hi Ky Alexandre, I would recommend reading this section which explains slot sharing b/w tasks. Link Quote - By default, Flink allows subtasks to share slots even if they are >

Re: Flink reactive deployment on with kubernetes operator

Thanks,nacisimsek.I will try your suggestion. --Original-- From: "nacisimsek"

?????? Flink reactive deployment on with kubernetes operator

Thanks,Gyula.I agree with you on Autoscaler,and I will try the latest Flink Operator version. ---- ??: "Gyula F??ra"

flink-runtime:1.14.6????????????

flinkflink-runtime:1.14.6 2024-07-10 16:48:09.700 WARN [XNIO-1 task-8-SendThread(102.195.8.107:2181)] org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [1164] Session 0x0 for server 102.195.8.107/102.195.8.107:2181, unexpected

Re: Can we use custom serialization/deserialization for kafka sources and sinks through the table api?

Hi Gabriel, You could consider overriding the value.serializer and value.deserializer (and similar for key) in the consumer and producer

Can we use custom serialization/deserialization for kafka sources and sinks through the table api?

Reading from a kafka topic with custom serialization/deserialization can be done using a KafkaSource configured with an implementation of KafkaRecordDeserializationSchema, which has access even to kafka headers which are used in my case for checking message integrity. How can we do the same but

Trying to read a file from S3 with flink on kubernetes

Hey guys. I'm trying to read a file from an internal S3 with flink on Kubernetes, but get a strange blocking error. Here is the code : MyFlinkJob.scala : ### package com.example.flink import org.apache.flink.api.common.serialization.SimpleStringSchema import

Re: Flink reactive deployment on with kubernetes operator

Hi Enric, You can try using persistent volume claim on your kubernetes cluster as a JobResultStore, instead of using a local path from your underlying host, and see if it works. apiVersion: v1 kind: PersistentVolumeClaim metadata: name: flink-data-pvc spec: resources: requests:

Kubernetes HA checkpoint not retained on termination

hi, I have a problem that Flink deletes checkpoint information on kubernetes HA setup even if execution.checkpointing.externalized-checkpoint-retention=RETAIN_ON_CANCELLATION is set. config documentation: "RETAIN_ON_CANCELLATION": Checkpoint state is kept when the owning job is cancelled or fails.

回复：Flink在HA模式，重启ZK集群，客户端任务提交异常

退订 回复的原邮件 | 发件人 | love_h1...@126.com | | 日期 | 2024年07月11日 16:10 | | 收件人 | user-zh@flink.apache.org | | 抄送至 | | | 主题 | Flink在HA模式，重启ZK集群，客户端任务提交异常 | 问题现象： Flink 1.11.6版本，Standalone HA模式， 滚动重启了ZK集群；在Flink集群的一个节点上使用flink run 命令提交多个任务； 部分任务提交失败，异常信息如下：

Flink在HA模式，重启ZK集群，客户端任务提交异常

问题现象： Flink 1.11.6版本，Standalone HA模式， 滚动重启了ZK集群；在Flink集群的一个节点上使用flink run 命令提交多个任务； 部分任务提交失败，异常信息如下： [Flink-DispatcherRestEndpoint-thread-2] - [WARN ] - [org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.createRpcInvocationMessage(line:290)] - Could not create remote rpc invocation

Re: Flink reactive deployment on with kubernetes operator

Hi Eric! The community cannot support old versions of the Flink operator, please upgrade to the latest version (1.9.0) Also, we do not recommend using the Reactive mode (with standalone). You should instead try Native Mode + Autoscaler which works much better in most cases. Cheers, Gyula

Flink reactive deployment on with kubernetes operator

Hi,Community: I hava encountered a problemwhen deploy reactive flink scheduler on kubernetes with flink kubernetes operator 1.6.0,the manifest and exception stack info listed as follows. Any clues would be appreciated.

RE: Flink Serialisation

After taking a closer look to the logs, I found out it was a `java.lang.OutOfMemoryError: Java heap space` error which confirms what I thought: the serialized object is too big. Here is the solution to increase the JVM heap:

Flink Serialisation

Hello, I was wondering if Flink has a size limit to serialize data. I have an object that stores a big 2D array and when I try to hand it over the next operator, I have the following error: ``` 2024-07-10 10:14:51,983 ERROR org.apache.flink.runtime.util.ClusterUncaughtExceptionHandler [] -

Buffer Priority

Hello,Community: I am puzzled by what the Priority means in Flink Buffer,it explains with example(as follows) in Buffer.java,but I still don't get what exactly is "it skipped buffers"??Could anyone give me a intuitive explanation? Thanks. /** Same as EVENT_BUFFER, but the event has been

Re:Re:Flink LAG-Function doesn't work as expected

Sorry, I mean "could not". -- Best！ Xuyang 在 2024-07-10 15:21:48，"Xuyang" 写道： Hi, which Flink version does you use? I could re-produce this bug in master. My test sql is below: ``` CREATE TABLE UNITS_DATA( proctime AS PROCTIME() , `IDENT` INT

Re:Flink LAG-Function doesn't work as expected

Hi, which Flink version does you use? I could re-produce this bug in master. My test sql is below: ``` CREATE TABLE UNITS_DATA( proctime AS PROCTIME() , `IDENT` INT , `STEPS_ID` INT , `ORDERS_ID` INT ) WITH ( 'connector' = 'datagen',

Event de duplication in flink with rabbitmq connector

Hi All, I have a Flink job with a RMQ source, tumbling windows (fires for each 2s), an aggregator, then a RMQ sink. Incremental RocksDB checkpointing is enabled with an interval of 5 minutes. I was trying to understand Flink failure recovery. My checkpoint X is started, I have sent one event to

Flink LAG-Function doesn't work as expected

Hi everyone, i'm new to flink and tried some queries with flink sql. Currently I have a problem with the LAG function. I want to emit a new record when the ORDERS_ID changes. To do this, I use the LAG function to detect whether this has changed. However, I noticed that every now and then I

Using BlobServer in FlinkDeployment

Hi all, I am deploying a FlinkDeployment CR in an Operator watched namespace. I have passed these configs in the flinkConfiguration: blob.server.address: "jobmanager" blob.server.port: "6128" blob.storage.directory: "/tmp/jars/" There are a couple of jars that I don't want to make part of the

Encountering scala.matchError in Flink 1.18.1 Query

Hi, community I encountered a scala.matchError when trying to obtain the table plan for the following query in Flink 1.18.1. The input data is read from Kafka, and the query is intended to perform a typical WordCount operation. The query is as follows. SPLIT_STRING is a Table Function UDF that

Flink Session jobs goes to reconciling state

Hi Team, I have a few session jobs for running jars. After creating jobs, the job goes from a running state to a reconciling state or upgrading state. How can I resolve this issue? [image: image.png] Thanks & Regards Nida Shaikh

java.lang.OutOfMemory:null

Hi Flink任务平稳运行一段时间，资源和数据都很平稳的情况下，一段时间后，忽然在解析json对象时报java.lang.OutOfMemory:null，然后cpu和内存就直线上升，直到完全将资源耗尽，报java.lang.OutOfMemory:java heap space，增加资源后，过一段时候会有同样的问题出现，这是什么原因，如果是内存泄漏，为什么会CPU和内存都完全平稳的运行一段时间，不应该是全程有直线上升的现象吗

Re: Parallelism of state processor jobs

Hi Junrui, I think you understood correctly. What I'm seeing is that each vertex has a single subtask, but multiple vertices are started in parallel in different slots. That is not a problem in my case, I _want_ to parallelize the work, it's just that this mechanism is very different from

Re: Parallelism of state processor jobs

Hi Alexis, Could you clarify what you mean by "If I add more slots to the task manager, I see the transformations actually start in parallel even though I submit the job with 'flink run -p 1'"? Are you asking if multiple slots are working simultaneously, or if a single JobVertex contains multiple

flinkcdc????postgrep????????checkpoint????????????????????????

PostgresSourceBuilder

Re: Parallelism of state processor jobs

Hi Junrui, Thanks for the confirmation. I tested some more and I'm seeing a strange behavior. I'm currently testing a single source stream that is fed to 6 identical transformations. The state processor api requires batch mode and, from what I can tell, I must specify a parallelism of 1 in the

Re: Parallelism of state processor jobs

Hi Alexis, For the SavepointWriter, I've briefly looked over the code and the write operation is enforced as non-parallel. Best, Junrui Alexis Sarda-Espinosa 于2024年7月6日周六 01:27写道： > Hi Gabor, > > Thanks for the quick response. What about SavepointWriter? In my case I'm > actually writing a

Re: Parallelism of state processor jobs

Hi Gabor, Thanks for the quick response. What about SavepointWriter? In my case I'm actually writing a job that will read from an existing savepoint and modify some of its data to write a new one. Regards, Alexis. Am Fr., 5. Juli 2024 um 17:37 Uhr schrieb Gabor Somogyi <

Re: Parallelism of state processor jobs

Hi Alexis, It depends. When one uses SavepointLoader to read metadata only then it's non-parallel. SavepointReader however is basically a normal batch job with all its features. G On Fri, Jul 5, 2024 at 5:21 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hello, > > Really quick

Parallelism of state processor jobs

Hello, Really quick question, when using the state processor API, are all transformations performed in a non-parallel fashion? Regards, Alexis.

Flink State and Filesystem sink

Hello, I am trying to implement a satellite image processing chain. Satellite images are stored as rasters which are heavy, (several GBs) in a FileSystem (I am currently using HDFS for testing purpose but will move on S3 when I'll deploy it on the cloud). So in order to reduce the processing

?????? puzzle on OperatorChain

Thanks for your confirmation,Yunfeng. ---- ??: "Yunfeng Zhou"

Re: puzzle on OperatorChain

Hi Enric, Yes that even if there is only one operator, StreamTask will still create an OperatorChain for it. OperatorChain provides an abstract to process events like endInputs, checkpoints and OperatorEvents in a unified way, no matter how may operators are running in the StreamTask. You may

Task manager memory go on increasing on idle stage also

Hi All, My task manager memory goes on increasing in idle stages also any reason why so. As a result of the above my job is failing. Thanks in advance. Thanks & regards, Ganesh Walse

puzzle on OperatorChain

Hello,guys: Does Flink transform all operators(including source operator) to OperatorChain evendisableoperatorchaining was set to true and even the OperatorChain contains only one single Operator. Thanks.

Re: Postgres-CDC start replication fails after stop/start on flink stream

Hi, David. We've met a similar problem of pg connection, the error message is 'Socket is closed' and we put a lot of effort into investigating, but we couldn't find the reason. Then we modify the publication mode[1] and only subscribe the changes of certain table with following connector options:

Re: watermark and barrier

Hi Enric, OperatorCoordinator is a mechanism allowing subtasks of the same operator to communicate with each other and thus unifying the behavior of subtasks running on different machines. It has mainly been used in source operators to distribute source splits. As for watermarks, there are

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

Thanks for the effort, Guyla! Best Regards Peter Huang On Wed, Jul 3, 2024 at 12:48 PM Őrhidi Mátyás wrote: > Thank you, Gyula! 拾 > Cheers > On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra wrote: > > > The Apache Flink community is very happy to announce the release of > Apache > > Flink

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

Thank you, Gyula! 拾 Cheers On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.9.0. > > The Flink Kubernetes Operator allows users to manage their Apache Flink > applications and their

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

Thank you, Gyula! 拾 Cheers On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.9.0. > > The Flink Kubernetes Operator allows users to manage their Apache Flink > applications and their

[ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.9.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Release blogpost:

[ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.9.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Release blogpost:

Flink LAG-Function doesn't work as expected

Hi everyone, i’m new to flink and tried some queries with flink sql. Currently I have a problem with the LAG function. I want to emit a new record when the ORDERS_ID changes. To do this, I use the LAG function to detect whether this has changed. However, I noticed that every now and then I

Re: Flink write ADLS show error: No FileSystem for scheme "file"

Hi, Gabor, I'm curious about why this happened in Azure file and not in other file format(I tried use s3 and it works OK) Gabor Somogyi 于2024年7月2日周二 16:59写道： > I see, thanks for sharing. > > The change what you've made makes sense. Let me explain the details. > Each and every plugin has it's

Re: Reminder: Help required to fix security vulnerabilities in Flink Docker image

Hi Alexis and Gabor , Thanks for your valuable suggestions. We tried implementing as per the suggestion given, updating the GOSE_VERSION to 1.17 and few other changes from our end and we see that all the jfrog x-ray vulnerabilities are fixed. Thanks for you support and help. Thanks, Elakiya

watermark and barrier

Hello,Community: Are watermark and checkpoint barrier just generated by source and coordinated by OperatorCoordinator/CheckpointCoordinator ? Any clues appreciated! ---- ??:

Postgres-CDC start replication fails after stop/start on flink stream

I have a flink stream using Postgres-CDC as a source. It's been operating mostly fine, but I recently had to stop, and then start the stream again. The stream is never able to start again as the replication never completes and Flink enters a restart loop. Upon starting the cdc reader task issues

Re: [External] Re: [External] Re:Re: Backpressure issue with Flink Sql Job

Hi Penny, As the join is an unwindowed join, we are performing the backfill since the data started becoming available. On Mon, Jul 1, 2024 at 3:14 PM Penny Rastogi wrote: > Hi Ashish, > > How are you performing the backfill operation? Some time window? Can you > specify details? > > I mean ,if

What the default partition assignment strategy for KafkaSourceBuilder

A simple flink task that consumes a kafka topic message and does some calculation. The number of partitions of the topic is 48, I set the parallel also 48 and expect one parallel consumes one partition. But after submitting the task I found that there's 5 parallels consuming two partitions and 5

??????Fail to pull image autoscaling

Hello,Community: Is there a reactive scheduling approach in flink which autoscales down reacting to input traffic flow reduction for deployment of flink on Kubernetes? Thanks. ---- ??:

Re: Flink write ADLS show error: No FileSystem for scheme "file"

I see, thanks for sharing. The change what you've made makes sense. Let me explain the details. Each and every plugin has it's own class loader. The reason behind that is to avoid dependency collision with Flink's main class loader. I think if the mentioned change works when it's added as normal

回复: Re:回复: Re:flink1.18 on yarn提交任务生成多个application

谢谢！！！ 祝好运， Liu 发件人: Xuyang 发送时间: 2024年7月2日 14:00 收件人: user-zh@flink.apache.org 主题: Re:回复: Re:flink1.18 on yarn提交任务生成多个application 可以参考下这[1] Tips: 社区新语法EXECUTESTATEMENTSET BEGIN ... END; ,也可以用 begin statement set; ... end;

Re:Stateful Function Roadmap

Hi, Ran. As far as I know, the development of stateFun has currently come to a standstill. You maybe need to fork this repo and pick up what your need. You can see the latest mail thread here[1]. [1] https://lists.apache.org/thread/mz5956pry8fzowdw399q0hq16zqg5f57 -- Best！

Re:回复: Re:flink1.18 on yarn提交任务生成多个application

可以参考下这[1] Tips: 社区新语法EXECUTESTATEMENTSET BEGIN ... END; ,也可以用 begin statement set; ... end; [1]https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/sql/insert/#insert-into-multiple-tables -- Best！ Xuyang 在 2024-07-02 11:42:32，"Liu Join" 写道：

Please suggest Flink OpenSearch connector ver 1.0.1-1.16 is compatible with which version of “opensearch-rest-high-level-client”

Hello Team, We are using the Flink version 1.16.3 and we are planning to use the Flink OpenSearch connector, which requires the dependency of "opensearch-rest-high-level-client". Latest version of opensearch-rest-high-level-client is 2.15 which is not working with Flink OpenSearch connector

回复: Re:flink1.18 on yarn提交任务生成多个application

你好，感谢回复，请问有官网文档么，我想详细了解下，谢谢！ 发件人: Xuyang 发送时间: 2024年7月2日 11:25 收件人: user-zh@flink.apache.org 主题: Re:flink1.18 on yarn提交任务生成多个application Hi, 如果是不想出现两个application的情况，可以试一下使用statement set将两个dml放在一起。否则的话，会被视为两个单独的操作，分成两个application。 sql： begin statement set;

Re:Flink cache support

Hi, Ganesh. Can you take a look if the cache strategy in jdbc lookup table can meet your requirement? [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/jdbc/#lookup-cache -- Best！ Xuyang At 2024-07-02 03:35:41, "Ganesh Walse" wrote:

Re:flink1.18 on yarn提交任务生成多个application

Hi, 如果是不想出现两个application的情况，可以试一下使用statement set将两个dml放在一起。否则的话，会被视为两个单独的操作，分成两个application。 sql： begin statement set; ... end; java & scala table api: tableEnv#createStatementSet -- Best！ Xuyang 在 2024-07-02 10:04:34，"Liu Join" 写道： >版本：flink1.18、hadoop3.0.0 >提交方式：per-job

Re: 这绝对算是bug

退订 On Mon, Jul 1, 2024 at 10:22 PM wjw_bigd...@163.com wrote: > 退订 > > > > 回复的原邮件 > | 发件人 | Cuixb | > | 日期 | 2024年07月01日 22:16 | > | 收件人 | user-zh@flink.apache.org | > | 抄送至 | | > | 主题 | Re: 这绝对算是bug | > GC不能说长，也绝对不短，大概计算了一下，24G内存，得有10秒左右无响应，多数在10秒内 > 发自我的 iPhone > > > 在

flink1.18 on yarn提交任务生成多个application

版本：flink1.18、hadoop3.0.0 提交方式：per-job 问题： 1. 使用flink sql编写的程序，数据源相同，输出定义了一个doris sink，以及一个表转流的print，将jar包提交到yarn会生成两个application，一个输出为doris sink，另一个输出为流转表的print 2. 使用flink sql编写的程序，数据源相同，输出定义了两个doris sink，表a和表b，将jar包提交到yarn会生成两个application，一个输出为doris sink表a，另一个输出为doris sink表b 请问这是什么原因

unsubscribe

Flink cache support

Hi All, My prerequisite for datastream is to cache 25-30 table data into the some java collection and use them later while processing the stream. Right now I am just writing select queries and using jdbc api storing those results in java collection and filtering data as per need. Do you have any

Stateful Function Roadmap

Hi team, We noticed that there wasn't any new version of Stateful Function released since last year. Is it still actively being developed? We also noticed that the dependencies of it were also old. Is there a roadmap regarding supporting newer Flink and Protobuf versions? Thanks, Ran

??????????

| ?? | wjw_bigd...@163.com | | | 2024??07??01?? 17:17 | | ?? | user-zh | | ?? | | | | ?? | | ?? | wjw_bigd...@163.com | | | 2024??07??01?? 17:12 | | ?? | user-zh | | ?? | | | |

回复：这绝对算是bug

退订 回复的原邮件 | 发件人 | Cuixb | | 日期 | 2024年07月01日 22:16 | | 收件人 | user-zh@flink.apache.org | | 抄送至 | | | 主题 | Re: 这绝对算是bug | GC不能说长，也绝对不短，大概计算了一下，24G内存，得有10秒左右无响应，多数在10秒内 发自我的 iPhone > 在 2024年7月1日，17:20，rui chen 写道： > > 建议检查一下JM的GC情况。 > > wjw_bigd...@163.com 于2024年7月1日周一 17:18写道： > >>

Re: JVM heap not getting freed

Hi Ganesh After the job is finished, was there a Full GC during this period? You can try to capture a heap dump to trigger FullGC and then check the actual memory usage. If the memory usage is still high after a Full GC, you should analyze the heap dump file. Best, Feng On Mon, Jul 1, 2024 at

Re: 这绝对算是bug

GC不能说长，也绝对不短，大概计算了一下，24G内存，得有10秒左右无响应，多数在10秒内 发自我的 iPhone > 在 2024年7月1日，17:20，rui chen 写道： > > 建议检查一下JM的GC情况。 > > wjw_bigd...@163.com 于2024年7月1日周一 17:18写道： > >> 退订 >> >> >> >> 回复的原邮件 >> | 发件人 | wjw_bigd...@163.com | >> | 日期 | 2024年07月01日 17:13 | >> | 收件人 | user-zh | >> | 抄送至 |

JVM heap not getting freed

Hi Team, After I ran my fink application on cluster and application runs successfully but my JVM heap is still showing 50% is filled. What will be the reason?

Re: [External] Re:Re: Backpressure issue with Flink Sql Job

Hi Ashish, How are you performing the backfill operation? Some time window? Can you specify details? I mean ,if it helps, you can check out https://www.ververica.com/blog/how-to-write-fast-flink-sql . Regards On Tue, Jun 25, 2024 at 4:30 PM Ashish Khatkar via user < user@flink.apache.org>

Re: 这绝对算是bug

建议检查一下JM的GC情况。 wjw_bigd...@163.com 于2024年7月1日周一 17:18写道： > 退订 > > > > 回复的原邮件 > | 发件人 | wjw_bigd...@163.com | > | 日期 | 2024年07月01日 17:13 | > | 收件人 | user-zh | > | 抄送至 | | > | 主题 | 回复：这绝对算是bug | > 退订 > > > > 回复的原邮件 > | 发件人 | 星海<2278179...@qq.com.INVALID> | > | 日期 | 2024年06月29日

????????????????bug

| ?? | wjw_bigd...@163.com | | | 2024??07??01?? 17:13 | | ?? | user-zh | | ?? | | | | bug | | ?? | <2278179...@qq.com.INVALID> | | | 2024??06??29?? 21:31 | | ?? | user-zh | | ??

??????????

| ?? | wjw_bigd...@163.com | | | 2024??07??01?? 17:12 | | ?? | user-zh | | ?? | | | | ?? | | ?? | zhanggongx | | | 2024??07??01?? 16:52 | | ?? | user-zh | | ?? | | | | |

????????????????bug

| ?? | <2278179...@qq.com.INVALID> | | | 2024??06??29?? 21:31 | | ?? | user-zh | | ?? | | | | ?? ??bug | ---- ??:

??????????

| ?? | zhanggongx | | | 2024??07??01?? 16:52 | | ?? | user-zh | | ?? | | | | |

Re: 这绝对算是bug

连接没问题，主要是tm一直在处理写入流，我也看了一下负载，其实不高，但就是不相应，导致报timeout，然后就是最开始那个错误！ 发自我的 iPhone > 在 2024年6月29日，16:49，Zhanghao Chen 写道： > > Hi，从报错看是 JM 丢主了，导致 TM 上 task 全部关停。看下 JM 侧是不是 HA 连接有问题呢？ > > Best, > Zhanghao Chen > > From: Cuixb > Sent: Saturday, June 29, 2024 10:31 >

Re: 这绝对算是bug

Hi，从报错看是 JM 丢主了，导致 TM 上 task 全部关停。看下 JM 侧是不是 HA 连接有问题呢？ Best, Zhanghao Chen From: Cuixb Sent: Saturday, June 29, 2024 10:31 To: user-zh@flink.apache.org Subject: 这绝对算是bug 生产环境Flink 1.16.2 2024-06-29 09:17:23 java.lang.Exception: Job leader for job id

elasticsearch-3.1.0

Dear Team, Elasticsearch 8 support has been added for flink but not exposed yet. As per reports, elasticsearch-3.1.0 release has it and we would like to try it out. Should we manually build the code and use the jar? Is there any mvn repo link that can be used? Any sample code( I have gone

这绝对算是bug

生产环境Flink 1.16.2 2024-06-29 09:17:23 java.lang.Exception: Job leader for job id 8ccdd299194a686e3ecda602c3c75bf3 lost leadership. at org.apache.flink.runtime.taskexecutor.TaskExecutor$JobLeaderListenerImpl.lambda$null$2(TaskExecutor.java:2310) at

回复：退订

退订 回复的原邮件 | 发件人 | 张胜军<15910621...@139.com> | | 发送日期 | 2024年6月28日 12:57 | | 收件人 | user-zh | | 主题 | 退订 | 退订 扫一扫, 快速添加名片到手机 The following is the content of the forwarded

退订

退订 扫一扫, 快速添加名片到手机 The following is the content of the forwarded email From：Yanquan Lv To：user-zh Date：2024-06-26 16:46:05 Subject：Re: Re：cdc读取oracle数据如何解析 可以的，通过设置 debezium 的

Fail to pull image autoscaling

Hi,Community: I'm using flink-kubernetes-operator of version 1.6 to deploy flink auto scaling (https://github.com/apache/flink-kubernetes-operator/blob/main/examples/autoscaling/autoscaling.yaml) on K8S,but encountered the errorfail to pull the image autoscaling-example,Could any one give me

Re: Flink write ADLS show error: No FileSystem for scheme "file"

Hi, Gabar, Thanks to reply, I make sure that not conflict in maven, all the hadoop dependency is in provided scope, and checked my result jar it not contains (src/main/resources/META-INF/services). This is my pom: http://maven.apache.org/POM/4.0.0;

Flink k8s operator starts wrong job config from FlinkSessionJob

Hi all, we're running a session cluster and I submit around 20 jobs to it at the same time by creating FlinkSessionJob Kubernetes resources. After sufficient time there are 20 jobs created and running healthy. However, it appears that some jobs are started with the wrong config. As a result some

Re: Flink write ADLS show error: No FileSystem for scheme "file"

Hi Xiao, I'm not quite convinced that the azure plugin ruined your workload, I would take a look at the dependency graph you've in the pom. Adding multiple deps can conflict in terms of class loader services (src/main/resources/META-INF/services). As an example you've 2 such dependencies where

Re: Re：cdc读取oracle数据如何解析

可以的，通过设置 debezium 的 decimal.handling.mode [1] 参数可以实现你的需求，转成 double 或者 string 来处理。 [1] https://debezium.io/documentation/reference/1.9/connectors/oracle.html#oracle-numeric-types

??????????

| ?? | <402987...@qq.com.INVALID> | | | 2024??06??26?? 16:38 | | ?? | user-zh | | ?? | | | | ?? | ---- ??:

回复：退订

在 2024-06-26 15:07:45，"15868861416" <15868861...@163.com> 写道： >你好，可以把ID和PRICE的类型改为NUMBER试一下，我这边flink-sql试过number类型对应到iceberg的decimal数据是正常的 > > >| | >博星 >| >| >15868861...@163.com >| > > > 回复的原邮件 >| 发件人 | Yanquan Lv | >| 发送日期 | 2024年06月26日 14:46 | >| 收件人 | | >| 主题 | Re:

回复： cdc读取oracle数据如何解析

你好，可以把ID和PRICE的类型改为NUMBER试一下，我这边flink-sql试过number类型对应到iceberg的decimal数据是正常的 | | 博星 | | 15868861...@163.com | 回复的原邮件 | 发件人 | Yanquan Lv | | 发送日期 | 2024年06月26日 14:46 | | 收件人 | | | 主题 | Re: 回复：cdc读取oracle数据如何解析 | 你好，你的 ID 和 PRINCE 字段类型是 decimal 吗，decimal 默认的展示方式是使用 BASE64 处理

Flink write ADLS show error: No FileSystem for scheme "file"

Hi, all I try to use Flink to write Azure Blob Storage which called ADLS, I put the flink-azure-fs-hadoop jar in plugins directory and when I start my write job it shows: Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file" at

Re: 回复：cdc读取oracle数据如何解析

你好，你的 ID 和 PRINCE 字段类型是 decimal 吗，decimal 默认的展示方式是使用 BASE64 处理 可以通过添加下面代码来让展示信息更直观。 Map customConverterConfigs = new HashMap<>(); customConverterConfigs.put(JsonConverterConfig.DECIMAL_FORMAT_CONFIG, "numeric"); JsonDebeziumDeserializationSchema schema = new

Re: [External] Re:Re: Backpressure issue with Flink Sql Job

Hi Xuyang, The input records are balanced across subtasks, with debloating buffers enabled, the records this subtask receives is less as compared to other subtasks. If the differences among all subtasks are not significant, we might be > encountering an IO bottleneck. In this case, we could try

回复：cdc读取oracle数据如何解析

退订真的很麻烦，，，退订了好几次没搞懂 回复的原邮件 | 发件人 | ha.fen...@aisino.com | | 日期 | 2024年06月25日 17:25 | | 收件人 | user-zh | | 抄送至 | | | 主题 | Re: 回复：cdc读取oracle数据如何解析 | 数据没问题 "ID" "NAME" "ADDTIME""PRICE" 1 "aa" 2024-6-25 14:21:33 12.22 发件人： 15868861416 发送时间： 2024-06-25 17:19 收件人：

回复：cdc读取oracle数据如何解析

检查一下oracle中的DEBEZIUM.PRODUCTS这个表的数据，看你解析中有字符串 | | 博星 | | 15868861...@163.com | 回复的原邮件 | 发件人 | ha.fen...@aisino.com | | 发送日期 | 2024年06月25日 15:54 | | 收件人 | user-zh | | 主题 | cdc读取oracle数据如何解析 | 根据文档的代码 JdbcIncrementalSource oracleChangeEventSource = new OracleSourceBuilder()

?????? flink kubernetes flink autoscale behavior

Thanks,Rion.I roughly understand.And I still want to know whether the deployment ofAdaptive Schedulerrelies on Kubernetes? Are there any cases of deploying FlinkAdaptive Scheduleron bare metal machine? Appreciated again. ---- ??:

Flink AsyncWriter如何进行固定速率的限速？这一块似乎有bug

Flink 1.16.0 搜索到社区有相关文章，其中的实例如下： https://flink.apache.org/2022/11/25/optimising-the-throughput-of-async-sinks-using-a-custom-ratelimitingstrategy/#rationale-behind-the-ratelimitingstrategy-interface public class TokenBucketRateLimitingStrategy implements RateLimitingStrategy {

Re: flink kubernetes flink autoscale behavior

Hi Eric,I believe you might be referring to use of the adaptive scheduler which should support these “in-place” scaling operations via:jobmanager.scheduler: adaptiveYou can see the documentation for Elastic Scaling here for additional details and configuration.On Jun 24, 2024, at 11:56 PM, Enric

Re:Re: Backpressure issue with Flink Sql Job

Hi, Ashish. Can you confirm whether, on the subtask label page of this sink materializer node, the input records for each subtask are approximately the same? If the input records for subtask number 5 are significantly larger compared to the others, it signifies a serious data skew, and it

  1   2   3   4   5   6   7   8   9   10   >