Re: Deployment on k8s via API

Maybe you could have a try on the flink-kubernetes-operator[1]. It is designed for using Kubernetes CRD to manage the Flink applications. [1]. https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-0.1/ Best, Yang Devin Bost 于2022年5月18日周三 08:29写道： > Hi, > > I'm looking at

Upgrade help

I see that Flink 1.15.0 supports Kafka 0.10 , and the Kafka used in our production cluster is 0.10.0.1. However, when we upgrade Flink from 1.11.3 to 1.15.0, it is prompted that many classes of Kafka cannot be found（eg:ProducerFencedException）. These classes are only available in the

????????Job??????TaskManager ??????oom?

?? ?? ??Flink 1.14 standalone Flink job??supervisorctl??JM??TM??systemd ?? jobflink??restart-strategy.fixed-delay.attempts: 2??

退订

退订

Re: flink on k8s native开启ha后根据sp启动任务报错找不到job id 0000

flink版本1.13.0 /home/hdfs/flink-1.13.0/bin/flink run-application \ -t kubernetes-application \ -s spPath \ -p 32 \ -Dresourcemanager.taskmanager-timeout=6 \ -Dkubernetes.namespace=xxx \ -Dkubernetes.service-account=xxx \ -Dkubernetes.taskmanager.service-account=xxx \

Re: Question of Flink Operator Application Cluster Deployment

Hi John, No such deployment or services in the K8S cluster. Same issue happens to the flink native kubernetes deployment. We have the podsecuritypolicy defined, but I have added flink service account into the psp. *Xiao Ma* *Geotab* Software Developer, Data Engineering | B.Sc, M.Sc Direct

Re: Question of Flink Operator Application Cluster Deployment

Hi Xiao, Is istio or something similar deployed to the K8S cluster? John On Tue, May 17, 2022 at 4:26 PM Xiao Ma wrote: > loop in > *Xiao Ma* > *Geotab* > Software Developer, Data Engineering | B.Sc, M.Sc > Direct +1 (416) 836 - 3541 > Toll-free +1 (877) 436 - 8221 > Visit

Deployment on k8s via API

Hi, I'm looking at my options for automating the deployment of Flink jobs on k8s (ideally using application mode), and I noticed that most of the examples of deploying Flink jobs in the docs use calls to the Flink binary, such as: $ ./bin/flink run-application \--target

Re: Flink MiniCluster: Multiple Task Managers

Hi Roman, Just used it. Thanks for your help. 14 Μαΐου 2022 10:06 ΜΜ, "Roman Grebennikov" έγραψε: > Hey Κωνσταντίνος, > > check out this sample code we use for testing > https://github.com/metarank/metarank . It is in > scala, but should be quite straightforward to port to java: > > val

Re: Kinesis Sink - Data being received with intermittent breaks

Hello Zain, Thanks for providing the additional information. Going back to the original issue: - You are seeing bursty throughput, but the job is keeping up? There is no backpressure? - What is the throughput at the sink? - On the graph screenshot, what is the period and stat (sum/average/etc)?

Memory configuration for Queue

Hi, I am using a kafka source with a kinesis sink and the speed of data coming in is not the same as data flowing out hence the need to configure a relatively larger queue to hold the data before backpressuring. Which memory configuration corresponds to this that I'll need to configure?

Fwd: Question of Flink Operator Application Cluster Deployment

loop in *Xiao Ma* *Geotab* Software Developer, Data Engineering | B.Sc, M.Sc Direct +1 (416) 836 - 3541 Toll-free +1 (877) 436 - 8221 Visit www.geotab.com Twitter | Facebook | YouTube

Metrics in Flink UI

Hi, I'm running a job on a local flink cluster but metrics are showing as Bytes received,records received,bytes sent,backpressure all 0 in the flink UI even though I'm receiving data in the sink. Do I need to additionally configure something to see these metrics work in real time?

RE: Checkpoint directories not cleared as TaskManagers run

Hi James, From reading the thread … I assume, your file:/tmp/Flink/State folder is not shared across all machines, right? In this case it cannot work: - checkpoints and savepoints need to go to a path that can be commonly accessed by jobmanager and all taskmanagers in order to work - as your

Re: Checkpoint directories not cleared as TaskManagers run

Thanks for your replay. To be clear on my setup with the problem: * 4 taskmanagers running across different containers and machines. Each container has its own filesystem including / and /tmp. * 1 jobmanager also running in its own container and machine. Also has its own filesystem.

Re: flink on k8s native开启ha后根据sp启动任务报错找不到job id 0000

Hi, shimin 用的哪个版本的 Flink？提交命令是什么呢？ Best, Weihua > 2022年5月17日 下午1:48，shimin huang 写道： > > flink on native k8s根据savepoint停止任务后在根据savepoint启动任务报错找不到job > 错误堆栈如下： > java.util.concurrent.ExecutionException: > org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find > Flink job

Re: Checkpoint directories not cleared as TaskManagers run

Hi, James. I may not get what the problem is. All checkpoints will store in the address as you set. IIUC, TMs will write some checkpoint info in their local dir and then upload them to the address and then delete local one. JM will write some metas of checkpoint to the address and also do the

Re: Checkpoint directories not cleared as TaskManagers run

Some further Googling says on a StackOverflow posting it is the jobmanager that does the deletion and not the taskmanagers. Currently my taskmanagers are writing their checkpoints to their own private disks (/tmp) rather than a share - so my suspicion is the jobmanager can't access the folder

Re: Channel became inactive while submitting job

Hi, Which version of Flink are you using? And what is the start cmd? Best, Weihua > 2022年5月17日 下午6:33，Zain Haider Nemati 写道： > > main method caused an error: Failed to execute job 'Tracer Processor'. > at >

Re: Does kafka key is supported in kafka sink table

Hey wang! Perhaps this is what you want: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#key-format & https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#key-fields ? Note that the fields *have* to be one of the "top"

Re: Does kafka key is supported in kafka sink table

Hey wang! Perhaps this is what you want: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#key-format & https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#key-fields ? Note that the fields *have* to be one of the "top"

Channel became inactive while submitting job

Hi, I am trying to run a job in my local cluster and facing this issue. org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'Tracer Processor'. at

Re: How to avoid multiple reading from same topic

Hi! In that case just put these two SQL queries into one job and use the same table as their source. The source task will be automatically reused (actually controlled by a configuration [1] but it is enabled by default) and the records will be only read once from Kafka and will be replicated

Re: How to avoid multiple reading from same topic

So let's assume if we are reading data from a topic named A and performing aggregation on 4 hour window and a different pipeline which does same aggregation for 6 hour window, so in this case we are reading same data couple of times and processing it for different intervals, can we do it using

Re: How to avoid multiple reading from same topic

Hi! I wanted to explore if we can avoid reading multiple times and read only > once > Could you elaborate more on the reason you need this? If both of your queries need full data from that topic you'll have to read the whole topic many times. If each query only need to consume a specific portion

How to avoid multiple reading from same topic

Hi Team, We have various SQL queries where we are querying the same kafka topic but both of the queries are completely different. I wanted to explore if we can avoid reading multiple times and read only once and perform different processing according to the query and dump data accordingly.

Re:如何更新中文文档？

Hi, 文档的贡献可以参考[1]。 1、一般来说，发现一个问题，就可以直接开issue，修文档bug，issue的component贴好文档和对应的模块就行。 2、关于你说的讨论区，因为开发者涉及国内外，因此现在基本都是在dev邮箱[2]讨论比较重要的决定，社区现在也在尝试建一个slack来更高效的讨论内容，详见[3]。pr会有热心同学给你review，如果有翻译错误的情况，会直接在你开的issue或pr下面讨论的。 ps: 你的图挂了，下次可以贴图床的链接。 [1]