Re: Regarding FLIP-91's status

Hi Matthias, Thanks for your quick response! I have sent a reply on the FLIP-91 thread, thanks for pointing me to it. @Jark Wu it'll be great if you have any context on this as well. Thanks, Sonam From: Matthias Pohl Sent: Friday, May 2

Re: Running multiple CEP pattern rules

Hi Dawid, Do you have any plans to bring this functionality in flink CEP in future ? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Running multiple CEP pattern rules

Hi Tejas, It will not work that way. Bear in mind that every application of CEP.pattern creates a new operator in the graph. The exceptions you are seeing most probably result from calculating that huge graph and sending that over. You are reaching the timeout on submitting that huge graph. You ca

Re: Statefun 2.2.2 Checkpoint restore NPE

Ok so after digging into it a bit it seems that the exception was thrown here: https://github.com/apache/flink-statefun/blob/release-2.2/statefun-flink/statefun-flink-io-bundle/src/main/java/org/apache/flink/statefun/flink/io/kafka/RoutableProtobufKafkaIngressDeserializer.java#L48 I think it'd be

Flink Metrics Naming

Can anyone give insight as to why Flink allows 2 metrics with the same “name”? For example, getRuntimeContext.addGroup(“group”, “group1”).counter(“myMetricName”); And getRuntimeContext.addGroup(“other_group”, “other_group1”).counter(“myMetricName”); Are totally valid. It seems that it has l

How to check is Kafka partition "idle" in emitRecordsWithTimestamps

Hello, I'm thinking about implementing custom Kafka connector which provides event alignment (similar to FLINK-10921, which seems abandoned). What is the way to determine is partition is idle from override of AbstractFetcher.emitRecordsWithTimestamps()? Does KafkaTopicPartitionState has this in

[ANNOUNCE] Apache Flink 1.13.1 released

|The Apache Flink community is very happy to announce the release of Apache Flink 1.13.1, which is the first bugfix release for the Apache Flink 1.13 series.|   |Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streami

Re: Triggering Savepoint fails to write data to S3 store

Yes, that would work. But it might be still interesting to understand why you ran into the timeout. Was it just a big state that just took longer than expected? Or some network issue? ...that's just for you to understand the underlying issue in a better way. But I'm glad the savepoint creation was

Re: How can I use different user run flink

Hi igyu, I'm not sure whether I can be of much help here because I'm not that familiar with Kerberos. But the Flink documentation [1] suggests deploying separate Flink clusters for each keytab. Did you try that? Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deploy

Re: JM cannot recover with Kubernetes HA

Hi Enrique, thanks for reaching out to the community. I'm not 100% sure what problem you're facing. The log messages you're sharing could mean that the Flink cluster still behaves as normal having some outages and the HA functionality kicking in. The behavior you're seeing with leaders for the dif

StreamingFileSink only writes data to MINIO during savepoint

On my kubernetes cluster when I set the StreamingFileSink to write to a local instance of S3 (MINIO - 500 GB) it only writes the data after I execute a savepoint The expected behavior is to write the data in real-time. I'm guessing the memory requirements have not been met or a configuration in MI

Re: [VOTE] Release 1.13.1, release candidate #1

Thank you all for the votes. I am happy to say we approved the release. I will write a separate summary mail. Best, Dawid On 28/05/2021 14:40, Robert Metzger wrote: > +1 (binding) > > - Tried out reactive mode in from the scala 2.11 binary locally (with > scale up & stop with savepoint) > - revi

Re: Statefun 2.2.2 Checkpoint restore NPE

Oh wow that Harness looks cool, I'll have to take a look at that. Unfortunately the JobManager UI seems to just show this: [image: image.png] Though it does seem that maybe the source function is where the failure is happening according to this? [image: image.png] Still investigating, but I do s

Re: Query related to Minimum scrape interval for Prometheus and fetching metrics of all vertices in a job through Flink Rest API

Hi Ashutosh, you can set the metrics update interval through metrics.fetcher.update-interval [1]. Unfortunately, there is no single endpoint to collect all the metrics in a more efficient way other than the metrics endpoints provided in [2]. I hope that helps. Best, Matthias [1] https://ci.apache

Re: Alternate way to pass plugin Jars

Hi Vijayendra, Thanks for reaching out to the Flink community. There is no other way I know of to achieve what you're looking for. The plugins support is provided through the respective ./plugins/ directory as described in the docs [1]. Best, Matthias [1] https://ci.apache.org/projects/flink/flin

Re: Heartbeat Timeout

Matthias, I increased the JVM Heap size as Jan suggested and it appears to be a memory leak in the user code (although I'm not sure why since this is a simple job that uses a loop to simulate data being written to an S3 data store). Yes, the logs show no apparent problem but the timestamp corresp

Re: [VOTE] Release 1.13.1, release candidate #1

+1 (binding) - Tried out reactive mode in from the scala 2.11 binary locally (with scale up & stop with savepoint) - reviewed website update - randomly checked a jar file in the staging repo (flink-python jar looks okay (I just checked superifically)) On Fri, May 28, 2021 at 5:16 AM Leonard Xu

Re: Triggering Savepoint fails to write data to S3 store

Hi Matthias, You are correct. After a few minutes I took another look at my savepoint folder and the data was there. I think increasing the timeout may resolve the problem? On Fri, May 28, 2021 at 8:21 AM Matthias Pohl wrote: > Hi Robert, > it would be interesting to see the corresponding tas

Re: Heartbeat Timeout

Hi Robert, increasing heap memory usage could be due to some memory leak in the user code. Have you analyzed a heap dump? About the TM logs you shared. I don't see anything suspicious there. Nothing about memory problems. Are those the correct logs? Best, Matthias On Thu, May 27, 2021 at 6:01 PM

Re: Triggering Savepoint fails to write data to S3 store

Hi Robert, it would be interesting to see the corresponding taskmanager/jobmanager logs. That would help in finding out why the savepoint creation failed. Just to verify: The savepoint data wasn't written to S3 even after the timeout happened, was it? Best, Matthias On Thu, May 27, 2021 at 7:50 P

Re: Regarding FLIP-91's status

Hi Sonam, It looks like it has been stale for some time. You might be able to restart the discussion replying to the respective thread in the dev mailing list [1]. You seem to be right about the repository based on Jark's reply in the related ticket FLINK-15472 [2]. I'm adding Jark to the thread. M

Re: KafkaSource

Hi Alexey, the two implementations are not compatible. You can find information on how to work around this in the Kafka Connector docs [1]. Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/#upgrading-to-the-latest-connector-version

Re: Flink upgraded from 1.10.0 to 1.12.0

Hi 王炳焱, thanks for reaching out to the Flink community and sorry for the late reply. Unfortunately, Flink SQL does not support state backwards compatibility. There is no clear pointer in the documentation that states that. I created FLINK-22799 [1] to cover this. In the mean time, you could try usi

Re: Statefun 2.2.2 Checkpoint restore NPE

If logs are not helping, I think the remaining option is to attach a debugger [1]. I'd probably add a breakpoint to LegacySourceFunctionThread#run and see what happens. If the issue is in recovery, you should add a breakpoint to StreamTask#beforeInvoke. [1] https://cwiki.apache.org/confluence/disp

Re: Flink in k8s operators list

https://github.com/lyft/flinkk8soperator On Fri, May 28, 2021, 10:09 Ilya Karpov wrote: > Hi there, > > I’m making a little research about the easiest way to deploy link job to > k8s cluster and manage its lifecycle by *k8s operator*. The list of > solutions is below: > - https://github.com/fint

Re: Statefun 2.2.2 Checkpoint restore NPE

Hi Tim, Any additional logs from before are highly appreciated, this would help us to trace this issue. By the way, do you see something in the JobManager's UI? On Fri, May 28, 2021 at 9:06 AM Tzu-Li (Gordon) Tai wrote: > Hi Timothy, > > It would indeed be hard to figure this out without any sta

Re: Enable Multiple kafka Consumer sources for job

For debug, you can just pull data from inputStream2. sudhansu069 [via Apache Flink User Mailing List archive.] < ml+s2336050n44010...@n4.nabble.com> 于2021年5月27日周四 下午11:22写道： > Hi Team , > > We are trying to build a data pipeline where we have to set up two > different kafka consumers for two diff

Re: Configure Kafka ingress through property files in Stateful function 3.0.0

Hi Jessy, I assume "consumer.properties" is a file you have included in your StateFun application's image? The ingress.spec.properties field in the module YAML specification file expects a list of key value pairs, not a properties file. See for example [1]. I think it could make sense to support

Flink in k8s operators list

Hi there, I’m making a little research about the easiest way to deploy link job to k8s cluster and manage its lifecycle by k8s operator. The list of solutions is below: - https://github.com/fintechstudios/ververica-platform-k8s-operator - https://github.com/GoogleCloudPlatform/flink-on-k8s-opera

Re: Statefun 2.2.2 Checkpoint restore NPE

Hi Timothy, It would indeed be hard to figure this out without any stack traces. Have you tried changing to debug level logs? Maybe you can also try using the StateFun Harness to restore and run your job in the IDE - in that case you should be able to see which code exactly is throwing this excep