Re: Exception handling

Hi Jacob, one of the contracts Flink has is that if a UDF throws an exception then this means that it has failed and that it needs recovery. Hence, it is the responsibility of the user to make sure that tolerable exceptions do not bubble up. If you have dirty input data then it might make sense to

Re: Watermarks in Event Time Temporal Join

Thanks for your example, Maciej I can explain more about the design. > Let's have events. > S1, id1, v1, 1 > S2, id1, v2, 1 > > Nothing is happening as none of the streams have reached the watermark. > Now let's add > S2, id2, v2, 101 > This should trigger join for id1 because we have all the kno

Any configuration for accelerating state processor

Hi, I have a job that uses the state processor to load data from checkpoints on google cloud storage to do some processing and then write the result to google cloud storage. The total data size is about 30-50 GB and the job may take more than 2 hours to finish. From the flame graph generated fr

Re: [Announcement] Analytics Zoo 0.10.0 release

Please See the following: 1) Analytics Zoo* PPML *supports running unmodified Flink programs in a secure fashion on an untrusted cloud ( https://analytics-zoo.readthedocs.io/en/latest/doc/PPML/Overview/ppml.html#trusted-realtime-compute-and-ml ) 2) Analytics Zoo *Cluster Serving* supports distrib

Stateful Functions, Kinesis, and ConsumerConfigConstants

When using Flink Stateful Function's KinesisIngressBuilder, I do not see a way to set things like ConsumerConfigConstants.SHARD_GETRECORDS_MAX or ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS Looking at KinesisSourceProvider, it appears that this is the spot that creates the FlinkKinesi

KafkaSourceBuilder causing invalid negative offset on checkpointing

Hello, I ran into an issue when using the new KafkaSourceBuilder (running Flink 1.12.2, scala 2.12.13, on ververica platform in a container with java 8). Initially it generated warnings on kafka configuration, but the job was able to consume and produce messages. The configuration 'client.id.prefi

Re: Taskmanager killed often after migrating to flink 1.12

Hi Till, Thank you for the response, we are currently running flink with an increased memory usage, so far the taskmanager is working fine, we will check if there is any further issue and will update you. Regards Sambaran On Wed, Apr 28, 2021 at 5:33 PM Till Rohrmann wrote: > Hi Sambaran, > >

Re: Checkpoint error - "The job has failed"

Oh interesting. Yea, could be. We'll soon update to v1.12. Thanks Robert and Yun! On Wed, Apr 28, 2021 at 1:30 AM Yun Tang wrote: > Hi Dan, > > You could refer to the "Fix Versions" in FLINK-16753 [1] and know that > this bug is resolved after 1.11.3 not 1.11.1. > > [1] https://issues.apache.

Re: Watermarks in Event Time Temporal Join

Hi Leonard, Let's assume we have two streams. S1 - id, value1, ts1 with watermark = ts1 - 1 S2 - id, value2, ts2 with watermark = ts2 - 1 Then we have following interval join SELECT id, value1, value2, ts1 FROM S1 JOIN S2 ON S1.id = S2.id and ts1 between ts2 - 1 and ts2 Let's have events. stream,

Re: Taskmanager killed often after migrating to flink 1.12

Hi Sambaran, could you also share the cause why the checkpoints could not be discarded with us? With Flink 1.10, we introduced a stricter memory model for the TaskManagers. That could be a reason why you see more TaskManagers being killed by the underlying resource management system. You could ma

Re: [Announcement] Analytics Zoo 0.10.0 release

Hi Jason, Thanks for sharing. I look up the term "Flink" on https://analytics-zoo.readthedocs.io/en/latest/ but it doesn't even exist. Do you mind sharing how does it relate to Flink users? Best, Yik San On Wed, Apr 28, 2021 at 10:48 PM Jason Dai wrote: > Hi Everyone, > > > I’m happy to announ

Queryable State unavailable after Kubernetes HA State cleanup

Hello, Stuck at this time. Any help will be appreciated. I am able to create a queryable state and also query the state. Everything works correctly. KeyedStream, Key> stream = sourceStream.keyBy(t2 -> t2.f0); stream.asQueryableState("queryableVO"); I deploy this on a Kubernetes cluster with F

Re: Application cluster - Job execution and cluster creation timeouts

Hey All, I know Application cluster is not widely used yet, I'm happy to be part of Flink community , test it and share the results. Following my previous email, I'd like to share more information and get your feedback. Scenario 4 : requestJobResult() gets out of sync. The result is very simil

How to implement a window that emits events at regular intervals and on specific events

Hello! I'm trying to figure out how to implement a window that will emit events at regular intervals or when a specific event is encountered. A bit of background. I have a stream of events from devices that will send events to our system whenever a user watches a video. These events include a uni

Re: [Stateful Functions] Help for calling remote stateful function (written in Python)

Dear Igal, dear List Thank you very much for your reply. Your advice was crucial to overcome the issue. I have now created a TypedValue manually and successfully managed to communicate with the remote function in Python. Nevertheless, I am still facing a strange behavior regarding the invocati

Fwd: [Announcement] Analytics Zoo 0.10.0 release

Hi Everyone, I’m happy to announce the 0.10.0 release for Analytics Zoo (distributed TensorFlow and PyTorch on Apache Spark/Flink & Ray); the highlights of this release include: - A re-designed document website

Flink Resuming From Checkpoint With "-s" FAILURE

Hello, I am running Flink version 1.10 on Amazon EMR. We use RocksDB/S3 for state. I have "Persist Checkpoints Externally" enabled. Periodically I must tear down the current infrastructure and bring it back up. To do this, I terminate the EMR, bring up a fresh EMR cluster, and then I resume the Fli

Key by Kafka partition / Kinesis shard

Hello To learn Flink I'm trying to build a simple application where I want to save events coming from Kinesis to S3. I want to subscribe to each shard, and within each shard I want to batch for 30 seconds, or until 1000 events are observed. These batches should then be uploaded to S3. What I don't

Re: Using Hive UDFs

Hi Youngwoo, That's no problem at all and glad to know the UDF works now. Yeah, before you can use a hive udf, you should register it into metastore. And that can be done via either Flink or Hive. Feel free to let me know if you encounter any other issues. On Wed, Apr 28, 2021 at 4:28 PM Youngwo

KafkaSourceBuilder causing invalid negative offset on checkpointing

Hello, I ran into some issues when using the new KafkaSourceBuilder (running Flink 1.12.2, scala 2.12.13, on ververica platform in a container with java 8). Initially it generated warnings on kafka configuration, but the job was able to consume and produce messages. The configuration 'client.i

Re: Checkpoint error - "The job has failed"

Hi Dan, You could refer to the "Fix Versions" in FLINK-16753 [1] and know that this bug is resolved after 1.11.3 not 1.11.1. [1] https://issues.apache.org/jira/browse/FLINK-16753 Best Yun Tang From: Dan Hill Sent: Tuesday, April 27, 2021 7:50 To: Yun Tang Cc:

Re: Using Hive UDFs

Hey Rui, My bad! You have already pointed out to me what I completely misunderstood. I've been confusing some of the steps to register udfs. And also, somehow, my metastore was a mess. So, I cleaned up the metastore and database and then, I created a database for hive catalog and registered the fu

Best practice for packaging and deploying Flink jobs on K8S

Hi, I have a PyFlink job that consists of: - Multiple Python files. - Multiple 3rdparty Python dependencies, specified in a `requirements.txt` file. - A few Java dependencies, mainly for external connectors. - An overall job config YAML file. Here's a simplified structure of the c