Re: Enabling savepoints when deploying in Application Mode

2022-04-11 Thread Gyula Fóra
Hi Lilli! I am not aware of any problems with savepoint restore in application mode. What you can try is to use the *execution.savepoint.path *configuration setting to control it instead of the CLI and see if it makes a difference for you. Otherwise, you could also check out the https://github.co

Reactive mode and checkpointing

2022-04-11 Thread aryan m
Hello ! Are there options in reactive mode to prevent a job from restarting if the last checkpoint failed or timed out due to any reason ? Thanks, AR

Discuss making KafkaSubscriber Public

2022-04-11 Thread Mason Chen
Hi Flink Devs, I was looking to contribute to https://issues.apache.org/jira/browse/FLINK-24660, which is a ticket to track changing the KafkaSubscriber from Internal to PublicEvolving. In the PR, it seems a few of us have agreement on making the subscriber pluggable in the KafkaSource, but I'd l

Re: How to process events with different JSON schemas in single Kinesis stream using PyFlink?

2022-04-11 Thread John Tipper
Hi Dian, Thank you very much, that worked very nicely. Kind regards, John From: Dian Fu Sent: 11 April 2022 06:32 To: John Tipper Cc: user@flink.apache.org Subject: Re: How to process events with different JSON schemas in single Kinesis stream using PyFlink?

Enabling savepoints when deploying in Application Mode

2022-04-11 Thread Lilli Pearson
Hi, Summary: I've run into a number of issues trying to marry savepoints with running Flink in Application Mode, and am wondering if anyone has suggestions on how to resolve them, or if savepoints and Application Mode simply aren't designed to work together. Context on app deployment: For long-ru

Re: Broadcast state corrupted ?

2022-04-11 Thread Alexey Trenikhun
I didn’t try same savepoint cross environments. Operator with broadcast state was added recently, I rolled back all environments, created save points with old version, upgraded to version with broadcast state, all 4 were upgraded fine, took savepoints in each environment and tried to restore fr

Avro deserialization issue

2022-04-11 Thread Anitha Thankappan
Hi, I developed a flink connector to read data from bigquery . The Bigquery read rows are in AVRO format. I tried it with 1.13.1 its working fine. But my requirement is 1.11.0, in that case the code: deserializer.open(RuntimeContextInitializationContextAdapters.deserializationAdapter(getRuntimeCon

RE: RocksDB's state size discrepancy with what's seen with state processor API

2022-04-11 Thread Alexis Sarda-Espinosa
Some additional information that I've gathered: * The number of unique keys in the system is 10, and that is correctly reflected in the state. * TTL for global window state is set to update on read and write, but the code has logic to remove old state based on event time. * Not sure

Re: Broadcast state corrupted ?

2022-04-11 Thread Chesnay Schepler
Am I understanding things correctly in that the same savepoint cannot be restored from in 1 environment, while it works fine in 3 others? If so, are they all relying on the same file, or copies of the savepoint? On 10/04/2022 22:39, Alexey Trenikhun wrote: Hello, We have KeyedBroadcastProcessFu

Re: Unsubscribe

2022-04-11 Thread Chesnay Schepler
To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org.

Re: Unsubscribed

2022-04-11 Thread Chesnay Schepler
To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org.

Re: Flink Kafka Issue with EXACTLY_ONCE semantics

2022-04-11 Thread Frank Dekervel
Hello Praneeth, that looks correct then. In that case, maybe somebody else can chime in. Are you aware of this post on SO: https://stackoverflow.com/questions/45047876/apache-kafka-exactly-once-implementation-not-sending-messages Frank On Sat, Apr 9, 2022 at 7:42 PM Praneeth Ramesh wrote: >