Re: Table API: Custom Kafka formats for Confluent Protobuf and JSON

Hey Yaroslav!, I suppose I will try it like this. The lookup would indeed be nice too, I will have a closer look at the corresponding source code. Thanks! -Theo > On 29. Nov 2022, at 17:41, Yaroslav Tkachenko wrote: > > Hey Theodor, > > That's pretty much it, assuming your Protobuf schema is

Re: Flink Table Kinesis sink not failing when sink fails

My text logs don't have a stack trace with this exception. I'm doing this inside Flink SQL with a standard Kinesis connector and JSON formatter. On Tue, Nov 29, 2022 at 6:38 PM yuxia wrote: > Which code line the error message happens? Maybe it will swallow the > exception and then log the error

Re: Query about flink job manager dashboard

After disabling the cancel, submit flags facing issues with below api calls. 1) /jars giving 404 2) /jars/upload 3) /jars/{jarid}/run Is there any config changes needed to have these apis work? On Mon, 28 Nov, 2022, 7:00 PM naga sudhakar, wrote: > Hi, > We are able to disable this cancela nd

Re: Flink Table Kinesis sink not failing when sink fails

Which code line the error message happens? Maybe it will swallow the exception and then log the error message, in which case Flink job won't fail since it seems like no exception happens. Best regards, Yuxia 发件人: "Dan Hill" 收件人: "User" 发送时间: 星期三, 2022年 11 月 30日 上午 8:06:52 主题: Flink Tab

Flink Table Kinesis sink not failing when sink fails

I set up a simple Flink SQL job (Flink v1.14.4) that writes to Kinesis. The job looks healthy but the records are not being written. I did not give enough IAM permissions to write to Kinesis. However, the Flink SQL job acts like it's healthy and checkpoints even though the Kinesis PutRecords call

Re: "An illegal reflective access operation has occurred" during KeyedStream process

I changed the ".map(...)" and ".print()" terminal statement to : .executeAndCollect() .forEachRemaining(System.out::println); The warnings were replaced with: WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.twitter.chill.java.ArraysAsListSeria

"An illegal reflective access operation has occurred" during KeyedStream process

Hello, Using Flink version 1.15.0, I recieve these warnings when trying a small example (code below): WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/Users/cjensen/data/tools/apache-maven-3.3.9/m2/or

Re: Flink 1.15.3 Docker image

Done, please let me know if you see anything unexpected. On Tue, Nov 29, 2022 at 7:07 PM Márton Balassi wrote: > Hi Ben, > > Thanks for reaching out. Since the image repo has been updated [1] I can > pick this up. Will let you know when done. > > [1] > https://github.com/apache/flink-docker/comm

Re: Flink 1.15.3 Docker image

Hi Ben, Thanks for reaching out. Since the image repo has been updated [1] I can pick this up. Will let you know when done. [1] https://github.com/apache/flink-docker/commit/a22c0f04972a1d8539d9213b52fc0728eac8c1fa On Tue, Nov 29, 2022 at 4:28 PM Roberts, Ben (Senior Developer) via user < user@f

Re: Table API: Custom Kafka formats for Confluent Protobuf and JSON

Hey Theodor, That's pretty much it, assuming your Protobuf schema is more or less fixed. But for a production workload, you'd need to add a Schema Registry lookup. I guess the implementation for that would be similar to what's in the Avro format. On Tue, Nov 29, 2022 at 2:26 AM Theodor Wübker wr

Flink 1.15.3 Docker image

Hi, Is there anyone able to publish the apache/flink docker image for release 1.15.3? I see that the release was announced on 2022-11-10, and the PR for the 1.15.3 dockerfiles (linked: https://github.com/apache/flink-docker/pull/140) has been approved and merged on 2022-11-25, but it looks lik

[SUMMARY] Flink 1.17 Release Sync 11/29/2022

Hi devs and users, I'd like to share some highlights from the release sync on 11/29/2022. 1. @Contributors please update your progress on the release 1.17 wiki page [1] before the sync meeting so that everyone could track it. 2. We have new CI stability tickets and owners should have been pinged

Table API: Custom Kafka formats for Confluent Protobuf and JSON

Hey all, so Confluent has Kafka serializers to serialize Protobuf, Avro and JSON that create messages with a magic byte followed by a 4 byte schema id followed by the actual payload (refer the docs

Re: Savepoint restore mode for the Kubernetes operator

Just to be clear, I don't think the operator must have special logic to find out if a savepoint was used as base for an incremental checkpoint, however, the operator logic might want to completely disable savepoint cleanup for a deployment if the user enabled CLAIM mode for it. At least that sounds

Re: Savepoint restore mode for the Kubernetes operator

The operator might call dispose on an old savepoint that’s true, but I am not sure if the dispose api call would actually corrupt it. Gyula On Tue, 29 Nov 2022 at 09:28, Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Hangxiang, > > but, if I understand correctly, setting restore m

Re: Savepoint restore mode for the Kubernetes operator

Hi Hangxiang, but, if I understand correctly, setting restore mode to CLAIM means that the job might create a new incremental checkpoint based on the savepoint, right? And if the operator then decides to clean up the savepoint, the checkpoint would be corrupted, no? Regards, Alexis. Am Mo., 28.