Hey all,
I’m building a Flink app that pulls in messages from a Kafka topic and writes
them out to disk using a custom bucketed sink. Each message needs to be parsed
using a schema that is also needed when writing in the sink. This schema is
read from a remote file on a distributed file system
Hi Julian,
Have you seen Broadcast State [1]? I have never used it personally, but it
sounds like something you want. Maybe your job should look like:
1. read raw messages from Kafka, without using the schema
2. read schema changes and broadcast them to 3. and 5.
3. deserialize kafka records in B
meditate on the docs further 🙂
Julian
From: Piotr Nowojski
Date: Wednesday, October 14, 2020 at 6:35 AM
To: "Jaffe, Julian"
Cc: "user@flink.apache.org"
Subject: Re: Broadcasting control messages to a sink
Hi Julian,
Have you seen Broadcast State [1]? I have never used i
t;
>
> Julian
>
>
>
> *From: *Piotr Nowojski
> *Date: *Wednesday, October 14, 2020 at 6:35 AM
> *To: *"Jaffe, Julian"
> *Cc: *"user@flink.apache.org"
> *Subject: *Re: Broadcasting control messages to a sink
>
>
>
> Hi Julian,
>
>
&
access the broadcast state within the sink, but perhaps I
> just haven’t thought about it the right way. I’ll meditate on the docs
> further 🙂
>
>
>
> Julian
>
>
>
> *From: *Piotr Nowojski
> *Date: *Wednesday, October 14, 2020 at 6:35 AM
> *To: *"Jaffe, Juli
: Thursday, October 15, 2020 at 4:12 AM
To: "Jaffe, Julian"
Cc: Piotr Nowojski , user
Subject: Re: Broadcasting control messages to a sink
Hi Jaffe,
I am also working on something similar type of a problem.
I am receiving a set of events in Avro format on different topics. I want to
con
Jaffe, Julian"
Cc: "user@flink.apache.org"
Subject: Re: Broadcasting control messages to a sink
Hi Julian,
I think the problem is that BroadcastProcessFunction and SinkFunction will be
executed by separate operators, so they won't be able to share state. If you
can not spl
---
>
>
>
> I hope this is helpful to someone.
>
>
>
> Julian
>
>
>
> *From: *Piotr Nowojski
> *Date: *Wednesday, October 14, 2020 at 11:22 PM
> *To: *"Jaffe, Julian"
> *Cc: *"user@flink.apache.org"
> *Subjec
Union --
>>
>> |
>>
>> ___
>>
>> | Sink |
>>
>> ---
>>
>>
>>
&g
fe, Julian" , "user@flink.apache.org"
Subject: Re: Broadcasting control messages to a sink
Hi Jafee,
Can u please help me out with the sample code how you have written the custom
sink and how you using this broadcast pattern to update schema at run time. It
will help me.
On Sat, Oct
10 matches
Mail list logo
