My usecase is that as soon as the avro message version is changed, I want to reload the job graph so that I can update the downstream iceberg table.
Iceberg FlinkSink take table schema during the job start and cannot be updated during runtime. So, I want to trigger graceful shutdown and restart the job. Can I reload the job graph to achieve that? On Wed, Dec 8, 2021 at 8:11 PM Arvid Heise <ar...@apache.org> wrote: > Hi Ayush, > > DeserializationSchema.isEndOfStream was only ever supported by Kafka. For > new Kafka source, the recommended way is to use the bounded mode like this > > KafkaSource<PartitionAndValue> source = > KafkaSource.<PartitionAndValue>builder() > ... > .setStartingOffsets(OffsetsInitializer.earliest()) > .setBounded(OffsetsInitializer.latest()) > .build(); > > You can implement your own OffsetsInitializer or use a provided one. > > On Wed, Dec 8, 2021 at 9:19 AM Hang Ruan <ruanhang1...@gmail.com> wrote: > >> There is no way to end the kafka stream from the deserializer. >> >> When would you want to end the stream? Could you explain why you need to >> end the kafka stream without using the offset? >> >> Ayush Chauhan <ayush.chau...@zomato.com> 于2021年12月8日周三 15:29写道： >> >>> >>> https://github.com/apache/flink/blob/release-1.13.1/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/deserializer/KafkaRecordDeserializationSchema.java#L69 >>> >>> >>> >>> On Wed, Dec 8, 2021 at 12:40 PM Robert Metzger <metrob...@gmail.com> >>> wrote: >>> >>>> Hi Ayush, >>>> >>>> I couldn't find the documentation you've mentioned. Can you send me a >>>> link to it? >>>> >>>> On Tue, Dec 7, 2021 at 10:40 AM Ayush Chauhan <ayush.chau...@zomato.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Can you please let me know the alternatives of isEndOfStream() as now >>>>> according to docs this method will no longer be used to determine the end >>>>> of the stream. >>>>> >>>>> -- >>>>> Ayush Chauhan >>>>> Data Platform >>>>> [image: mobile-icon] +91 9990747111 >>>>> >>>>> >>>>> This email is intended only for the person or the entity to whom it is >>>>> addressed. If you are not the intended recipient, please delete this email >>>>> and contact the sender. >>>>> >>>> >>> >>> -- >>> Ayush Chauhan >>> Data Platform >>> [image: mobile-icon] +91 9990747111 >>> >>> >>> This email is intended only for the person or the entity to whom it is >>> addressed. If you are not the intended recipient, please delete this email >>> and contact the sender. >>> >> -- Ayush Chauhan Data Platform [image: mobile-icon] +91 9990747111 -- This email is intended only for the person or the entity to whom it is addressed. If you are not the intended recipient, please delete this email and contact the sender.
