Flink falls back on to kryo serializer for GenericTypes
Hello, How to avoid flink's kryo serializer for GenericTypes ? Kryo is having some performance issues. Tried below but no luck. env.getConfig().disableForceKryo(); env.getConfig().enableForceAvro(); Tried this - env.getConfig().disableGenericTypes(); getting - Generic types have been disabled in the ExecutionConfig and type org.apache.avro.generic.GenericRecord is treated as a generic type Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States
Deserialize avro message without reader schema.
Hi, I'm trying to deserialize avro messages from the kafka topic as a consumer. As a kafka consumer, I do not want to hold the reader schema on my end. Is there a way to deserialize avro messages to GenericRecord without providing the reader schema ? Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States
Re: Flink config driven tool ?
You are just spamming the inbox by sending these emails. You can just ignore sending those emails if it isn't adding any value. Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States On Wed, Jun 15, 2022 at 1:41 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > I would have helped you if it was written in Scala instead of Java. > > On Wed, Jun 15, 2022 at 2:22 AM Rakshit Ramesh < > rakshit.ram...@datakaveri.org> wrote: > >> I'm working on such a thing. >> It's in early stages and needs a lot more work. >> I'm open to collaborating. >> https://github.com/datakaveri/iudx-adaptor-framework >> >> On Tue, 7 Jun 2022 at 23:49, sri hari kali charan Tummala < >> kali.tumm...@gmail.com> wrote: >> >>> Hi Flink Community, >>> >>> can someone point me to a good config-driven flink data movement tool >>> Github repos? Imagine I build my ETL dag connecting source --> >>> transformations --> target just using a config file. >>> >>> below are a few spark examples:- >>> https://github.com/mvrpl/big-shipper >>> https://github.com/BitwiseInc/Hydrograph >>> >>> Thanks & Regards >>> Sri Tummala >>> >>> > > -- > Thanks & Regards > Sri Tummala > >
Re: Apache flink doesn't work with avro kafka topic with multiple event types
Thanks, I'll check it out. On Mon, Jun 13, 2022 at 2:40 AM Qingsheng Ren wrote: > Hi Sucheth, > > If you are referring to Table / SQL API, I'm afraid it doesn't support > schema evolution or different types from one Kafka table. An > alternative way is to consume the topic with raw format [1] and do > deserialization with a UDTF. If you are using the DataStream API, you > can implement the KafkaRecordDeserializationSchema interface to > deserialize the message consumed from Kafka. > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/raw/ > > Best regards, > > Qingsheng > > On Sun, Jun 12, 2022 at 12:53 PM Sucheth S wrote: > > > > Hi, > > > > Apache Flink doesn't work with Avro serialized kafka topic with multiple > event types ( > > TopicRecordNameStrategy for subject ) > > > > Is there a way to read a generic record from avro serialized kafka topic > which can have messages with different schemas, basically > TopicRecordNameStrategy for the subject in the schema registry. ? > > > > > > Regards, > > Sucheth Shivakumar > > website : https://sucheths.com > > mobile : +1(650)-576-8050 > > San Mateo, United States > -- Regards, Sucheth Shivakumar website: https://sucheths.com Mobile : +1(650)-576-8050 San Mateo, United States
Apache flink doesn't work with avro kafka topic with multiple event types
Hi, Apache Flink doesn't work with Avro serialized kafka topic with multiple event types ( TopicRecordNameStrategy for subject ) Is there a way to read a generic record from avro serialized kafka topic which can have messages with different schemas, basically TopicRecordNameStrategy for the subject in the schema registry. ? Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States
Re: filesink part files roll over
Thank you! On Mon, Jun 6, 2022 at 2:57 AM Qingsheng Ren wrote: > Hi Sucheth, > > Please see https://issues.apache.org/jira/browse/FLINK-27910 > > Best, > > Qingsheng > > > On Jun 5, 2022, at 23:21, Sucheth S wrote: > > > > Hi, > > > > Can someone please help me with this please - > https://stackoverflow.com/q/72496963/9125940 ? > > > > Regards, > > Sucheth Shivakumar > > website : https://sucheths.com > > mobile : +1(650)-576-8050 > > San Mateo, United States > > -- Regards, Sucheth Shivakumar website: https://sucheths.com Mobile : +1(650)-576-8050 San Mateo, United States
Re: Need help to join Apache Flink community on.Slack
I got it, thanks! Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States On Sun, Jun 5, 2022 at 6:48 PM Jing Ge wrote: > Hi Sucheth, > > Just invited you, please check. Thanks! > > Best Regards, > Jing > > On Sun, Jun 5, 2022 at 6:06 PM Sucheth S wrote: > >> Hello Jing, >> >> Can you please add me - suchet...@gmail.com >> >> >> On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala < >> kali.tumm...@gmail.com> wrote: >> >>> Hi Jing, >>> >>> Please add me kali.tumm...@gmail.com. >>> >>> Thanks >>> Sri >>> >>> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: >>> >>>> Hi Santhosh, >>>> >>>> just invited you. Please check your email. Looking forward to knowing >>>> your story! Thanks! >>>> >>>> To anyone else who wants to join, please send an email to >>>> user@flink.apache.org, you might have a better chance to get the >>>> invite. Thanks. >>>> >>>> Regards, >>>> Jing >>>> >>>> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat < >>>> santhoshvenkat1...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Can you please invite me to join the apache flink slack community >>>>> channel. We have adopted apache flink and would like to participate in the >>>>> community forum. >>>>> >>>>> Thank you. >>>>> >>>>> Regards >>>>> >>>> >>> >>> -- >>> Thanks & Regards >>> Sri Tummala >>> >>> -- >> Regards, >> Sucheth Shivakumar >> website: https://sucheths.com >> Mobile : +1(650)-576-8050 >> San Mateo, United States >> >
Re: Need help to join Apache Flink community on.Slack
Hello Jing, Can you please add me - suchet...@gmail.com On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Jing, > > Please add me kali.tumm...@gmail.com. > > Thanks > Sri > > On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: > >> Hi Santhosh, >> >> just invited you. Please check your email. Looking forward to knowing >> your story! Thanks! >> >> To anyone else who wants to join, please send an email to >> user@flink.apache.org, you might have a better chance to get the invite. >> Thanks. >> >> Regards, >> Jing >> >> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat < >> santhoshvenkat1...@gmail.com> wrote: >> >>> Hi, >>> >>> Can you please invite me to join the apache flink slack community >>> channel. We have adopted apache flink and would like to participate in the >>> community forum. >>> >>> Thank you. >>> >>> Regards >>> >> > > -- > Thanks & Regards > Sri Tummala > > -- Regards, Sucheth Shivakumar website: https://sucheths.com Mobile : +1(650)-576-8050 San Mateo, United States
filesink part files roll over
Hi, Can someone please help me with this please - https://stackoverflow.com/q/72496963/9125940 ? Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States
