Re: Flink Kafka EXACTLY_ONCE causing KafkaException ByteArraySerializer is not an instance of Serializer
Has anyone had success with using exactly_once in a kafka producer in flink? As of right now I don't think the code shown in the docs (https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-producer) actually works. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink Kafka EXACTLY_ONCE causing KafkaException ByteArraySerializer is not an instance of Serializer
Hi Vikash, It's a bug about classloader used in `abortTransaction()` method in `FlinkKafkaProducer`, Flink version 1.10.0. I think it has been fixed in 1.10.1 and 1.11 according to FLINK-16262. Are you using Flink version 1.10.0? Vikash Dat 于2020年7月30日周四 下午9:26写道: > Has anyone had success with using exactly_once in a kafka producer in > flink? > As of right now I don't think the code shown in the docs > ( > https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-producer > ) > actually works. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ > -- Best Regards, *Qingsheng Ren* Electrical and Computer Engineering Carnegie Mellon University Email: renqs...@gmail.com
Re: Flink Kafka EXACTLY_ONCE causing KafkaException ByteArraySerializer is not an instance of Serializer
Thanks for the reply. I am currently using 1.10 but also saw it happens in 1.10.1 when experimenting. I have not tried 1.11 since EMR only has up to 1.10 at the moment. Are there any known work arounds? On Fri, Jul 31, 2020 at 02:42 Qingsheng Ren wrote: > Hi Vikash, > > It's a bug about classloader used in `abortTransaction()` method in > `FlinkKafkaProducer`, Flink version 1.10.0. I think it has been fixed in > 1.10.1 and 1.11 according to FLINK-16262. Are you using Flink version > 1.10.0? > > > Vikash Dat 于2020年7月30日周四 下午9:26写道: > >> Has anyone had success with using exactly_once in a kafka producer in >> flink? >> As of right now I don't think the code shown in the docs >> ( >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-producer >> ) >> actually works. >> >> >> >> -- >> Sent from: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >> > > > -- > Best Regards, > > *Qingsheng Ren* > > Electrical and Computer Engineering > Carnegie Mellon University > > Email: renqs...@gmail.com >
Re: Flink Kafka EXACTLY_ONCE causing KafkaException ByteArraySerializer is not an instance of Serializer
Hi Vikash, Sorry for the late reply. Is your version of Flink kafka *connector* 1.10.1 too? Actually it's a bug in the connector, so I think you need to upgrade the connector to 1.10.1 too, not just Flink itself. I tried Flink 1.10.0/1.10.1 + flink-kafka-connector 1.10.0 and indeed reproduced the bug. After upgrading flink-kafka-connector to 1.10.1, the error disappeared. On Fri, Jul 31, 2020 at 7:02 PM Vikash Dat wrote: > Thanks for the reply. I am currently using 1.10 but also saw it happens in > 1.10.1 when experimenting. I have not tried 1.11 since EMR only has up to > 1.10 at the moment. Are there any known work arounds? > > On Fri, Jul 31, 2020 at 02:42 Qingsheng Ren wrote: > >> Hi Vikash, >> >> It's a bug about classloader used in `abortTransaction()` method in >> `FlinkKafkaProducer`, Flink version 1.10.0. I think it has been fixed in >> 1.10.1 and 1.11 according to FLINK-16262. Are you using Flink version >> 1.10.0? >> >> >> Vikash Dat 于2020年7月30日周四 下午9:26写道: >> >>> Has anyone had success with using exactly_once in a kafka producer in >>> flink? >>> As of right now I don't think the code shown in the docs >>> ( >>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-producer >>> ) >>> actually works. >>> >>> >>> >>> -- >>> Sent from: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >>> >> >> >> -- >> Best Regards, >> >> *Qingsheng Ren* >> >> Electrical and Computer Engineering >> Carnegie Mellon University >> >> Email: renqs...@gmail.com >> > -- Best Regards, *Qingsheng Ren* Electrical and Computer Engineering Carnegie Mellon University Email: renqs...@gmail.com