Re: Spark streaming job for kafka transaction does not consume read_committed messages correctly.

2024-04-14 Thread Kidong Lee
on_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Sun, 14 Apr 2024 at 13:40, Kidong Lee wrote: > >> >> Because spark streaming for kafk transaction does not work correctly to >> suit my need, I moved to another ap

Re: Spark streaming job for kafka transaction does not consume read_committed messages correctly.

2024-04-13 Thread Kidong Lee
// commit offset. consumer.commitAsync(offsetMap, null); } }); In addition, I increased max.poll.records to 10. Even if this raw kafka consumer approach is not so scalable, it consumes read_committed messages from kafka correctly and is enough for me at the moment. - Kidong. 2024년 4월 12일 (금

Re: Spark streaming job for kafka transaction does not consume read_committed messages correctly.

2024-04-13 Thread Kidong Lee
t of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/

Spark streaming job for kafka transaction does not consume read_committed messages correctly.

2024-04-12 Thread Kidong Lee
action does not handle such as kafka consumer properties like isolation.level=read_committed and enable.auto.commit=false correctly. Any help appreciated. - Kidong. -- *이기동 * *Kidong Lee* Email: mykid...@gmail.com Chango: https://cloudcheflabs.github.io/chango-private-docs Web Site: http://www.clo

Re: spark thrift server as hive on spark running on kubernetes, and more.

2021-12-14 Thread Kidong Lee
Kubernetes in cluster mode using DataRoaster spark operator. Please see my blog how to do so: https://t.co/T3SXG0mZFB Cheers, - Kidong Lee 2021년 9월 10일 (금) 오전 8:38, Kidong Lee 님이 작성: > Hi, > > Recently, I have open-sourced a tool called DataRoaster( > https://github.com/cloudcheflabs

spark thrift server as hive on spark running on kubernetes, and more.

2021-09-09 Thread Kidong Lee
. - see also demo https://github.com/cloudcheflabs/dataroaster#dataroaster-demo Thank you. - Kidong Lee.