Hi, community Right now i am using the batch to kafka in pyspark to send dataframe into kafka https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#writing-the-output-of-batch-queries-to-kafka
The solution works with not so big dataframe. While the dataframe is big, since the write to kafka is synchronize, it doesn't batch data at all, so each call to kafka is only 1-2 records (basically 1-2 row in the dataframe), and the performance is pretty bad. I even set kafka.batch.size to 500k, but i don't think it is working due to synchronous producing. I see a lot of changes happening on the structure streaming side, but batch to kafka seems not much changes, or at least i didn't find any. Is there anyway i can make the write to kafka asynchronous, or is there a way to make the producer batch more data that the config i missed? Thank you. On Wed, Jul 31, 2024 at 2:38 PM Rommel Holmes <rommelhol...@gmail.com> wrote: > Hi, community > > Right now i am using the batch to kafka in pyspark to send dataframe into > kafka > https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#writing-the-output-of-batch-queries-to-kafka > > > The solution works with not so big dataframe. > While the dataframe is big, since the write to kafka is synchronize, it > doesn't batch data at all, so each call to kafka is only 1-2 > records (basically 1-2 row in the dataframe), and the performance is pretty > bad. I even set kafka.batch.size to 500k, but i don't think it is working > due to synchronous producing. > > I see a lot of changes happening on the structure streaming side, but > batch to kafka seems not much changes, or at least i didn't find any. > > Is there anyway i can make the write to kafka asynchronous, or is there a > way to make the producer batch more data that the config i missed? > > Thank you. > > > -- > Yours > Rommel > > -- Yours Rommel ************************************* I waited patiently for the LORD; he turned to me and heard my cry. He lifted me out of the slimy pit, out of the mud and mire; he set my feet on a rock and gave me a firm place to stand. *************************************
