I have Spark code that writes a batch to Kafka as specified here:
https://spark.apache.org/docs/2.4.0/structured-streaming-kafka-integration.html
The code looks like the following:
df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
\
.write \
.format("kafka") \
.option("kafka.bootstrap.servers",
"host1:port1,host2:port2") \
.option("topic", "topic1") \
.save()
However the data only gets written to Kafka partition 0. How can I get it
written uniformly to all partitions in the same topic ?
Thanks in advance,
-- Femi
http://dataphantik.com
"Great spirits have always encountered violent opposition from mediocre
minds." - Albert Einstein.