Wang Yuan created SPARK-35212:
---------------------------------
Summary: Spark Streaming LocationStrategy should provide a random
option that mapping kafka partitions randomly to spark executors
Key: SPARK-35212
URL: https://issues.apache.org/jira/browse/SPARK-35212
Project: Spark
Issue Type: New Feature
Components: DStreams
Affects Versions: 3.1.1
Reporter: Wang Yuan
There are three LocationStrategy: PreferBrokers, PreferConsistent, PreferFixed.
I got a scenario that I need a random one. There are plenty of topic partitions
that are varies from each other with different records inside. And I have a lot
of executors. PreferBrokers does not help here. PreferConsistent will make
things worse that some executor will always get heavy tasks. PreferFixed does
not help too, because it is fixed, neither to say I have to create a mapping
manually.
A random LocationStrategy should dispatch a topic partition to different
executors in different window. This would balance the load among spark
executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
