Hi all, I'm wondering if dynamic resource allocation works in spark+kafka streaming applications. Here're some questions: Will structured streaming be supported? Is the number of consumers always equal to the number of the partitions of subscribed topic (let's say there's only one topic)? If consumers is evenly distributed across executors, will newly added executor(through dynamic resource allocation) trigger a consumer reassignment? Would it be simply a bad idea to use dynamic resource allocation in streaming app, because there's no way to scale down number of executors unless no data is coming in? Any thoughts are welcomed.
Lingzhe Sun Hirain Technology
