Thank you for the response. But the reference does not seem to be answering any of those questions.
BS Lingzhe Sun From: ashok34...@yahoo.com Date: 2023-01-29 04:01 To: User; Lingzhe Sun Subject: Re: spark+kafka+dynamic resource allocation Hi, Worth checking this link https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun <lingzhe....@hirain.com> wrote: Hi all, I'm wondering if dynamic resource allocation works in spark+kafka streaming applications. Here're some questions: Will structured streaming be supported? Is the number of consumers always equal to the number of the partitions of subscribed topic (let's say there's only one topic)? If consumers is evenly distributed across executors, will newly added executor(through dynamic resource allocation) trigger a consumer reassignment? Would it be simply a bad idea to use dynamic resource allocation in streaming app, because there's no way to scale down number of executors unless no data is coming in? Any thoughts are welcomed. Lingzhe Sun Hirain Technology
