Re: Re: spark+kafka+dynamic resource allocation
Sure, I suggest that you add a note to that Jira and express your interest. HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 30 Jan 2023 at 01:56, Lingzhe Sun wrote: > Hi Mich， > > Thanks for the information. I myself think this open issue should have > higher priority as more streaming application is being built using spark. > Hope this becomes a feature in the coming future. > > -- > BR > Lingzhe Sun > > > *From:* Mich Talebzadeh > *Date:* 2023-01-30 02:14 > *To:* Lingzhe Sun > *CC:* ashok34...@yahoo.com; User > *Subject:* Re: Re: spark+kafka+dynamic resource allocation > Hi, > > Spark Structured Streaming currently does not support dynamic allocation > (see SPARK-24815: Structured Streaming should support dynamic allocation > <https://issues.apache.org/jira/browse/SPARK-24815>). which is still open > > > Autoscaling in Cloud offerings > <https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/autoscaling#autoscaling_and_spark_streaming>like > Google Dataproc does not support Spark Structured Streaming either > > > HTH > > >view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Sun, 29 Jan 2023 at 01:49, Lingzhe Sun wrote: > >> Thank you for the response. But the reference does not seem to be >> answering any of those questions. >> >> BS >> Lingzhe Sun >> >> >> *From:* ashok34...@yahoo.com >> *Date:* 2023-01-29 04:01 >> *To:* User ; Lingzhe Sun >> *Subject:* Re: spark+kafka+dynamic resource allocation >> Hi, >> >> Worth checking this link >> >> >> https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation >> >> On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun < >> lingzhe@hirain.com> wrote: >> >> >> Hi all, >> >> I'm wondering if dynamic resource allocation works in spark+kafka >> streaming applications. Here're some questions: >> >>- Will structured streaming be supported? >>- Is the number of consumers always equal to the number of the >>partitions of subscribed topic (let's say there's only one topic)? >>- If consumers is evenly distributed across executors, will newly >>added executor(through dynamic resource allocation) trigger a consumer >>reassignment? >>- Would it be simply a bad idea to use dynamic resource allocation in >>streaming app, because there's no way to scale down number of executors >>unless no data is coming in? >> >> Any thoughts are welcomed. >> >> Lingzhe Sun >> Hirain Technology >> >>
Re: Re: spark+kafka+dynamic resource allocation
Hi Mich， Thanks for the information. I myself think this open issue should have higher priority as more streaming application is being built using spark. Hope this becomes a feature in the coming future. BR Lingzhe Sun From: Mich Talebzadeh Date: 2023-01-30 02:14 To: Lingzhe Sun CC: ashok34...@yahoo.com; User Subject: Re: Re: spark+kafka+dynamic resource allocation Hi, Spark Structured Streaming currently does not support dynamic allocation (see SPARK-24815: Structured Streaming should support dynamic allocation). which is still open Autoscaling in Cloud offerings like Google Dataproc does not support Spark Structured Streaming either HTH view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Sun, 29 Jan 2023 at 01:49, Lingzhe Sun wrote: Thank you for the response. But the reference does not seem to be answering any of those questions. BS Lingzhe Sun From: ashok34...@yahoo.com Date: 2023-01-29 04:01 To: User; Lingzhe Sun Subject: Re: spark+kafka+dynamic resource allocation Hi, Worth checking this link https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun wrote: Hi all, I'm wondering if dynamic resource allocation works in spark+kafka streaming applications. Here're some questions: Will structured streaming be supported? Is the number of consumers always equal to the number of the partitions of subscribed topic (let's say there's only one topic)? If consumers is evenly distributed across executors, will newly added executor(through dynamic resource allocation) trigger a consumer reassignment? Would it be simply a bad idea to use dynamic resource allocation in streaming app, because there's no way to scale down number of executors unless no data is coming in? Any thoughts are welcomed. Lingzhe Sun Hirain Technology
Re: Re: spark+kafka+dynamic resource allocation
Hi, Spark Structured Streaming currently does not support dynamic allocation (see SPARK-24815: Structured Streaming should support dynamic allocation <https://issues.apache.org/jira/browse/SPARK-24815>). which is still open Autoscaling in Cloud offerings <https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/autoscaling#autoscaling_and_spark_streaming>like Google Dataproc does not support Spark Structured Streaming either HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Sun, 29 Jan 2023 at 01:49, Lingzhe Sun wrote: > Thank you for the response. But the reference does not seem to be > answering any of those questions. > > BS > Lingzhe Sun > > > *From:* ashok34...@yahoo.com > *Date:* 2023-01-29 04:01 > *To:* User ; Lingzhe Sun > *Subject:* Re: spark+kafka+dynamic resource allocation > Hi, > > Worth checking this link > > > https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation > > On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun < > lingzhe@hirain.com> wrote: > > > Hi all, > > I'm wondering if dynamic resource allocation works in spark+kafka > streaming applications. Here're some questions: > >- Will structured streaming be supported? >- Is the number of consumers always equal to the number of the >partitions of subscribed topic (let's say there's only one topic)? >- If consumers is evenly distributed across executors, will newly >added executor(through dynamic resource allocation) trigger a consumer >reassignment? >- Would it be simply a bad idea to use dynamic resource allocation in >streaming app, because there's no way to scale down number of executors >unless no data is coming in? > > Any thoughts are welcomed. > > Lingzhe Sun > Hirain Technology > >
Re: Re: spark+kafka+dynamic resource allocation
Thank you for the response. But the reference does not seem to be answering any of those questions. BS Lingzhe Sun From: ashok34...@yahoo.com Date: 2023-01-29 04:01 To: User; Lingzhe Sun Subject: Re: spark+kafka+dynamic resource allocation Hi, Worth checking this link https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun wrote: Hi all, I'm wondering if dynamic resource allocation works in spark+kafka streaming applications. Here're some questions: Will structured streaming be supported? Is the number of consumers always equal to the number of the partitions of subscribed topic (let's say there's only one topic)? If consumers is evenly distributed across executors, will newly added executor(through dynamic resource allocation) trigger a consumer reassignment? Would it be simply a bad idea to use dynamic resource allocation in streaming app, because there's no way to scale down number of executors unless no data is coming in? Any thoughts are welcomed. Lingzhe Sun Hirain Technology
Re: spark+kafka+dynamic resource allocation
Hi,
Worth checking this link
https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun
wrote:
#yiv9684413148 body {line-height:1.5;}#yiv9684413148 ol, #yiv9684413148 ul
{margin-top:0px;margin-bottom:0px;list-style-position:inside;}#yiv9684413148
body {font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0,
0);line-height:1.5;}#yiv9684413148 body
{font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0,
0);line-height:1.5;}Hi all,
I'm wondering if dynamic resource allocation works in spark+kafka streaming
applications. Here're some questions:
- Will structured streaming be supported?
- Is the number of consumers always equal to the number of the partitions of
subscribed topic (let's say there's only one topic)?
- If consumers is evenly distributed across executors, will newly added
executor(through dynamic resource allocation) trigger a consumer reassignment?
- Would it be simply a bad idea to use dynamic resource allocation in
streaming app, because there's no way to scale down number of executors unless
no data is coming in?
Any thoughts are welcomed.
Lingzhe SunHirain Technology
