spark+kafka+dynamic resource allocation

Hi all, I'm wondering if dynamic resource allocation works in spark+kafka streaming applications. Here're some questions: Will structured streaming be supported? Is the number of consumers always equal to the number of the partitions of subscribed topic (let's say there's only one topic)? If

Spark SQL question

this SQL works select 1 as *`data.group`* from tbl group by *data.group* Since there's no such field as *data,* I thought the SQL has to look like this select 1 as *`data.group`* from tbl group by `*data.group`* But that gives and error (cannot resolve '`data.group`') ... I'm no expert in

Re: Question regarding Spark 3.X performance

Hi Mich, I'll gather them and send them to you :) Many thanks, Thanasis Στις Παρ 27 Ιαν 2023 στις 11:40 π.μ., ο/η Mich Talebzadeh < mich.talebza...@gmail.com> έγραψε: > > Hi Athanasios > > > Thanks for the details. Since I believe this is Spark streaming, the all > important indicator is the

Re: Question regarding Spark 3.X performance

Hi Athanasios Thanks for the details. Since I believe this is Spark streaming, the all important indicator is the Processing Time defined by Spark GUI as Time taken to process all jobs of a batch versus the batch interval. The Scheduling Delay and the Total Delay are additional indicators of