Re: Scheduling jobs using FAIR pool

2024-04-02 Thread Varun Shah
. Regards, Varun Shah On Mon, Apr 1, 2024, 18:50 Hussein Awala wrote: > IMO the questions are not limited to Databricks. > > > The Round-Robin distribution of executors only work in case of empty > executors (achievable by enabling dynamic allocation). In case the jobs > (par

Re: Scheduling jobs using FAIR pool

2024-04-01 Thread Varun Shah
Hi Mich, I did not post in the databricks community, as most of the questions were related to spark itself. But let me also post the question on databricks community. Thanks, Varun Shah On Mon, Apr 1, 2024, 16:28 Mich Talebzadeh wrote: > Hi, > > Have you put this question to Databri

Scheduling jobs using FAIR pool

2024-03-31 Thread Varun Shah
scaling down is an issue (known already for which SPIP is already in discussion, but it does not cater to submitting multiple streams in a single cluster. Thanks for reading !! Looking forward to your suggestions Regards, Varun Shah

Re: A proposal for creating a Knowledge Sharing Hub for Apache Spark Community

2024-03-19 Thread Varun Shah
d answered by the community over a period of time and hence, if possible, we could leverage it as the starting point for building a community before creating a complete new website from scratch. Any thoughts on this? Regards, Varun Shah On Mon, Mar 18, 2024, 16:29 Mich Talebzadeh wrote: > So

Re: Pyspark Write Batch Streaming Data to Snowflake Fails with more columns

2024-02-10 Thread Varun Shah
tion link : https://docs.snowflake.com/en/user-guide/spark-connector-use#working-with-timestamps-and-time-zones Thank you once again for your help. Regards, Varun Shah On Sat, Feb 10, 2024, 04:01 Mich Talebzadeh wrote: > Hi Varun, > > I am no expert on Snowflake, however, the issue y

Pyspark Write Batch Streaming Data to Snowflake Fails with more columns

2024-02-09 Thread Varun Shah
searching online, but did not get any such articles. Looking forward to hearing from you. Regards, Varun Shah

Re: Spark Scala SBT Local build fails

2023-07-18 Thread Varun Shah
++ DEV community On Mon, Jul 17, 2023 at 4:14 PM Varun Shah wrote: > Resending this message with a proper Subject line > > Hi Spark Community, > > I am trying to set up my forked apache/spark project locally for my 1st > Open Source Contribution, by building and creating a pa