Unsubscribe

[Frank Luo]

Unsubscribe

Re: Ammonite as REPL for Spark Connect

[Holden Karau]

I am +1 to the general concept of including Ammonite magic 彩.

On Wed, Mar 22, 2023 at 4:58 PM Herman van Hovell
 wrote:

> Ammonite is maintained externally by Li Haoyi et al. We are including it
> as a 'provided' dependency. The integration bits and pieces (1 file) are
> included in Apache Spark.
>
> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
> wrote:
>
>>
>> Will this be maintained externally or included into Apache Spark ?
>>
>> Regards ,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>  wrote:
>>
>>> Hi All,
>>>
>>> For Spark Connect Scala Client we are working on making the REPL
>>> experience a bit nicer . In
>>> a nutshell we want to give users a turn key scala REPL, that works even if
>>> you don't have a Spark distribution on your machine (through coursier
>>> ). We are using Ammonite
>>>  instead of the standard scala REPL for this, the
>>> main reason for going with Ammonite is that it is easier to customize, and
>>> IMO has a superior user experience.
>>>
>>> Does anyone object to doing this?
>>>
>>> Kind regards,
>>> Herman
>>>
>>> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Ammonite as REPL for Spark Connect

[Herman van Hovell]

Ammonite is maintained externally by Li Haoyi et al. We are including it as
a 'provided' dependency. The integration bits and pieces (1 file) are
included in Apache Spark.

On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
wrote:

>
> Will this be maintained externally or included into Apache Spark ?
>
> Regards ,
> Mridul
>
>
>
> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>  wrote:
>
>> Hi All,
>>
>> For Spark Connect Scala Client we are working on making the REPL
>> experience a bit nicer . In
>> a nutshell we want to give users a turn key scala REPL, that works even if
>> you don't have a Spark distribution on your machine (through coursier
>> ). We are using Ammonite 
>> instead of the standard scala REPL for this, the main reason for going with
>> Ammonite is that it is easier to customize, and IMO has a superior user
>> experience.
>>
>> Does anyone object to doing this?
>>
>> Kind regards,
>> Herman
>>
>>

Re: Ammonite as REPL for Spark Connect

[Mridul Muralidharan]

Will this be maintained externally or included into Apache Spark ?

Regards ,
Mridul



On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
 wrote:

> Hi All,
>
> For Spark Connect Scala Client we are working on making the REPL
> experience a bit nicer . In a
> nutshell we want to give users a turn key scala REPL, that works even if
> you don't have a Spark distribution on your machine (through coursier
> ). We are using Ammonite 
> instead of the standard scala REPL for this, the main reason for going with
> Ammonite is that it is easier to customize, and IMO has a superior user
> experience.
>
> Does anyone object to doing this?
>
> Kind regards,
> Herman
>
>

Ammonite as REPL for Spark Connect

[Herman van Hovell]

Hi All,

For Spark Connect Scala Client we are working on making the REPL experience
a bit nicer . In a nutshell we
want to give users a turn key scala REPL, that works even if you don't have
a Spark distribution on your machine (through coursier
). We are using Ammonite 
instead of the standard scala REPL for this, the main reason for going with
Ammonite is that it is easier to customize, and IMO has a superior user
experience.

Does anyone object to doing this?

Kind regards,
Herman

Re: Graceful stop for spark streaming.

[Mich Talebzadeh]

Read this

[SPARK-42485] SPIP: Shutting down spark structured streaming when the
streaming process completed current process - ASF JIRA (apache.org)


HTH

Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 22 Mar 2023 at 00:38, Srinivas Rishindra 
wrote:

> Hi Team,
>
> Does anyone know about any ongoing work to gracefully stop spark streaming
> applications externally using "yarn application -stop" or by using any
> other method?
>
> My understanding is that in order to make use of "yarn application -stop",
> spark needs to implement "org.apache.hadoop.yarn.client.api.AppAdminClient".
> Does anyone have any thoughts or knowledge about this?
>
> Best Regards
> Rishi
>
>
>
>