[
https://issues.apache.org/jira/browse/SPARK-23497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410858#comment-16410858
]
Felix Cheung commented on SPARK-23497:
--------------------------------------
you should probably follow up with sparklyr/rstudio on this.
> Sparklyr Applications doesn't disconnect spark driver in client mode
> --------------------------------------------------------------------
>
> Key: SPARK-23497
> URL: https://issues.apache.org/jira/browse/SPARK-23497
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 2.1.0
> Reporter: bharath kumar
> Priority: Major
>
> Hello,
> When we use Sparklyr to connect to Yarn cluster manager in client mode or
> cluster mode, Spark driver will not disconnect unless we mention the
> spark_disconnect(sc) in the code.
> Does it make sense to add a timeout feature for driver to exit after certain
> amount of time, in client mode or cluster mode. I think its only happening
> with connection from Sparklyr to Yarn. Some times the driver stays there for
> weeks and holds minimum resources .
> *More Details:*
> Yarn -2.7.0
> Spark -2.1.0
> Rversion:
> Microsoft R Open 3.4.2
> Rstudio Version:
> rstudio-server-1.1.414-1.x86_64
> yarn application -status application_id
> 18/01/22 09:08:45 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM
> address to resourcemanager.com/resourcemanager:8032
>
> Application Report :
> Application-Id : application_id
> Application-Name : sparklyr
> Application-Type : SPARK
> User : userid
> Queue : root.queuename
> Start-Time : 1516245523965
> Finish-Time : 0
> Progress : 0%
> State : RUNNING
> Final-State : UNDEFINED
> Tracking-URL : N/A
> RPC Port : -1
> AM Host : N/A
> Aggregate Resource Allocation :266468 MB-seconds, 59 vcore-seconds
> Diagnostics : N/A
>
> [http://spark.rstudio.com/]
>
> I can provide more details if required
>
> Thanks,
> Bharath
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
