[jira] [Created] (SPARK-31908) Spark UI shows wrong driver memory configurations if driver memory is provided at run time

[Rahul Kumar (Jira)]

Rahul Kumar created SPARK-31908:
---

 Summary: Spark UI shows wrong driver memory configurations if 
driver memory is provided at run time
 Key: SPARK-31908
 URL: https://issues.apache.org/jira/browse/SPARK-31908
 Project: Spark
  Issue Type: Bug
  Components: Web UI
Affects Versions: 2.4.3
Reporter: Rahul Kumar


I discovered that in cluster mode , when driver memory is provided via 
spark.driver.memory configuration at run time after creating spark session, 
spark doesn't pick this configurations at run time as application master is 
already launched by that time and picks the default spark driver memory 
configuration(1GB). 
However, on spark UI page, in environment tab, it still shows driver memory as 
the value passed via configurations at run time, which makes identifying and 
debugging this scenario more difficult. Driver memory should be shown as the 
value which spark is actually using in job.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-6646) Spark 2.0: Rearchitecting Spark for Mobile Platforms

[Rahul Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390169#comment-14390169
 ] 

Rahul Kumar commented on SPARK-6646:


Love this idea, what about private cloud in pocket :-) store data on smart 
phone, do processing on it, small mobile based web server that power cool 
visualization reports. Lot of time our smart phones are idle we can share 
resources :-) 4 GB RAM, quadcore processer, LTE network not bad for a single 
node in cluster.

 Spark 2.0: Rearchitecting Spark for Mobile Platforms
 

 Key: SPARK-6646
 URL: https://issues.apache.org/jira/browse/SPARK-6646
 Project: Spark
  Issue Type: Improvement
  Components: Project Infra
Reporter: Reynold Xin
Assignee: Reynold Xin
Priority: Blocker
 Attachments: Spark on Mobile - Design Doc - v1.pdf


 Mobile computing is quickly rising to dominance, and by the end of 2017, it 
 is estimated that 90% of CPU cycles will be devoted to mobile hardware. 
 Spark’s project goal can be accomplished only when Spark runs efficiently for 
 the growing population of mobile users.
 Designed and optimized for modern data centers and Big Data applications, 
 Spark is unfortunately not a good fit for mobile computing today. In the past 
 few months, we have been prototyping the feasibility of a mobile-first Spark 
 architecture, and today we would like to share with you our findings. This 
 ticket outlines the technical design of Spark’s mobile support, and shares 
 results from several early prototypes.
 Mobile friendly version of the design doc: 
 https://databricks.com/blog/2015/04/01/spark-2-rearchitecting-spark-for-mobile.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org