[jira] [Commented] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-27 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304012#comment-15304012
 ] 

Sean Owen commented on SPARK-15579:
---

Is the issue that the RDD will be GCed at some point in the near future and 
will be unpersisted? that could explain all the behavior here. But that is by 
design. If you can reproduce it when holding a ref to it, that's a clear 
problem.

> SparkUI: Storage page is empty even if things are cached
> 
>
> Key: SPARK-15579
> URL: https://issues.apache.org/jira/browse/SPARK-15579
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core, Web UI
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>
> scala> sc.parallelize(1 to 1, 5000).cache().count()
> SparkUI storage page is empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-27 Thread Dongjoon Hyun (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303786#comment-15303786
 ] 

Dongjoon Hyun commented on SPARK-15579:
---

Also, in Executors page, the memory usage shows only the size of `rdd1`.

> SparkUI: Storage page is empty even if things are cached
> 
>
> Key: SPARK-15579
> URL: https://issues.apache.org/jira/browse/SPARK-15579
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core, Web UI
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>
> scala> sc.parallelize(1 to 1, 5000).cache().count()
> SparkUI storage page is empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-27 Thread Dongjoon Hyun (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303784#comment-15303784
 ] 

Dongjoon Hyun commented on SPARK-15579:
---

Hi, [~andrewor14]. 

The dangling RDD seems to be removed.

For the following testcases, I can see both at the first. But, after refreshing 
Storage page multiple times (about 1 minutes), `rdd2` is gone.
{code}
scala> val rdd1 = sc.parallelize(1 to 1, 5000).setName("rdd1").cache()
rdd1: org.apache.spark.rdd.RDD[Int] = rdd1 ParallelCollectionRDD[0] at 
parallelize at :24

scala> rdd1.count()
res0: Long = 1 

scala> sc.parallelize(1 to 1, 5000).setName("rdd2").cache().count()
res1: Long = 1 
{code}

Maybe, not a bug?

> SparkUI: Storage page is empty even if things are cached
> 
>
> Key: SPARK-15579
> URL: https://issues.apache.org/jira/browse/SPARK-15579
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core, Web UI
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>
> scala> sc.parallelize(1 to 1, 5000).cache().count()
> SparkUI storage page is empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-26 Thread Andrew Or (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302930#comment-15302930
 ] 

Andrew Or commented on SPARK-15579:
---

I tried this on 0f61d6efb45b9ee94fa663f67c4489fbdae2eded, which is literally 
the latest commit as of the writing of this message

> SparkUI: Storage page is empty even if things are cached
> 
>
> Key: SPARK-15579
> URL: https://issues.apache.org/jira/browse/SPARK-15579
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core, Web UI
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>
> scala> sc.parallelize(1 to 1, 5000).cache().count()
> SparkUI storage page is empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-26 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302869#comment-15302869
 ] 

Sean Owen commented on SPARK-15579:
---

I just tried this on a build from master today and I see one cached RDD 
immediately.

> SparkUI: Storage page is empty even if things are cached
> 
>
> Key: SPARK-15579
> URL: https://issues.apache.org/jira/browse/SPARK-15579
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core, Web UI
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>
> scala> sc.parallelize(1 to 1, 5000).cache().count()
> SparkUI storage page is empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org