No, empty parens do no matter when calling no-arg methods in Scala.
This invocation should work as-is and should result in the RDD showing
in Storage. I see that when I run it right now.
Since it really does/should work, I'd look at other possibilities --
is it maybe taking a short time to start
Hi Sean/Sameer,
It seems you're both right. In the python shell I need to explicitly call
the empty parens data.cache(), then run an action and it appears in the
storage tab. Using the scala shell I can just call data.cache without the
parens, run an action tthat works.
Thanks for your help.
Hi All,
When I load an RDD with:
data = sc.textFile(somefile)
I don't see the resulting RDD in the SparkContext gui on localhost:4040 in
/storage.
Is there something special I need to do to allow me to view this? I tried
but scala and python shells but same result.
Thanks
Stuart
Hey Stuart,
The RDD won't show up under the Storage tab in the UI until it's been
cached. Basically Spark doesn't know what the RDD will look like until it's
cached, b/c up until then the RDD is just on disk (external to Spark). If
you launch some transformations + an action on an RDD that is
Sorry too quick to pull the trigger on my original email. I should have
added that I'm tried using persist() and cache() but no joy.
I'm doing this:
data = sc.textFile(somedata)
data.cache
data.count()
but I still can't see anything in the storage?
On 31 October 2014 10:42, Sameer
Hi Stuart,
You're close!
Just add a () after the cache, like: data.cache()
...and then run the .count() action on it and you should be good to see it
in the Storage UI!
- Sameer
On Thu, Oct 30, 2014 at 4:50 PM, Stuart Horsman stuart.hors...@gmail.com
wrote:
Sorry too quick to pull the
