gt;
>> I'm in 2.0.1 at the moment... is there any way that works that far back?
>>
>> On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath > > wrote:
>>
>>> Dataset does have storageLevel. So you can use isCached = (storageLevel
>>> != StorageLevel.NO
6 AM, Nick Pentreath
> wrote:
>
>> Dataset does have storageLevel. So you can use isCached = (storageLevel
>> != StorageLevel.NONE) as a test.
>>
>> Arguably isCached could be added to dataset too, shouldn't be a
>> controversial change.
>>
>
Ah, in 2.1.0.
I'm in 2.0.1 at the moment... is there any way that works that far back?
On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath
wrote:
> Dataset does have storageLevel. So you can use isCached = (storageLevel !=
> StorageLevel.NONE) as a test.
>
> Arguably isCached
Dataset does have storageLevel. So you can use isCached = (storageLevel !=
StorageLevel.NONE) as a test.
Arguably isCached could be added to dataset too, shouldn't be a
controversial change.
On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld
wrote:
> I'm currently porting some of
I'm currently porting some of our code from RDDs to Datasets.
With RDDs it's pretty easy to figure out if they are cached or not.
I notice that the catalog has a function for determining this on Datasets
too, but it's private[sql]. Is there any reason for it not to be public?
Is there any way at
Thanks Cheng. Tried it out and saw the InMemoryColumnarTableScan word in the
physical plan.
From: Cheng Lian [mailto:lian.cs@gmail.com]
Sent: Friday, December 12, 2014 11:37 PM
To: Judy Nash; user@spark.apache.org
Subject: Re: Spark SQL API Doc & IsCached as SQL command
There isn’t a
equivalent SQL Query for Scala command:
IsCached() ?
2)Is there a documentation spec I can reference for question like this?
Closest doc I can find is this one:
https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses
Thanks,
Judy
http://spark.apache.org/docs/latest/sql-programming-guide.html#caching-data-in-memory
On Fri, Dec 12, 2014 at 3:14 PM, Judy Nash
wrote:
>
> Hello,
>
>
>
> Few questions on Spark SQL:
>
>
>
> 1) Does Spark SQL support equivalent SQL Query for Scala command:
>
Hello,
Few questions on Spark SQL:
1) Does Spark SQL support equivalent SQL Query for Scala command:
IsCached() ?
2) Is there a documentation spec I can reference for question like this?
Closest doc I can find is this one:
https://spark.apache.org/docs/1.1.0/sql-programming
