Re: isCached
Thanks for the info On Fri, Sep 1, 2017 at 12:06 PM, Nick Pentreath <nick.pentre...@gmail.com> wrote: > No unfortunately not - as i recall storageLevel accesses some private > methods to get the result. > > On Fri, 1 Sep 2017 at 17:55, Nathan Kronenfeld > <nkronenfeld@uncharted.software> > wrote: > >> Ah, in 2.1.0. >> >> I'm in 2.0.1 at the moment... is there any way that works that far back? >> >> On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <nick.pentre...@gmail.com >> > wrote: >> >>> Dataset does have storageLevel. So you can use isCached = (storageLevel >>> != StorageLevel.NONE) as a test. >>> >>> Arguably isCached could be added to dataset too, shouldn't be a >>> controversial change. >>> >>> On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld >>> <nkronenfeld@uncharted.software> >>> wrote: >>> >>>> I'm currently porting some of our code from RDDs to Datasets. >>>> >>>> With RDDs it's pretty easy to figure out if they are cached or not. >>>> >>>> I notice that the catalog has a function for determining this on >>>> Datasets too, but it's private[sql]. Is there any reason for it not to be >>>> public? Is there any way at the moment to determine if a dataset is cached >>>> or not? >>>> >>>> Thanks in advance >>>>-Nathan Kronenfeld >>>> >>> >>
Re: isCached
No unfortunately not - as i recall storageLevel accesses some private methods to get the result. On Fri, 1 Sep 2017 at 17:55, Nathan Kronenfeld <nkronenfeld@uncharted.software> wrote: > Ah, in 2.1.0. > > I'm in 2.0.1 at the moment... is there any way that works that far back? > > On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <nick.pentre...@gmail.com> > wrote: > >> Dataset does have storageLevel. So you can use isCached = (storageLevel >> != StorageLevel.NONE) as a test. >> >> Arguably isCached could be added to dataset too, shouldn't be a >> controversial change. >> >> On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld >> <nkronenfeld@uncharted.software> wrote: >> >>> I'm currently porting some of our code from RDDs to Datasets. >>> >>> With RDDs it's pretty easy to figure out if they are cached or not. >>> >>> I notice that the catalog has a function for determining this on >>> Datasets too, but it's private[sql]. Is there any reason for it not to be >>> public? Is there any way at the moment to determine if a dataset is cached >>> or not? >>> >>> Thanks in advance >>>-Nathan Kronenfeld >>> >> >
Re: isCached
Ah, in 2.1.0. I'm in 2.0.1 at the moment... is there any way that works that far back? On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <nick.pentre...@gmail.com> wrote: > Dataset does have storageLevel. So you can use isCached = (storageLevel != > StorageLevel.NONE) as a test. > > Arguably isCached could be added to dataset too, shouldn't be a > controversial change. > > On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld > <nkronenfeld@uncharted.software> > wrote: > >> I'm currently porting some of our code from RDDs to Datasets. >> >> With RDDs it's pretty easy to figure out if they are cached or not. >> >> I notice that the catalog has a function for determining this on Datasets >> too, but it's private[sql]. Is there any reason for it not to be public? >> Is there any way at the moment to determine if a dataset is cached or not? >> >> Thanks in advance >>-Nathan Kronenfeld >> >
Re: isCached
Dataset does have storageLevel. So you can use isCached = (storageLevel != StorageLevel.NONE) as a test. Arguably isCached could be added to dataset too, shouldn't be a controversial change. On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld <nkronenfeld@uncharted.software> wrote: > I'm currently porting some of our code from RDDs to Datasets. > > With RDDs it's pretty easy to figure out if they are cached or not. > > I notice that the catalog has a function for determining this on Datasets > too, but it's private[sql]. Is there any reason for it not to be public? > Is there any way at the moment to determine if a dataset is cached or not? > > Thanks in advance >-Nathan Kronenfeld >
isCached
I'm currently porting some of our code from RDDs to Datasets. With RDDs it's pretty easy to figure out if they are cached or not. I notice that the catalog has a function for determining this on Datasets too, but it's private[sql]. Is there any reason for it not to be public? Is there any way at the moment to determine if a dataset is cached or not? Thanks in advance -Nathan Kronenfeld
RE: Spark SQL API Doc IsCached as SQL command
Thanks Cheng. Tried it out and saw the InMemoryColumnarTableScan word in the physical plan. From: Cheng Lian [mailto:lian.cs@gmail.com] Sent: Friday, December 12, 2014 11:37 PM To: Judy Nash; user@spark.apache.org Subject: Re: Spark SQL API Doc IsCached as SQL command There isn’t a SQL statement that directly maps SQLContext.isCached, but you can use EXPLAIN EXTENDED to check whether the underlying physical plan is a InMemoryColumnarTableScan. On 12/13/14 7:14 AM, Judy Nash wrote: Hello, Few questions on Spark SQL: 1) Does Spark SQL support equivalent SQL Query for Scala command: IsCached(table name) ? 2) Is there a documentation spec I can reference for question like this? Closest doc I can find is this one: https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses Thanks, Judy
Spark SQL API Doc IsCached as SQL command
Hello, Few questions on Spark SQL: 1) Does Spark SQL support equivalent SQL Query for Scala command: IsCached(table name) ? 2) Is there a documentation spec I can reference for question like this? Closest doc I can find is this one: https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses Thanks, Judy
Re: Spark SQL API Doc IsCached as SQL command
http://spark.apache.org/docs/latest/sql-programming-guide.html#caching-data-in-memory On Fri, Dec 12, 2014 at 3:14 PM, Judy Nash judyn...@exchange.microsoft.com wrote: Hello, Few questions on Spark SQL: 1) Does Spark SQL support equivalent SQL Query for Scala command: IsCached(table name) ? 2) Is there a documentation spec I can reference for question like this? Closest doc I can find is this one: https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses Thanks, Judy
Re: Spark SQL API Doc IsCached as SQL command
There isn’t a SQL statement that directly maps |SQLContext.isCached|, but you can use |EXPLAIN EXTENDED| to check whether the underlying physical plan is a |InMemoryColumnarTableScan|. On 12/13/14 7:14 AM, Judy Nash wrote: Hello, Few questions on Spark SQL: 1)Does Spark SQL support equivalent SQL Query for Scala command: IsCached(table name) ? 2)Is there a documentation spec I can reference for question like this? Closest doc I can find is this one: https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses Thanks, Judy
