Thanks for the info On Fri, Sep 1, 2017 at 12:06 PM, Nick Pentreath <nick.pentre...@gmail.com> wrote:
> No unfortunately not - as i recall storageLevel accesses some private > methods to get the result. > > On Fri, 1 Sep 2017 at 17:55, Nathan Kronenfeld > <nkronenfeld@uncharted.software> > wrote: > >> Ah, in 2.1.0. >> >> I'm in 2.0.1 at the moment... is there any way that works that far back? >> >> On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <nick.pentre...@gmail.com >> > wrote: >> >>> Dataset does have storageLevel. So you can use isCached = (storageLevel >>> != StorageLevel.NONE) as a test. >>> >>> Arguably isCached could be added to dataset too, shouldn't be a >>> controversial change. >>> >>> On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld >>> <nkronenfeld@uncharted.software> >>> wrote: >>> >>>> I'm currently porting some of our code from RDDs to Datasets. >>>> >>>> With RDDs it's pretty easy to figure out if they are cached or not. >>>> >>>> I notice that the catalog has a function for determining this on >>>> Datasets too, but it's private[sql]. Is there any reason for it not to be >>>> public? Is there any way at the moment to determine if a dataset is cached >>>> or not? >>>> >>>> Thanks in advance >>>> -Nathan Kronenfeld >>>> >>> >>
