Hi Justin, Does the schema of your data have any decimal, array, map, or struct type?
Thanks, Yin On Tue, Apr 7, 2015 at 6:31 PM, Justin Yip <[email protected]> wrote: > Hello, > > I have a parquet file of around 55M rows (~ 1G on disk). Performing simple > grouping operation is pretty efficient (I get results within 10 seconds). > However, after called DataFrame.cache, I observe a significant performance > degrade, the same operation now takes 3+ minutes. > > My hunch is that DataFrame cannot leverage its columnar format after > persisting in memory. But cannot find anywhere from the doc mentioning this. > > Did I miss anything? > > Thanks! > > Justin >
