Re: Kryo serialization mismatch in spark sql windowing function

2016-04-06 Thread Soam Acharya
shaded Kryo is being used here. > > /cc +Marcelo Vanzin <van...@cloudera.com> and +Steve Loughran > <ste...@hortonworks.com>, who may know more. > > On Wed, Apr 6, 2016 at 6:08 PM Soam Acharya <s...@altiscale.com> wrote: > >> Hi folks, >> >> I ha

Kryo serialization mismatch in spark sql windowing function

2016-04-06 Thread Soam Acharya
Hi folks, I have a build of Spark 1.6.1 on which spark sql seems to be functional outside of windowing functions. For example, I can create a simple external table via Hive: CREATE EXTERNAL TABLE PSTable (pid int, tty string, time string, cmd string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','

sqlContext.cacheTable + yarn client mode

2016-03-30 Thread Soam Acharya
Hi folks, I understand that invoking sqlContext.cacheTable("tableName") will load the table into a compressed in-memory columnar format. When Spark is launched via spark shell in YARN client mode, is the table loaded into the local Spark driver process in addition to the executors in the Hadoop