Re: Kryo serialization mismatch in spark sql windowing function

2016-04-06 Thread Soam Acharya
Hi Josh, Appreciate the response! Also, Steve - we meet again :) At any rate, here's the output (a lot of it anyway) of running spark-sql with the verbose option so that you can get a sense of the settings and the classpath. Does anything stand out? Using properties file:

Re: Kryo serialization mismatch in spark sql windowing function

2016-04-06 Thread Josh Rosen
Spark is compiled against a custom fork of Hive 1.2.1 which added shading of Protobuf and removed shading of Kryo. What I think that what's happening here is that stock Hive 1.2.1 is taking precedence so the Kryo instance that it's returning is an instance of shaded/relocated Hive version rather

Kryo serialization mismatch in spark sql windowing function

2016-04-06 Thread Soam Acharya
Hi folks, I have a build of Spark 1.6.1 on which spark sql seems to be functional outside of windowing functions. For example, I can create a simple external table via Hive: CREATE EXTERNAL TABLE PSTable (pid int, tty string, time string, cmd string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','