hello all, in case anyone is interested, i just wrote a short blog about using shapeless in spark to optimize data layout in memory.
blog is here: http://tresata.com/tresata-open-sources-spark-columnar code is here: https://github.com/tresata/spark-columnar
