Is there a way to read a Parquet File as ColumnarBatch?

2019-04-22 Thread Priyanka Gomatam
Hi, I am new to Spark and have been playing around with the Parquet reader code. I have two questions: 1. I saw the code that starts at DataSourceScanExec class, and moves on to the ParquetFileFormat class and does a VectorizedParquetRecordReader. I tried doing a spark.read.parquet(...) and

Re: Is there a way to read a Parquet File as ColumnarBatch?

2019-04-22 Thread Jacek Laskowski
Hi Priyanka, I've been exploring this part of Spark SQL and could help a little bit. > but for some reason it never hit the breakpoints I placed in these classes. Was this for local[*]? I ran "SPARK_SUBMIT_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" ./bin/spark-shel