Hi All,
I wanted to experiment a little bit with TableScan and PrunedScan.
My first test was to print columns from various SQL queries.
To make this test easier, i just took spark-csv and i replaced TableScan
with PrunedScan.
I then changed buildScan method of CsvRelation from
def BuildScan = {
to
def buildScan(requiredColumns: Array[String]) = {?
This was the only modification i did to CsvRelation.scala. And I added
print of requiredColums to log.
I then took the same CSV file and run very simple SELECT query on it.
I noticed that when CsvRelation used TableScan - all worked correctly.
But when i used PrunedScan - it didn?t worked and returned empty columns /
or columns in wrong order.
Why is this happens? Is it some bug? Because I thought that PrunedScan
suppose to work exactly the same as TableScan and i can modify freely
TableScan to PrunedScan. I thought that the only difference is that
buildScan of PrunedScan has requiredColumns as parameter.
Can someone explain me the behavior i saw?
I am using Spark 1.5 from trunk.
Thanks a lot
Gil.
