Thanks Eric. That is the case as most of my fields are optional. So it
seems that the problem comes from Parquet.
On Sat, Jul 19, 2014 at 8:27 AM, Eric Friedman eric.d.fried...@gmail.com
wrote:
Can position be null? Looks like there may be constraints with predicate
push down in that case. https://github.com/apache/spark/pull/511/
On Jul 18, 2014, at 8:04 PM, Christos Kozanitis kozani...@berkeley.edu
wrote:
Hello
What is the order with which SparkSQL deserializes parquet fields? Is it
possible to modify it?
I am using SparkSQL to query a parquet file that consists of a lot of
fields (around 30 or so). Let me call an example table MyTable and let's
suppose the name of one of its fields is position.
The query that I am executing is:
sql(select * from MyTable where position = 243189160)
The query plan that I get from this query is:
Filter (position#6L:6 = 243189160)
ParquetTableScan
[contig.contigName#0,contig.contigLength#1L,contig.contigMD5#2,contig.referenceURL#3,contig.assembly#4,contig.species#5,position#6L,rangeOffset#7,rangeLength#8,referenceBase#9,readBase#10,sangerQuality#11,mapQuality#12,numSoftClipped#13,numReverseStrand#14,countAtPosition#15,readName#16,readStart#17L,readEnd#18L,recordGroupSequencingCenter#19,recordGroupDescription#20,recordGroupRunDateEpoch#21L,recordGroupFlowOrder#22,recordGroupKeySequence#23,recordGroupLibrary#24,recordGroupPredictedMedianInsertSize#25,recordGroupPlatform#26,recordGroupPlatformUnit#27,recordGroupSample#28],
(ParquetRelation hdfs://
ec2-54-89-87-167.compute-1.amazonaws.com:9000/genomes/hg00096.plup), None
I expect 14 entries in the output but the execution of
.collect.foreach(println) takes forever to run on my cluster (more than an
hour).
Is it safe to assume in my example that SparkSQL deserializes all fields
first before applying the filter? If so, can a user change this behavior?
To support my assumption I replaced * with position, so my new query
is of the form sql(select position from MyTable where position =
243189160) and this query runs much faster on the same hardware (2-3
minutes vs 65 min).
Any ideas?
thanks
Christos
