Deer impala comunity, I'm reading the docs of the newly runtime filtering technique, http://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html#runtime_filtering While being quite enlightening, this document still confuses me in some aspects.
Question 1, >> For other file formats (text, Avro, RCFile, and SequenceFile), >> runtime filtering speeds up queries against partitioned tables only. Is it a technical problem to implement runtime filtering for other formats? Will they be supported in the future? Question 2, >> explain select s from yy2 where year in (select year from yy where year between 2000 and 2005); >> +----------------------------------------------------------+ >> | Explain String | >> +----------------------------------------------------------+ >> | Estimated Per-Host Requirements: Memory=16.00MB VCores=2 | >> | | >> | 04:EXCHANGE [UNPARTITIONED] | >> | | | >> | 02:HASH JOIN [LEFT SEMI JOIN, BROADCAST] | >> | | hash predicates: year = year | >> | | runtime filters: RF000 <- year | >> | | | >> | |--03:EXCHANGE [BROADCAST] | >> | | | | >> | | 01:SCAN HDFS [dpp.yy] | >> | | partitions=2/4 files=2 size=468B | >> | | | >> | 00:SCAN HDFS [dpp.yy2] | >> | partitions=2/3 files=2 size=468B | >> | runtime filters: RF000 -> year | >> +----------------------------------------------------------+ How does the planner be able to tell that there gonna be 2/3 partitions surviving after runtime fitering? Question 3, >> When the spill-to-disk mechanism is activated on a particular host >> during a query, that host does not produce any filters while >> processing that query. Isn't the spill-to-disk ON by default? However, through my observation, runtime filtering is still enabled. What does this quote sentence mean? Question 4, In which scenarios will Impala encounter Memory limit exceeded when spill-to-disk is enabled. Any help is highly appreciated! Regards, Amos
