Re: Limitations in the IndexPredicateAnalyzer
Hi Navis, ok then I know. I'll have a look at that later on. /Petter 2013/12/18 Navis류승우 > IndexPredicateAnalyzer in hive supports AND conjunction only because it's > simple. Anyone can implement one that supports conjunctions like OR, CASE, > etc. if needed. If you might provide that to hive community, it would be > really appreciated. > > ps. > There is a draft patch handling OR conjuncted predicates on HBaseHandler > (thanks to Teddy). > I'll arrange that to be included into hive trunk in sometime. > > > > > 2013/12/11 Petter von Dolwitz (Hem) > > Hi, >> >> I use the HiveStoragePredicateHandler interface on my storage handler to >> be able to push down filters to my record reader for early pruning. It >> seems like the IndexPredicateAnalyzer is a bit limited on what expression >> that could be pushed down. >> >> From a comment in IndexPredicateAnalyzer: >> "We can only push down stuff which appears as part of a pure >> conjunction: reject OR, CASE, etc." >> >> As such I can push down simple AND expressions in my where clause (and >> also IN RLIKE and LIKE) but I cannot use OR. >> >> Does anybody know why this limitation exists? A guess from my side is >> that it was originally developed for HBASE integration where OR expressions >> might not translate well. Having other data sources with custom indexes >> (or relational databases for that matter) would benefit from being able to >> handle a wider spectrum of expression. >> >> Thanks, >> Petter >> >> >> >
Re: Limitations in the IndexPredicateAnalyzer
IndexPredicateAnalyzer in hive supports AND conjunction only because it's simple. Anyone can implement one that supports conjunctions like OR, CASE, etc. if needed. If you might provide that to hive community, it would be really appreciated. ps. There is a draft patch handling OR conjuncted predicates on HBaseHandler (thanks to Teddy). I'll arrange that to be included into hive trunk in sometime. 2013/12/11 Petter von Dolwitz (Hem) > Hi, > > I use the HiveStoragePredicateHandler interface on my storage handler to > be able to push down filters to my record reader for early pruning. It > seems like the IndexPredicateAnalyzer is a bit limited on what expression > that could be pushed down. > > From a comment in IndexPredicateAnalyzer: > "We can only push down stuff which appears as part of a pure conjunction: > reject OR, CASE, etc." > > As such I can push down simple AND expressions in my where clause (and > also IN RLIKE and LIKE) but I cannot use OR. > > Does anybody know why this limitation exists? A guess from my side is that > it was originally developed for HBASE integration where OR expressions > might not translate well. Having other data sources with custom indexes > (or relational databases for that matter) would benefit from being able to > handle a wider spectrum of expression. > > Thanks, > Petter > > >
Limitations in the IndexPredicateAnalyzer
Hi, I use the HiveStoragePredicateHandler interface on my storage handler to be able to push down filters to my record reader for early pruning. It seems like the IndexPredicateAnalyzer is a bit limited on what expression that could be pushed down. >From a comment in IndexPredicateAnalyzer: "We can only push down stuff which appears as part of a pure conjunction: reject OR, CASE, etc." As such I can push down simple AND expressions in my where clause (and also IN RLIKE and LIKE) but I cannot use OR. Does anybody know why this limitation exists? A guess from my side is that it was originally developed for HBASE integration where OR expressions might not translate well. Having other data sources with custom indexes (or relational databases for that matter) would benefit from being able to handle a wider spectrum of expression. Thanks, Petter
