Hi Richard,
I am not sure how to support user-defined type. But regarding your second question, you can have a walkaround as following. Suppose you have a struct a, and want to filter a.c with a.c > X. You can define a alias C as a.c, and add extra column C to the schema of the relation, and your query would be C > X instead of a.c > X. In this way, in the buildScan you would have GreaterThan(C, X). You then can programmatically convert C to a.c. Note that in the buildScan required columns would also have an extra column C you need to returned in the buildScan RDD. It looks complicated, but I think it would work. Thanks. Zhan Zhang ________________________________ From: Richard Eggert <richard.egg...@gmail.com> Sent: Saturday, September 19, 2015 3:59 PM To: User Subject: PrunedFilteredScan does not work for UDTs and Struct fields I defined my own relation (extending BaseRelation) and implemented the PrunedFilteredScan interface, but discovered that if the column referenced in a WHERE = clause is a user-defined type or a field of a struct column, then Spark SQL passes NO filters to the PrunedFilteredScan.buildScan method, rendering the interface useless. Is there really no way to implement a relation to optimize on such fields? -- Rich
