+1 to implementing IN feature instead. We are also looking for IN / NOT-IN
cases where the inclusion/exclusion set is very large.
-Gautam
Sent from my iPhone
> On Mar 6, 2019, at 5:38 PM, Anton Okolnychyi
> wrote:
>
> For some reason, I thought there was a blocker there. As Iceberg is not
For some reason, I thought there was a blocker there. As Iceberg is not using
org.apache.parquet.filter2.predicate.FilterApi in its Parquet reader then makes
sense to fix, of course.
> On 5 Mar 2019, at 18:38, Ryan Blue wrote:
>
> Would it make sense to add support for IN expressions instead?
Would it make sense to add support for IN expressions instead? I'd rather
get that done than build work-arounds.
On Tue, Mar 5, 2019 at 10:33 AM Anton Okolnychyi
wrote:
> Hey,
>
> Iceberg Spark data source rewrites IN predicates as a mix of OR/EQ. I am
> wondering if it makes sense to introduce
Hey,
Iceberg Spark data source rewrites IN predicates as a mix of OR/EQ. I am
wondering if it makes sense to introduce a threshold when this rewrite happens
until [1] is resolved. We can have something similar to
“spark.sql.parquet.pushdown.inFilterThreshold” in Spark.
We have experienced a pe
