Re: Strange behavior with 'not' and filter pushdown

Wrapping this up -- fix is in 2.1.0 and has been backported to the 2.0.x branch, as well. On Mon, Feb 13, 2017 at 6:41 PM, Everett Anderson wrote: > Went ahead and opened > > https://issues.apache.org/jira/browse/SPARK-19586 > > though I'd generally expect to just close it as

Re: Strange behavior with 'not' and filter pushdown

Went ahead and opened https://issues.apache.org/jira/browse/SPARK-19586 though I'd generally expect to just close it as fixed in 2.1.0 and roll on. On Sat, Feb 11, 2017 at 5:01 PM, Everett Anderson wrote: > On the plus side, looks like this may be fixed in 2.1.0: > > ==

Re: Strange behavior with 'not' and filter pushdown

On the plus side, looks like this may be fixed in 2.1.0: == Physical Plan == *HashAggregate(keys=[], functions=[count(1)]) +- Exchange SinglePartition +- *HashAggregate(keys=[], functions=[partial_count(1)]) +- *Project +- *Filter NOT isnotnull(username#14) +-

Re: Strange behavior with 'not' and filter pushdown

Bumping this thread. Translating "where not(username is not null)" into a filter of [IsNotNull(username), Not(IsNotNull(username))] seems like a rather severe bug. Spark 1.6.2: explain select count(*) from parquet_table where not( username is not null) == Physical Plan ==

Strange behavior with 'not' and filter pushdown

Hi, I have an application where I’m filtering data with SparkSQL with simple WHERE clauses. I also want the ability to show the unmatched rows for any filter, and so am wrapping the previous clause in `NOT()` to get the inverse. Example: Filter: username is not null Inverse filter: