Re: Preventing predicate pushdown

Thanks, filled https://issues.apache.org/jira/browse/SPARK-24288 Pozdrawiam / Best regards, Tomek On 2018-05-15 18:29, Wenchen Fan wrote: applying predict pushdown is an optimization, and it makes sense to provide configs to turn off certain optimizations. Feel free to create a JIRA. Thanks,

Re: Preventing predicate pushdown

applying predict pushdown is an optimization, and it makes sense to provide configs to turn off certain optimizations. Feel free to create a JIRA. Thanks, Wenchen On Tue, May 15, 2018 at 8:33 PM, Tomasz Gawęda wrote: > Hi, > > while working with JDBC datasource I saw

Preventing predicate pushdown

Hi, while working with JDBC datasource I saw that many "or" clauses with non-equality operators causes huge performance degradation of SQL query to database (DB2). For example: val df = spark.read.format("jdbc").(other options to parallelize load).load() df.where(s"(date1 > $param1 and