[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin updated SPARK-9182: ------------------------------- Assignee: Cheng Lian > filter and groupBy on DataFrames are not passed through to jdbc source > ---------------------------------------------------------------------- > > Key: SPARK-9182 > URL: https://issues.apache.org/jira/browse/SPARK-9182 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.4.1 > Reporter: Greg Rahn > Assignee: Cheng Lian > Priority: Critical > > When running all of these API calls, the only one that passes the filter > through to the backend jdbc source is equality. All filters in these > commands should be able to be passed through to the jdbc database source. > {code} > val url="jdbc:postgresql:grahn" > val prop = new java.util.Properties > val emp = sqlContext.read.jdbc(url, "emp", prop) > emp.filter(emp("sal") === 5000).show() > emp.filter(emp("sal") < 5000).show() > emp.filter("sal = 3000").show() > emp.filter("sal > 2500").show() > emp.filter("sal >= 2500").show() > emp.filter("sal < 2500").show() > emp.filter("sal <= 2500").show() > emp.filter("sal != 3000").show() > emp.filter("sal between 3000 and 5000").show() > emp.filter("ename in ('SCOTT','BLAKE')").show() > {code} > We see from the PostgreSQL query log the following is run, and see that only > equality predicates are passed through. > {code} > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE > sal = 5000 > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE > sal = 3000 > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > LOG: execute <unnamed>: SET extra_float_digits = 3 > LOG: execute <unnamed>: SELECT > "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org