[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990328#comment-14990328 ] Sergey Shelukhin commented on HIVE-12092: - {noformat} + if (constValue != null) { + } {noformat} should return something. Also as far as I understand it doesn't take care of double backslash correctly. State machine with substring at the end might be better. Also perhaps there's a better way than a static method with type check to call searchArgument. Perhaps it can just be callable on GenericUDF, null by default, and the Bridge would delegate to UDFLike if the class is correct? Or something. Or, I wonder if it could just be done when creating SARG from UDF, so that SARG-specific code stays SARG-specific without Hive making assumptions about what SARG can and cannot do? There may be a good reason to not do it this way, I just wonder. > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993300#comment-14993300 ] Gopal V commented on HIVE-12092: [~sershe]: fixed that. bq. Or, I wonder if it could just be done when creating SARG from UDF, so that SARG-specific code stays SARG-specific without Hive making assumptions about what SARG can and cannot do? That's exactly where the code is plugged in right now - the AST convertor. I haven't got any code in the SARGs codebase as such - this is written this way so that it doesn't add a ql/ dep into ORC. > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993778#comment-14993778 ] Hive QA commented on HIVE-12092: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12770971/HIVE-12092.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5949/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5949/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5949/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-5949/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive dbc7abe..cc12612 branch-1 -> origin/branch-1 b29705e..2ae1c5c master -> origin/master + git reset --hard HEAD HEAD is now at b29705e HIVE-12288: Bloom-1 filters for Vectorized map-joins (Gopal V, reviewed by Matt McCline) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at 2ae1c5c HIVE-12346:Internally used variables in HiveConf should not be settable via command (Chaoyu Tang, reviewed by Xuefu Zhang) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12770971 - PreCommit-HIVE-TRUNK-Build > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000690#comment-15000690 ] Gopal V commented on HIVE-12092: With this patch NOT LIKE is handled is by returning only YES_NO & NO from the LIKE SARG. NOT(YES_NO) = YES_NO and NOT(NO) = YES so the implementation also works if there is a negation. > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch, > HIVE-12092.3.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000976#comment-15000976 ] Sergey Shelukhin commented on HIVE-12092: - Can you add tests for \%, \_ and \\? > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch, > HIVE-12092.3.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000998#comment-15000998 ] Hive QA commented on HIVE-12092: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12771759/HIVE-12092.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9780 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_like org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_predicate_pushdown org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6001/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6001/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6001/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12771759 - PreCommit-HIVE-TRUNK-Build > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch, > HIVE-12092.3.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001331#comment-15001331 ] Prasanth Jayachandran commented on HIVE-12092: -- Shouldn't assert check for GenericUDFOPEqualOrGreaterThan instead of just GenericUDFOPGreaterThan? Can you add unit tests for minPrefixPattern()? Also explain output showing rewrite of LIKE "abc%" to >= "abc" will be good. Mostly looks good otherwise. > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch, > HIVE-12092.3.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001344#comment-15001344 ] Gopal V commented on HIVE-12092: Yes, I should be checking for GenericUDFOPEqualOrGreaterThan. The explain doesn't work, since the SARGs aren't printed in the explain at all - I'll write unit tests for minPrefix and ConvertAstToSearchArg. > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch, > HIVE-12092.3.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001352#comment-15001352 ] Prasanth Jayachandran commented on HIVE-12092: -- I missed the searchArgument in UDFLike. For some reason I thought SARG will get the rewritten UDF. > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch, > HIVE-12092.3.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[ https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049247#comment-15049247 ] Gunther Hagleitner commented on HIVE-12092: --- In your example above you say: x like "https:%" -> x >= "https:". Doesn't it also guarantee that: x < "https;" and if so wouldn't that narrow the space further? > SARGS: UDFLike prefix cases needs to be translated into >= sargs > > > Key: HIVE-12092 > URL: https://issues.apache.org/jira/browse/HIVE-12092 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch, > HIVE-12092.3.patch > > > A query which follows the following format > {{select * from table where access_url like "https:%" ;}} > needs to rewrite SARGs as > {{access_url >= 'https:'}} > to get a significant hit-rate on a simple expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)