[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990328#comment-14990328
]
Sergey Shelukhin commented on HIVE-12092:
-
{noformat}
+ if (constValue != null) {
+ }
{noformat} should return something.
Also as far as I understand it doesn't take care of double backslash correctly.
State machine with substring at the end might be better.
Also perhaps there's a better way than a static method with type check to call
searchArgument. Perhaps it can just be callable on GenericUDF, null by default,
and the Bridge would delegate to UDFLike if the class is correct? Or something.
Or, I wonder if it could just be done when creating SARG from UDF, so that
SARG-specific code stays SARG-specific without Hive making assumptions about
what SARG can and cannot do? There may be a good reason to not do it this way,
I just wonder.
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993300#comment-14993300
]
Gopal V commented on HIVE-12092:
[~sershe]: fixed that.
bq. Or, I wonder if it could just be done when creating SARG from UDF, so that
SARG-specific code stays SARG-specific without Hive making assumptions about
what SARG can and cannot do?
That's exactly where the code is plugged in right now - the AST convertor. I
haven't got any code in the SARGs codebase as such - this is written this way
so that it doesn't add a ql/ dep into ORC.
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993778#comment-14993778
]
Hive QA commented on HIVE-12092:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12770971/HIVE-12092.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5949/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5949/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5949/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-5949/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
dbc7abe..cc12612 branch-1 -> origin/branch-1
b29705e..2ae1c5c master -> origin/master
+ git reset --hard HEAD
HEAD is now at b29705e HIVE-12288: Bloom-1 filters for Vectorized map-joins
(Gopal V, reviewed by Matt McCline)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 2ae1c5c HIVE-12346:Internally used variables in HiveConf should
not be settable via command (Chaoyu Tang, reviewed by Xuefu Zhang)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12770971 - PreCommit-HIVE-TRUNK-Build
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000690#comment-15000690
]
Gopal V commented on HIVE-12092:
With this patch NOT LIKE is handled is by returning only YES_NO & NO from the
LIKE SARG.
NOT(YES_NO) = YES_NO and NOT(NO) = YES
so the implementation also works if there is a negation.
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000976#comment-15000976
]
Sergey Shelukhin commented on HIVE-12092:
-
Can you add tests for \%, \_ and \\?
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000998#comment-15000998
]
Hive QA commented on HIVE-12092:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12771759/HIVE-12092.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9780 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_like
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_predicate_pushdown
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6001/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6001/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6001/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12771759 - PreCommit-HIVE-TRUNK-Build
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001331#comment-15001331
]
Prasanth Jayachandran commented on HIVE-12092:
--
Shouldn't assert check for GenericUDFOPEqualOrGreaterThan instead of just
GenericUDFOPGreaterThan?
Can you add unit tests for minPrefixPattern()? Also explain output showing
rewrite of LIKE "abc%" to >= "abc" will be good. Mostly looks good otherwise.
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001344#comment-15001344
]
Gopal V commented on HIVE-12092:
Yes, I should be checking for GenericUDFOPEqualOrGreaterThan.
The explain doesn't work, since the SARGs aren't printed in the explain at all
- I'll write unit tests for minPrefix and ConvertAstToSearchArg.
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001352#comment-15001352
]
Prasanth Jayachandran commented on HIVE-12092:
--
I missed the searchArgument in UDFLike. For some reason I thought SARG will get
the rewritten UDF.
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12092) SARGS: UDFLike prefix cases needs to be translated into >= sargs
[
https://issues.apache.org/jira/browse/HIVE-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049247#comment-15049247
]
Gunther Hagleitner commented on HIVE-12092:
---
In your example above you say: x like "https:%" -> x >= "https:". Doesn't it
also guarantee that: x < "https;" and if so wouldn't that narrow the space
further?
> SARGS: UDFLike prefix cases needs to be translated into >= sargs
>
>
> Key: HIVE-12092
> URL: https://issues.apache.org/jira/browse/HIVE-12092
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12092.1.patch, HIVE-12092.2.patch,
> HIVE-12092.3.patch
>
>
> A query which follows the following format
> {{select * from table where access_url like "https:%" ;}}
> needs to rewrite SARGs as
> {{access_url >= 'https:'}}
> to get a significant hit-rate on a simple expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
