[jira] [Commented] (HIVE-22880) ACID: All delete event readers should ignore ORC SARGs
[ https://issues.apache.org/jira/browse/HIVE-22880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040731#comment-17040731 ] Peter Vary commented on HIVE-22880: --- How hard would it be to create a test case? > ACID: All delete event readers should ignore ORC SARGs > -- > > Key: HIVE-22880 > URL: https://issues.apache.org/jira/browse/HIVE-22880 > Project: Hive > Issue Type: Bug > Components: Transactions, Vectorization >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Blocker > Attachments: HIVE-22880.1.patch > > > Delete delta readers should not apply any SARGs other than the ones related > to the transaction id ranges within the inserts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22913) HiveReduceExpressionsWithStatsRule should not invoke simplifcation during end of visits
[ https://issues.apache.org/jira/browse/HIVE-22913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-22913: --- > HiveReduceExpressionsWithStatsRule should not invoke simplifcation during end > of visits > --- > > Key: HIVE-22913 > URL: https://issues.apache.org/jira/browse/HIVE-22913 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > * doing a simplification is a full visit of the subtree > * in case the rule makes a structural change; it will invoke simplification > during exiting the recursion > https://github.com/apache/hive/blob/5012954396c98c94d0ff64fe3bbdb74e6077f190/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java#L275 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22879) Optimise jar file loading in CalcitePlanner
[
https://issues.apache.org/jira/browse/HIVE-22879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040728#comment-17040728
]
Zoltan Haindrich commented on HIVE-22879:
-
[~rajesh.balamohan] I think HIVE-22881 should make this issue go away - could
you test it ?
In case you are looking at profiler outputs: keep an eye on the total time
spent in and under the RexSimplify class; especially in case of complex
filter/join conditions - let me know if it eats up resources for no good reason
:D
It's not something which is easily fixable; but it would be good to know how
much it would worth focusing on.
> Optimise jar file loading in CalcitePlanner
> ---
>
> Key: HIVE-22879
> URL: https://issues.apache.org/jira/browse/HIVE-22879
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Rajesh Balamohan
>Priority: Major
>
> {{CalcitePlanner}} internally uses {{org.codehaus.janino.UnitCompiler
> (calcite dependency)}} and this appears to load the jars in every thread.
> Need to check if this can be avoided.
> Here is an example.
> {noformat}
> at java.util.zip.ZipFile.getEntry(Native Method)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:310)
> - locked <0x0005c1af21c0> (a java.util.jar.JarFile)
> at java.util.jar.JarFile.getEntry(JarFile.java:240)
> at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1005)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:212)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> - locked <0x0005caa3be88> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
> at org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
> - locked <0x000686136868> (a
> org.codehaus.janino.ClassLoaderIClassLoader)
> at
> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
> at
> org.codehaus.janino.UnitCompiler.reclassifyName(UnitCompiler.java:8478)
> at
> org.codehaus.janino.UnitCompiler.reclassifyName(UnitCompiler.java:8471)
> at org.codehaus.janino.UnitCompiler.reclassify(UnitCompiler.java:8331)
> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6855)
> at org.codehaus.janino.UnitCompiler.access$14200(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$22$2$1.visitAmbiguousName(UnitCompiler.java:6497)
> at
> org.codehaus.janino.UnitCompiler$22$2$1.visitAmbiguousName(UnitCompiler.java:6494)
> at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:4224)
> at
> org.codehaus.janino.UnitCompiler$22$2.visitLvalue(UnitCompiler.java:6494)
> at
> org.codehaus.janino.UnitCompiler$22$2.visitLvalue(UnitCompiler.java:6490)
> at org.codehaus.janino.Java$Lvalue.accept(Java.java:4148)
> at
> org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6490)
> at
> org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6469)
> at org.codehaus.janino.Java$Rvalue.accept(Java.java:4116)
> at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6469)
> at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:9026)
> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:7106)
> at org.codehaus.janino.UnitCompiler.access$15800(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$22$2.visitMethodInvocation(UnitCompiler.java:6517)
> at
> org.codehaus.janino.UnitCompiler$22$2.visitMethodInvocation(UnitCompiler.java:6490)
> at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:5073)
> at
> org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6490)
> at
> org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6469)
> at org.codehaus.janino.Java$Rvalue.accept(Java.java:4116)
> at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6469)
> at
> org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:9237)
> at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:9123)
> at
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040722#comment-17040722
]
Hive QA commented on HIVE-22899:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993903/HIVE-22899.4.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18045 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats_noscan_2]
(batchId=133)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20742/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20742/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20742/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993903 - PreCommit-HIVE-Build
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[ https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada reassigned HIVE-19368: --- Assignee: (was: Anurag Mantripragada) > Metastore: log a warning with table-name + partition-count when > get_partitions returns >10k partitions > -- > > Key: HIVE-19368 > URL: https://issues.apache.org/jira/browse/HIVE-19368 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Gopal Vijayaraghavan >Priority: Major > Attachments: HIVE-19368.1.patch > > > Ran into this particular letter from the trenches & would like a normal WARN > log for it. > https://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches/24 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-20363) Use integer constants for frequently used serde classes.
[
https://issues.apache.org/jira/browse/HIVE-20363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned HIVE-20363:
---
Assignee: (was: Anurag Mantripragada)
> Use integer constants for frequently used serde classes.
>
>
> Key: HIVE-20363
> URL: https://issues.apache.org/jira/browse/HIVE-20363
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Reporter: Anurag Mantripragada
>Priority: Major
> Attachments: HIVE-20363.1.patch
>
>
> Serde libraries are stored as fully qualified class names which are long
> strings, we can get improvements in I/O and storage if we store integer
> constants for frequently used serde classes in the backend DB.
> For example:
> {code:java}
> org.apache.hadoop.hive.serde2.avro.AvroSerDe"---> 1
> parquet.hive.serde.ParquetHiveSerDe" ---> 2
> ...{code}
> Review the patch at:
> https://reviews.apache.org/r/68546/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20363) Use integer constants for frequently used serde classes.
[
https://issues.apache.org/jira/browse/HIVE-20363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated HIVE-20363:
Status: Open (was: Patch Available)
> Use integer constants for frequently used serde classes.
>
>
> Key: HIVE-20363
> URL: https://issues.apache.org/jira/browse/HIVE-20363
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Reporter: Anurag Mantripragada
>Assignee: Anurag Mantripragada
>Priority: Major
> Attachments: HIVE-20363.1.patch
>
>
> Serde libraries are stored as fully qualified class names which are long
> strings, we can get improvements in I/O and storage if we store integer
> constants for frequently used serde classes in the backend DB.
> For example:
> {code:java}
> org.apache.hadoop.hive.serde2.avro.AvroSerDe"---> 1
> parquet.hive.serde.ParquetHiveSerDe" ---> 2
> ...{code}
> Review the patch at:
> https://reviews.apache.org/r/68546/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[ https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated HIVE-19368: Status: Open (was: Patch Available) > Metastore: log a warning with table-name + partition-count when > get_partitions returns >10k partitions > -- > > Key: HIVE-19368 > URL: https://issues.apache.org/jira/browse/HIVE-19368 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Gopal Vijayaraghavan >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-19368.1.patch > > > Ran into this particular letter from the trenches & would like a normal WARN > log for it. > https://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches/24 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22905) Transaction is not aborted when query cancelled, only when session is closed
[ https://issues.apache.org/jira/browse/HIVE-22905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040703#comment-17040703 ] Aron Hamvas commented on HIVE-22905: [~gopalv], this could happen for any query that is blocked in DbLockManager.lock(), waiting for a lock to be acquired. > Transaction is not aborted when query cancelled, only when session is closed > > > Key: HIVE-22905 > URL: https://issues.apache.org/jira/browse/HIVE-22905 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.2 >Reporter: Aron Hamvas >Assignee: Aron Hamvas >Priority: Major > Attachments: HIVE-22905.patch > > > Reproduction: > # Start HMS > # Start HS2 > # Start beeline > # Execute a query and cancel query after transaction is started and before > locks are acquired (yeah, that will take a debugger with breakpoints in > DbLockManager) > # Do not terminate Hive session > DbLockManager will keep spinning in the checkLock loop as long as the Hive > session is not closed since abortTxns is not invoked on HMS, and until the > lock is acquired (even though the query could have been canceled long time > ago). > Driver only checks in close() method if there are locks acquired but it does > not check if a txn is open. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22881) Revise non-recommended Calcite api calls
[
https://issues.apache.org/jira/browse/HIVE-22881?focusedWorklogId=389814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389814
]
ASF GitHub Bot logged work on HIVE-22881:
-
Author: ASF GitHub Bot
Created on: 20/Feb/20 06:42
Start Date: 20/Feb/20 06:42
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #919: HIVE-22881
rexutil usage
URL: https://github.com/apache/hive/pull/919#discussion_r381806268
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java
##
@@ -272,7 +274,11 @@ public RexNode visitCall(RexCall call) {
// If we did not reduce, check the children nodes
RexNode node = super.visitCall(call);
if (node != call) {
-node = RexUtil.simplify(rexBuilder, node);
+// FIXME if this rule will make some changes; then it will invoke
simplify on all subtrees during exiting the recursion
Review comment:
It was there before; and I can't really do anything right now with
it...leaving a comment might be usefull if it lights up on someones profiler
To fix this we would need some way to keep track which nodes were simplified
already...that should be fixed in Calcite;and it might not even possible
(because the set of predicates might be different )
We might consider moving the simplification outside of the recursive phase -
that will certainly reduce the re-visit pressure; but could result in reduced
simplification in some cases (however considering how many places we are doing
this, that's unlikely) - it could be explored in a separate patch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389814)
Time Spent: 0.5h (was: 20m)
> Revise non-recommended Calcite api calls
>
>
> Key: HIVE-22881
> URL: https://issues.apache.org/jira/browse/HIVE-22881
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch,
> HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> RexUtil.simplify* methods
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause
[
https://issues.apache.org/jira/browse/HIVE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040666#comment-17040666
]
Shubham Chaurasia commented on HIVE-22903:
--
[~rameshkumar] Thanks a lot for the review.
{quote}
We should probably loop through the groupBatches and skip reseting if it is a
row_number and a constant(And probably this might fix
https://issues.apache.org/jira/browse/HIVE-22909 too).
{quote}
Sorry I could not understand this. Currently, it's like
{code:java}
if (!isPartitionOrderBy && !skipResetEvaluatorsForRowNumber) {
groupBatches.resetEvaluators();
}
{code}
Does looping though groupBatches (evaluators ? ) mean something like
{code:java}
public void resetEvaluators() {
for (VectorPTFEvaluatorBase evaluator : evaluators) {
if (!isPartitionOrderBy && !skipResetEvaluatorsForRowNumber) {
evaluator.resetEvaluator();
}
}
}
{code}
I was confused because these flags isPartitionOrderBy and
skipResetEvaluatorsForRowNumber are common to all the evaluators and would not
change for a particular evaluator.
> Vectorized row_number() resets the row number after one batch in case of
> constant expression in partition clause
>
>
> Key: HIVE-22903
> URL: https://issues.apache.org/jira/browse/HIVE-22903
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22903.01.patch, HIVE-22903.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorized row number implementation resets the row number when constant
> expression is passed in partition clause.
> Repro Query
> {code}
> select row_number() over(partition by 1) r1, t from over10k_n8;
> Or
> select row_number() over() r1, t from over10k_n8;
> {code}
> where table over10k_n8 contains more than 1024 records.
> This happens because currently in VectorPTFOperator, we reset evaluators if
> only partition clause is there.
> {code:java}
> // If we are only processing a PARTITION BY, reset our evaluators.
> if (!isPartitionOrderBy) {
> groupBatches.resetEvaluators();
> }
> {code}
> To resolve, we should also check if the entire partition clause is a constant
> expression, if it is so then we should not do
> {{groupBatches.resetEvaluators()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040663#comment-17040663
]
Hive QA commented on HIVE-22899:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20742/dev-support/hive-personality.sh
|
| git revision | master / 5012954 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20742/yetus/patch-asflicense-problems.txt
|
| modules | C: ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20742/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22863) Commit compaction txn if it is opened but compaction is skipped
[
https://issues.apache.org/jira/browse/HIVE-22863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040640#comment-17040640
]
Hive QA commented on HIVE-22863:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993869/HIVE-22863.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18047 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation2 (batchId=292)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20741/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20741/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20741/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993869 - PreCommit-HIVE-Build
> Commit compaction txn if it is opened but compaction is skipped
> ---
>
> Key: HIVE-22863
> URL: https://issues.apache.org/jira/browse/HIVE-22863
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22863.01.patch, HIVE-22863.02.patch,
> HIVE-22863.03.patch, HIVE-22863.04.patch
>
>
> Currently if a table does not have enough directories to compact, compaction
> is skipped and the compaction is either (a) marked ready for cleaning or (b)
> marked compacted. However, the txn the compaction runs in is never committed,
> it remains open, so TXNS and TXN_COMPONENTS will never be cleared of
> information about the attempted compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22863) Commit compaction txn if it is opened but compaction is skipped
[
https://issues.apache.org/jira/browse/HIVE-22863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040616#comment-17040616
]
Hive QA commented on HIVE-22863:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 1534 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
57s{color} | {color:red} ql generated 1 new + 1534 unchanged - 0 fixed = 1535
total (was 1534) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Possible null pointer dereference of RemoteCompactorThread.msc in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() Dereferenced at
Worker.java:RemoteCompactorThread.msc in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() Dereferenced at
Worker.java:[line 270] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20741/dev-support/hive-personality.sh
|
| git revision | master / 5012954 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20741/yetus/new-findbugs-ql.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20741/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20741/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Commit compaction txn if it is opened but compaction is skipped
> ---
>
> Key: HIVE-22863
> URL: https://issues.apache.org/jira/browse/HIVE-22863
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22863.01.patch, HIVE-22863.02.patch,
> HIVE-22863.03.patch, HIVE-22863.04.patch
>
>
> Currently if a table does not have enough directories to compact, compaction
> is skipped and the compaction is either (a) marked ready for cleaning or (b)
> marked compacted. However, the txn the compaction runs in is never committed,
> it remains open, so TXNS and TXN_COMPONENTS will never be cleared of
> information about the attempted compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22862) Remove unnecessary calls to isEnoughToCompact
[
https://issues.apache.org/jira/browse/HIVE-22862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040605#comment-17040605
]
Hive QA commented on HIVE-22862:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993867/HIVE-22862.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18045 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20740/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20740/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20740/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993867 - PreCommit-HIVE-Build
> Remove unnecessary calls to isEnoughToCompact
> -
>
> Key: HIVE-22862
> URL: https://issues.apache.org/jira/browse/HIVE-22862
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Attachments: HIVE-22862.01.patch, HIVE-22862.01.patch,
> HIVE-22862.02.patch, HIVE-22862.03.patch, HIVE-22862.04.patch
>
>
> QueryCompactor.Util#isEnoughToCompact is called in Worker#run once before any
> sort of compaction is run, after this it is called in 3 other places during
> compaction unnecessarily.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22891) Skip PartitonDesc Extraction In CombineHiveRecord For Non-LLAP Execution Mode
[
https://issues.apache.org/jira/browse/HIVE-22891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040590#comment-17040590
]
Syed Shameerur Rahman commented on HIVE-22891:
--
[~szita] Thank You for the review. Yes, it makes sense to keep the check in
HiveInputFormat#wrapForLlap intact. I have addressed your comments in the
HIVE-22891.03.patch. Please take a look.
> Skip PartitonDesc Extraction In CombineHiveRecord For Non-LLAP Execution Mode
> -
>
> Key: HIVE-22891
> URL: https://issues.apache.org/jira/browse/HIVE-22891
> Project: Hive
> Issue Type: Task
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22891.01.patch, HIVE-22891.02.patch,
> HIVE-22891.03.patch
>
>
> {code:java}
> try {
> // TODO: refactor this out
> if (pathToPartInfo == null) {
> MapWork mrwork;
> if (HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
> mrwork = (MapWork) Utilities.getMergeWork(jobConf);
> if (mrwork == null) {
> mrwork = Utilities.getMapWork(jobConf);
> }
> } else {
> mrwork = Utilities.getMapWork(jobConf);
> }
> pathToPartInfo = mrwork.getPathToPartitionInfo();
> } PartitionDesc part = extractSinglePartSpec(hsplit);
> inputFormat = HiveInputFormat.wrapForLlap(inputFormat, jobConf, part);
> } catch (HiveException e) {
> throw new IOException(e);
> }
> {code}
> The above piece of code in CombineHiveRecordReader.java was introduced in
> HIVE-15147. This overwrites inputFormat based on the PartitionDesc which is
> not required in non-LLAP mode of execution as the method
> HiveInputFormat.wrapForLlap() simply returns the previously defined
> inputFormat in case of non-LLAP mode. The method call extractSinglePartSpec()
> has some serious performance implications. If there are large no. of small
> files, each call in the method extractSinglePartSpec() takes approx ~ (2 - 3)
> seconds. Hence the same query which runs in Hive 1.x / Hive 2 is way faster
> than the query run on latest hive.
> {code:java}
> 2020-02-11 07:15:04,701 INFO [main]
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl: Reading ORC rows from
> 2020-02-11 07:15:06,468 WARN [main]
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader: Multiple partitions
> found; not going to pass a part spec to LLAP IO: {{logdate=2020-02-03,
> hour=01, event=win}} and {{logdate=2020-02-03, hour=02, event=act}}
> 2020-02-11 07:15:06,468 INFO [main]
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader: succeeded in getting
> org.apache.hadoop.mapred.FileSplit{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22891) Skip PartitonDesc Extraction In CombineHiveRecord For Non-LLAP Execution Mode
[
https://issues.apache.org/jira/browse/HIVE-22891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22891:
-
Attachment: HIVE-22891.03.patch
> Skip PartitonDesc Extraction In CombineHiveRecord For Non-LLAP Execution Mode
> -
>
> Key: HIVE-22891
> URL: https://issues.apache.org/jira/browse/HIVE-22891
> Project: Hive
> Issue Type: Task
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22891.01.patch, HIVE-22891.02.patch,
> HIVE-22891.03.patch
>
>
> {code:java}
> try {
> // TODO: refactor this out
> if (pathToPartInfo == null) {
> MapWork mrwork;
> if (HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
> mrwork = (MapWork) Utilities.getMergeWork(jobConf);
> if (mrwork == null) {
> mrwork = Utilities.getMapWork(jobConf);
> }
> } else {
> mrwork = Utilities.getMapWork(jobConf);
> }
> pathToPartInfo = mrwork.getPathToPartitionInfo();
> } PartitionDesc part = extractSinglePartSpec(hsplit);
> inputFormat = HiveInputFormat.wrapForLlap(inputFormat, jobConf, part);
> } catch (HiveException e) {
> throw new IOException(e);
> }
> {code}
> The above piece of code in CombineHiveRecordReader.java was introduced in
> HIVE-15147. This overwrites inputFormat based on the PartitionDesc which is
> not required in non-LLAP mode of execution as the method
> HiveInputFormat.wrapForLlap() simply returns the previously defined
> inputFormat in case of non-LLAP mode. The method call extractSinglePartSpec()
> has some serious performance implications. If there are large no. of small
> files, each call in the method extractSinglePartSpec() takes approx ~ (2 - 3)
> seconds. Hence the same query which runs in Hive 1.x / Hive 2 is way faster
> than the query run on latest hive.
> {code:java}
> 2020-02-11 07:15:04,701 INFO [main]
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl: Reading ORC rows from
> 2020-02-11 07:15:06,468 WARN [main]
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader: Multiple partitions
> found; not going to pass a part spec to LLAP IO: {{logdate=2020-02-03,
> hour=01, event=win}} and {{logdate=2020-02-03, hour=02, event=act}}
> 2020-02-11 07:15:06,468 INFO [main]
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader: succeeded in getting
> org.apache.hadoop.mapred.FileSplit{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22881) Revise non-recommended Calcite api calls
[ https://issues.apache.org/jira/browse/HIVE-22881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040587#comment-17040587 ] Jesus Camacho Rodriguez commented on HIVE-22881: Left a minor comment. +1 > Revise non-recommended Calcite api calls > > > Key: HIVE-22881 > URL: https://issues.apache.org/jira/browse/HIVE-22881 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch, > HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch > > Time Spent: 10m > Remaining Estimate: 0h > > RexUtil.simplify* methods -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22881) Revise non-recommended Calcite api calls
[
https://issues.apache.org/jira/browse/HIVE-22881?focusedWorklogId=389761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389761
]
ASF GitHub Bot logged work on HIVE-22881:
-
Author: ASF GitHub Bot
Created on: 20/Feb/20 02:43
Start Date: 20/Feb/20 02:43
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #919: HIVE-22881
rexutil usage
URL: https://github.com/apache/hive/pull/919#discussion_r381679711
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java
##
@@ -272,7 +274,11 @@ public RexNode visitCall(RexCall call) {
// If we did not reduce, check the children nodes
RexNode node = super.visitCall(call);
if (node != call) {
-node = RexUtil.simplify(rexBuilder, node);
+// FIXME if this rule will make some changes; then it will invoke
simplify on all subtrees during exiting the recursion
Review comment:
Is this to fix in this patch or in a follow-up? Can you create a JIRA if
that is the case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389761)
Time Spent: 20m (was: 10m)
> Revise non-recommended Calcite api calls
>
>
> Key: HIVE-22881
> URL: https://issues.apache.org/jira/browse/HIVE-22881
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch,
> HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> RexUtil.simplify* methods
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22862) Remove unnecessary calls to isEnoughToCompact
[
https://issues.apache.org/jira/browse/HIVE-22862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040583#comment-17040583
]
Hive QA commented on HIVE-22862:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-20740/patches/PreCommit-HIVE-Build-20740.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20740/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove unnecessary calls to isEnoughToCompact
> -
>
> Key: HIVE-22862
> URL: https://issues.apache.org/jira/browse/HIVE-22862
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Attachments: HIVE-22862.01.patch, HIVE-22862.01.patch,
> HIVE-22862.02.patch, HIVE-22862.03.patch, HIVE-22862.04.patch
>
>
> QueryCompactor.Util#isEnoughToCompact is called in Worker#run once before any
> sort of compaction is run, after this it is called in 3 other places during
> compaction unnecessarily.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22534) ACID: Improve Compactor thread logging
[
https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040579#comment-17040579
]
Hive QA commented on HIVE-22534:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993866/HIVE-22534.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20739/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20739/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20739/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-02-20 02:17:41.705
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20739/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-02-20 02:17:41.708
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5012954 HIVE-22896 : Increase fast hashtable size on detecting
initial collision (Rajesh Balamohan via Ashutosh Chauhan)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5012954 HIVE-22896 : Increase fast hashtable size on detecting
initial collision (Rajesh Balamohan via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-02-20 02:17:42.510
+ rm -rf ../yetus_PreCommit-HIVE-Build-20739
+ mkdir ../yetus_PreCommit-HIVE-Build-20739
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20739
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20739/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java: does
not exist in index
Trying to apply the patch with -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java:71
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java:71
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-20739
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993866 - PreCommit-HIVE-Build
> ACID: Improve Compactor thread logging
> --
>
> Key: HIVE-22534
> URL: https://issues.apache.org/jira/browse/HIVE-22534
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee:
[jira] [Commented] (HIVE-22904) Compaction cleaner cannot find COMPACTION_QUEUE table using postgres db
[
https://issues.apache.org/jira/browse/HIVE-22904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040578#comment-17040578
]
Hive QA commented on HIVE-22904:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993863/HIVE-22904.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18045 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20738/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20738/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993863 - PreCommit-HIVE-Build
> Compaction cleaner cannot find COMPACTION_QUEUE table using postgres db
> ---
>
> Key: HIVE-22904
> URL: https://issues.apache.org/jira/browse/HIVE-22904
> Project: Hive
> Issue Type: Bug
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-22904.01.patch, HIVE-22904.02.patch,
> HIVE-22904.03.patch
>
>
> In CompactionTxnHandler
> {code:java}
> delete from COMPACTION_QUEUE where cq_id = ?
> {code}
> fails with
> {code:java}
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22912) Support native submission of Hive queries to a Kubernetes Cluster
[ https://issues.apache.org/jira/browse/HIVE-22912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040563#comment-17040563 ] Gopal Vijayaraghavan commented on HIVE-22912: - Are you referring to HIVE-22359? > Support native submission of Hive queries to a Kubernetes Cluster > - > > Key: HIVE-22912 > URL: https://issues.apache.org/jira/browse/HIVE-22912 > Project: Hive > Issue Type: New Feature >Reporter: Surbhi Aggarwal >Priority: Major > > So many big data applications are already integrated or trying to natively > integrate with Kubernetes engine. Should we not work together to support hive > with this engine? > If efforts are already being spent on this, please point me to it. Thanks ! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause
[
https://issues.apache.org/jira/browse/HIVE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040547#comment-17040547
]
Ramesh Kumar Thangarajan commented on HIVE-22903:
-
[~ShubhamChaurasia] Great work identifying the issue and the fix. I think this
fix does the work. But I am thinking the fix is more specific to single
argument, whereas the issue is more generic. We should probably loop through
the groupBatches and skip reseting if it is a row_number and a constant(And
probably this might fix https://issues.apache.org/jira/browse/HIVE-22909 too).
Do you think this make sense?
> Vectorized row_number() resets the row number after one batch in case of
> constant expression in partition clause
>
>
> Key: HIVE-22903
> URL: https://issues.apache.org/jira/browse/HIVE-22903
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22903.01.patch, HIVE-22903.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorized row number implementation resets the row number when constant
> expression is passed in partition clause.
> Repro Query
> {code}
> select row_number() over(partition by 1) r1, t from over10k_n8;
> Or
> select row_number() over() r1, t from over10k_n8;
> {code}
> where table over10k_n8 contains more than 1024 records.
> This happens because currently in VectorPTFOperator, we reset evaluators if
> only partition clause is there.
> {code:java}
> // If we are only processing a PARTITION BY, reset our evaluators.
> if (!isPartitionOrderBy) {
> groupBatches.resetEvaluators();
> }
> {code}
> To resolve, we should also check if the entire partition clause is a constant
> expression, if it is so then we should not do
> {{groupBatches.resetEvaluators()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22904) Compaction cleaner cannot find COMPACTION_QUEUE table using postgres db
[
https://issues.apache.org/jira/browse/HIVE-22904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040544#comment-17040544
]
Hive QA commented on HIVE-22904:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20738/dev-support/hive-personality.sh
|
| git revision | master / 5012954 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20738/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20738/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Compaction cleaner cannot find COMPACTION_QUEUE table using postgres db
> ---
>
> Key: HIVE-22904
> URL: https://issues.apache.org/jira/browse/HIVE-22904
> Project: Hive
> Issue Type: Bug
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-22904.01.patch, HIVE-22904.02.patch,
> HIVE-22904.03.patch
>
>
> In CompactionTxnHandler
> {code:java}
> delete from COMPACTION_QUEUE where cq_id = ?
> {code}
> fails with
> {code:java}
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21543) Use FilterHooks for show compactions
[
https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040537#comment-17040537
]
Hive QA commented on HIVE-21543:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993850/HIVE-21543.07.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18045 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20737/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20737/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20737/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993850 - PreCommit-HIVE-Build
> Use FilterHooks for show compactions
>
>
> Key: HIVE-21543
> URL: https://issues.apache.org/jira/browse/HIVE-21543
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-21543.01.patch, HIVE-21543.02.patch,
> HIVE-21543.03.patch, HIVE-21543.04.patch, HIVE-21543.05.patch,
> HIVE-21543.06.patch, HIVE-21543.07.patch
>
>
> Use FilterHooks for checking dbs/tables/partitions for showCompactions
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21543) Use FilterHooks for show compactions
[
https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040522#comment-17040522
]
Hive QA commented on HIVE-21543:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
44s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20737/dev-support/hive-personality.sh
|
| git revision | master / 5012954 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20737/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20737/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use FilterHooks for show compactions
>
>
> Key: HIVE-21543
> URL: https://issues.apache.org/jira/browse/HIVE-21543
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-21543.01.patch, HIVE-21543.02.patch,
> HIVE-21543.03.patch, HIVE-21543.04.patch, HIVE-21543.05.patch,
> HIVE-21543.06.patch, HIVE-21543.07.patch
>
>
> Use FilterHooks for checking dbs/tables/partitions for showCompactions
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22907) Break up DDLSemanticAnalyzer - extract the rest of the Alter Table analyzers
[ https://issues.apache.org/jira/browse/HIVE-22907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22907: -- Attachment: HIVE-22907.02.patch > Break up DDLSemanticAnalyzer - extract the rest of the Alter Table analyzers > > > Key: HIVE-22907 > URL: https://issues.apache.org/jira/browse/HIVE-22907 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Attachments: HIVE-22907.01.patch, HIVE-22907.02.patch > > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #15: extract the rest of the alter table analyzers from > DDLSemanticAnalyzer, and move them under the new package. Remove > DDLSemanticAnalyzer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040514#comment-17040514 ] Marta Kuczora commented on HIVE-21164: -- The failing test is unrelated to this patch and locally runs successful, so reattach the same patch to get a green run. > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Marta Kuczora >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch, > HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch, > HIVE-21164.15.patch, HIVE-21164.16.patch, HIVE-21164.17.patch, > HIVE-21164.18.patch, HIVE-21164.19.patch, HIVE-21164.2.patch, > HIVE-21164.20.patch, HIVE-21164.21.patch, HIVE-21164.22.patch, > HIVE-21164.3.patch, HIVE-21164.4.patch, HIVE-21164.5.patch, > HIVE-21164.6.patch, HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040514#comment-17040514 ] Marta Kuczora edited comment on HIVE-21164 at 2/19/20 11:32 PM: The failing test is unrelated to this patch and locally runs successfully, so reattach the same patch to get a green run. was (Author: kuczoram): The failing test is unrelated to this patch and locally runs successful, so reattach the same patch to get a green run. > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Marta Kuczora >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch, > HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch, > HIVE-21164.15.patch, HIVE-21164.16.patch, HIVE-21164.17.patch, > HIVE-21164.18.patch, HIVE-21164.19.patch, HIVE-21164.2.patch, > HIVE-21164.20.patch, HIVE-21164.21.patch, HIVE-21164.22.patch, > HIVE-21164.3.patch, HIVE-21164.4.patch, HIVE-21164.5.patch, > HIVE-21164.6.patch, HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-21164: - Attachment: HIVE-21164.22.patch > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Marta Kuczora >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch, > HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch, > HIVE-21164.15.patch, HIVE-21164.16.patch, HIVE-21164.17.patch, > HIVE-21164.18.patch, HIVE-21164.19.patch, HIVE-21164.2.patch, > HIVE-21164.20.patch, HIVE-21164.21.patch, HIVE-21164.22.patch, > HIVE-21164.3.patch, HIVE-21164.4.patch, HIVE-21164.5.patch, > HIVE-21164.6.patch, HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[
https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040504#comment-17040504
]
Hive QA commented on HIVE-21164:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993849/HIVE-21164.21.patch
{color:green}SUCCESS:{color} +1 due to 21 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18048 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_materialized]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20736/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20736/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20736/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993849 - PreCommit-HIVE-Build
> ACID: explore how we can avoid a move step during inserts/compaction
>
>
> Key: HIVE-21164
> URL: https://issues.apache.org/jira/browse/HIVE-21164
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch,
> HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch,
> HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch,
> HIVE-21164.15.patch, HIVE-21164.16.patch, HIVE-21164.17.patch,
> HIVE-21164.18.patch, HIVE-21164.19.patch, HIVE-21164.2.patch,
> HIVE-21164.20.patch, HIVE-21164.21.patch, HIVE-21164.3.patch,
> HIVE-21164.4.patch, HIVE-21164.5.patch, HIVE-21164.6.patch,
> HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch
>
>
> Currently, we write compacted data to a temporary location and then move the
> files to a final location, which is an expensive operation on some cloud file
> systems. Since HIVE-20823 is already in, it can control the visibility of
> compacted data for the readers. Therefore, we can perhaps avoid writing data
> to a temporary location and directly write compacted data to the intended
> final path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22896) Increase fast hashtable size on detecting initial collision
[ https://issues.apache.org/jira/browse/HIVE-22896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22896: Fix Version/s: 4.0.0 Assignee: Rajesh Balamohan Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, [~rajesh.balamohan] ! > Increase fast hashtable size on detecting initial collision > --- > > Key: HIVE-22896 > URL: https://issues.apache.org/jira/browse/HIVE-22896 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22896.1.patch > > > This would help in avoiding collisions and helps in burning lesser CPU cycles > during probing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22896) Increase fast hashtable size on detecting initial collision
[ https://issues.apache.org/jira/browse/HIVE-22896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040495#comment-17040495 ] Ashutosh Chauhan commented on HIVE-22896: - +1 > Increase fast hashtable size on detecting initial collision > --- > > Key: HIVE-22896 > URL: https://issues.apache.org/jira/browse/HIVE-22896 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22896.1.patch > > > This would help in avoiding collisions and helps in burning lesser CPU cycles > during probing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22905) Transaction is not aborted when query cancelled, only when session is closed
[ https://issues.apache.org/jira/browse/HIVE-22905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040485#comment-17040485 ] Gopal Vijayaraghavan commented on HIVE-22905: - [~hamvas.aron]: does this happen for a query with a missing table error? > Transaction is not aborted when query cancelled, only when session is closed > > > Key: HIVE-22905 > URL: https://issues.apache.org/jira/browse/HIVE-22905 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.2 >Reporter: Aron Hamvas >Assignee: Aron Hamvas >Priority: Major > Attachments: HIVE-22905.patch > > > Reproduction: > # Start HMS > # Start HS2 > # Start beeline > # Execute a query and cancel query after transaction is started and before > locks are acquired (yeah, that will take a debugger with breakpoints in > DbLockManager) > # Do not terminate Hive session > DbLockManager will keep spinning in the checkLock loop as long as the Hive > session is not closed since abortTxns is not invoked on HMS, and until the > lock is acquired (even though the query could have been canceled long time > ago). > Driver only checks in close() method if there are locks acquired but it does > not check if a txn is open. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[
https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040480#comment-17040480
]
Hive QA commented on HIVE-21164:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1534 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
5s{color} | {color:red} ql: The patch generated 35 new + 2704 unchanged - 26
fixed = 2739 total (was 2730) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 21 new + 165
unchanged - 21 fixed = 186 total (was 186) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 15 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} ql generated 0 new + 1533 unchanged - 1 fixed = 1533
total (was 1534) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} streaming in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} streaming in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20736/dev-support/hive-personality.sh
|
| git revision | master / 335c2b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
|
[jira] [Commented] (HIVE-22359) LLAP: when a node restarts with the exact same host/port in kubernetes it is not detected as a task failure
[
https://issues.apache.org/jira/browse/HIVE-22359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040473#comment-17040473
]
Prasanth Jayachandran commented on HIVE-22359:
--
The test failure seems unrelated. Splitted the patch into 2 separate patches.
Will wait for another precommit run.
> LLAP: when a node restarts with the exact same host/port in kubernetes it is
> not detected as a task failure
> ---
>
> Key: HIVE-22359
> URL: https://issues.apache.org/jira/browse/HIVE-22359
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal Vijayaraghavan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22359.1.patch, HIVE-22359.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> │ <14>1 2019-10-16T22:16:39.233Z
> query-coordinator-0-5.query-coordinator-0-service.compute-1569601454-l2x9.svc.cluster.local
> query-coordinator 1 461e5ad9-f05f-11e9-85f7-06e84765763e [mdc@18060
> class="te │
> │ zplugins.LlapTaskCommunicator" level="INFO" thread="IPC Server handler 4 on
> 3"] The tasks we expected to be on the node are not there:
> attempt_1569601631911__1_04_34_0, attempt_15696016319 │
> │ 11__1_04_71_0, attempt_1569601631911__1_04_000191_0,
> attempt_1569601631911__1_04_000211_0,
> attempt_1569601631911__1_04_000229_0,
> attempt_1569601631911__1_04_000231_0, attempt_1 │
> │ 569601631911__1_04_000235_0, attempt_1569601631911__1_04_000242_0,
> attempt_1569601631911__1_04_000160_1,
> attempt_1569601631911__1_04_12_2,
> attempt_1569601631911__1_04_03_2, │
> │ attempt_1569601631911__1_04_56_2,
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22908) AM caching connections to LLAP based on hostname and port does not work in kubernetes
[ https://issues.apache.org/jira/browse/HIVE-22908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-22908: - Status: Patch Available (was: Open) > AM caching connections to LLAP based on hostname and port does not work in > kubernetes > - > > Key: HIVE-22908 > URL: https://issues.apache.org/jira/browse/HIVE-22908 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-22908.1.patch > > > AM is caching all connections to LLAP services using combination of hostname > and port which does not work in kubernetes environment where hostname of pod > and port can be same with statefulset. This causes AM to talk to old LLAP > which could have died or OOM/Pod kill etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22908) AM caching connections to LLAP based on hostname and port does not work in kubernetes
[ https://issues.apache.org/jira/browse/HIVE-22908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-22908: - Attachment: HIVE-22908.1.patch > AM caching connections to LLAP based on hostname and port does not work in > kubernetes > - > > Key: HIVE-22908 > URL: https://issues.apache.org/jira/browse/HIVE-22908 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-22908.1.patch > > > AM is caching all connections to LLAP services using combination of hostname > and port which does not work in kubernetes environment where hostname of pod > and port can be same with statefulset. This causes AM to talk to old LLAP > which could have died or OOM/Pod kill etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22359) LLAP: when a node restarts with the exact same host/port in kubernetes it is not detected as a task failure
[
https://issues.apache.org/jira/browse/HIVE-22359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-22359:
-
Attachment: HIVE-22359.2.patch
> LLAP: when a node restarts with the exact same host/port in kubernetes it is
> not detected as a task failure
> ---
>
> Key: HIVE-22359
> URL: https://issues.apache.org/jira/browse/HIVE-22359
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal Vijayaraghavan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22359.1.patch, HIVE-22359.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> │ <14>1 2019-10-16T22:16:39.233Z
> query-coordinator-0-5.query-coordinator-0-service.compute-1569601454-l2x9.svc.cluster.local
> query-coordinator 1 461e5ad9-f05f-11e9-85f7-06e84765763e [mdc@18060
> class="te │
> │ zplugins.LlapTaskCommunicator" level="INFO" thread="IPC Server handler 4 on
> 3"] The tasks we expected to be on the node are not there:
> attempt_1569601631911__1_04_34_0, attempt_15696016319 │
> │ 11__1_04_71_0, attempt_1569601631911__1_04_000191_0,
> attempt_1569601631911__1_04_000211_0,
> attempt_1569601631911__1_04_000229_0,
> attempt_1569601631911__1_04_000231_0, attempt_1 │
> │ 569601631911__1_04_000235_0, attempt_1569601631911__1_04_000242_0,
> attempt_1569601631911__1_04_000160_1,
> attempt_1569601631911__1_04_12_2,
> attempt_1569601631911__1_04_03_2, │
> │ attempt_1569601631911__1_04_56_2,
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22562) Harmonize SessionState.getUserName
[ https://issues.apache.org/jira/browse/HIVE-22562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22562: Attachment: HIVE-22562.05.patch > Harmonize SessionState.getUserName > -- > > Key: HIVE-22562 > URL: https://issues.apache.org/jira/browse/HIVE-22562 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22562.01.patch, HIVE-22562.02.patch, > HIVE-22562.03.patch, HIVE-22562.04.patch, HIVE-22562.05.patch > > > we might have 2 different user names at the same time: > * > [getUserName()|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L1912] > ** a method which relies on the userName field of the SessionState > * > [getUserFromAuthenticator()|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L1291] > ** a method which uses the authenticator to do the heavy lifting > * there all kind of interesting call sites like: > ** there are some which are [prefering the authenticator over > getUserName()|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L254] > ** there are some which [use getUserName() regardless authenticator, but have > fixme|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L1669] > ** and there are some which are just using the authenticator with or without > notes/etc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause
[
https://issues.apache.org/jira/browse/HIVE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040461#comment-17040461
]
Hive QA commented on HIVE-22903:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993885/HIVE-22903.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18046 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20735/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20735/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20735/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993885 - PreCommit-HIVE-Build
> Vectorized row_number() resets the row number after one batch in case of
> constant expression in partition clause
>
>
> Key: HIVE-22903
> URL: https://issues.apache.org/jira/browse/HIVE-22903
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22903.01.patch, HIVE-22903.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorized row number implementation resets the row number when constant
> expression is passed in partition clause.
> Repro Query
> {code}
> select row_number() over(partition by 1) r1, t from over10k_n8;
> Or
> select row_number() over() r1, t from over10k_n8;
> {code}
> where table over10k_n8 contains more than 1024 records.
> This happens because currently in VectorPTFOperator, we reset evaluators if
> only partition clause is there.
> {code:java}
> // If we are only processing a PARTITION BY, reset our evaluators.
> if (!isPartitionOrderBy) {
> groupBatches.resetEvaluators();
> }
> {code}
> To resolve, we should also check if the entire partition clause is a constant
> expression, if it is so then we should not do
> {{groupBatches.resetEvaluators()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22893) Enhance data size estimation for fields computed by UDFs
[
https://issues.apache.org/jira/browse/HIVE-22893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22893:
Attachment: HIVE-22893.08.patch
> Enhance data size estimation for fields computed by UDFs
>
>
> Key: HIVE-22893
> URL: https://issues.apache.org/jira/browse/HIVE-22893
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22893.01.patch, HIVE-22893.02.patch,
> HIVE-22893.03.patch, HIVE-22893.04.patch, HIVE-22893.05.patch,
> HIVE-22893.06.patch, HIVE-22893.07.patch, HIVE-22893.08.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Right now if we have columnstat on a column ; we use that to estimate things
> about the column; - however if an UDF is executed on a column ; the resulting
> column is treated as unknown thing and defaults are assumed.
> An improvement could be to give wide estimation(s) in case of frequently used
> udf.
> For example; consider {{substr(c,1,1)}} ; no matter what the input; the
> output is at most a 1 long string
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.13.patch > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22562) Harmonize SessionState.getUserName
[ https://issues.apache.org/jira/browse/HIVE-22562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22562: Attachment: HIVE-22562.04.patch > Harmonize SessionState.getUserName > -- > > Key: HIVE-22562 > URL: https://issues.apache.org/jira/browse/HIVE-22562 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22562.01.patch, HIVE-22562.02.patch, > HIVE-22562.03.patch, HIVE-22562.04.patch > > > we might have 2 different user names at the same time: > * > [getUserName()|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L1912] > ** a method which relies on the userName field of the SessionState > * > [getUserFromAuthenticator()|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L1291] > ** a method which uses the authenticator to do the heavy lifting > * there all kind of interesting call sites like: > ** there are some which are [prefering the authenticator over > getUserName()|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L254] > ** there are some which [use getUserName() regardless authenticator, but have > fixme|https://github.com/apache/hive/blob/ab71e5a22834b5fdd17d6e4ddb54bcd324ae97d7/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L1669] > ** and there are some which are just using the authenticator with or without > notes/etc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.12.patch > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22893) Enhance data size estimation for fields computed by UDFs
[
https://issues.apache.org/jira/browse/HIVE-22893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22893:
Attachment: HIVE-22893.07.patch
> Enhance data size estimation for fields computed by UDFs
>
>
> Key: HIVE-22893
> URL: https://issues.apache.org/jira/browse/HIVE-22893
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22893.01.patch, HIVE-22893.02.patch,
> HIVE-22893.03.patch, HIVE-22893.04.patch, HIVE-22893.05.patch,
> HIVE-22893.06.patch, HIVE-22893.07.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Right now if we have columnstat on a column ; we use that to estimate things
> about the column; - however if an UDF is executed on a column ; the resulting
> column is treated as unknown thing and defaults are assumed.
> An improvement could be to give wide estimation(s) in case of frequently used
> udf.
> For example; consider {{substr(c,1,1)}} ; no matter what the input; the
> output is at most a 1 long string
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22893) Enhance data size estimation for fields computed by UDFs
[
https://issues.apache.org/jira/browse/HIVE-22893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22893:
Attachment: HIVE-22893.06.patch
> Enhance data size estimation for fields computed by UDFs
>
>
> Key: HIVE-22893
> URL: https://issues.apache.org/jira/browse/HIVE-22893
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22893.01.patch, HIVE-22893.02.patch,
> HIVE-22893.03.patch, HIVE-22893.04.patch, HIVE-22893.05.patch,
> HIVE-22893.06.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Right now if we have columnstat on a column ; we use that to estimate things
> about the column; - however if an UDF is executed on a column ; the resulting
> column is treated as unknown thing and defaults are assumed.
> An improvement could be to give wide estimation(s) in case of frequently used
> udf.
> For example; consider {{substr(c,1,1)}} ; no matter what the input; the
> output is at most a 1 long string
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause
[
https://issues.apache.org/jira/browse/HIVE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040412#comment-17040412
]
Hive QA commented on HIVE-22903:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 1534 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20735/dev-support/hive-personality.sh
|
| git revision | master / 335c2b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20735/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20735/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20735/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized row_number() resets the row number after one batch in case of
> constant expression in partition clause
>
>
> Key: HIVE-22903
> URL: https://issues.apache.org/jira/browse/HIVE-22903
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22903.01.patch, HIVE-22903.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorized row number implementation resets the row number when constant
> expression is passed in partition clause.
> Repro Query
> {code}
> select row_number() over(partition by 1) r1, t from over10k_n8;
> Or
> select row_number() over() r1, t from over10k_n8;
> {code}
> where table over10k_n8 contains more than 1024 records.
> This happens because currently in VectorPTFOperator, we reset evaluators if
> only partition clause is there.
> {code:java}
> // If we are only processing a PARTITION BY, reset our evaluators.
> if (!isPartitionOrderBy) {
> groupBatches.resetEvaluators();
> }
> {code}
> To resolve, we should also check if the entire partition clause is a constant
> expression, if it is so then we should not do
> {{groupBatches.resetEvaluators()}}
--
This
[jira] [Updated] (HIVE-22893) Enhance data size estimation for fields computed by UDFs
[
https://issues.apache.org/jira/browse/HIVE-22893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22893:
Attachment: HIVE-22893.05.patch
> Enhance data size estimation for fields computed by UDFs
>
>
> Key: HIVE-22893
> URL: https://issues.apache.org/jira/browse/HIVE-22893
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22893.01.patch, HIVE-22893.02.patch,
> HIVE-22893.03.patch, HIVE-22893.04.patch, HIVE-22893.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Right now if we have columnstat on a column ; we use that to estimate things
> about the column; - however if an UDF is executed on a column ; the resulting
> column is treated as unknown thing and defaults are assumed.
> An improvement could be to give wide estimation(s) in case of frequently used
> udf.
> For example; consider {{substr(c,1,1)}} ; no matter what the input; the
> output is at most a 1 long string
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22881) Revise non-recommended Calcite api calls
[ https://issues.apache.org/jira/browse/HIVE-22881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040411#comment-17040411 ] Zoltan Haindrich commented on HIVE-22881: - [~jcamachorodriguez], [~vgarg] Could you please take a look? > Revise non-recommended Calcite api calls > > > Key: HIVE-22881 > URL: https://issues.apache.org/jira/browse/HIVE-22881 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch, > HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch > > Time Spent: 10m > Remaining Estimate: 0h > > RexUtil.simplify* methods -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22881) Revise non-recommended Calcite api calls
[ https://issues.apache.org/jira/browse/HIVE-22881?focusedWorklogId=389598=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389598 ] ASF GitHub Bot logged work on HIVE-22881: - Author: ASF GitHub Bot Created on: 19/Feb/20 20:22 Start Date: 19/Feb/20 20:22 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #919: HIVE-22881 rexutil usage URL: https://github.com/apache/hive/pull/919 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 389598) Remaining Estimate: 0h Time Spent: 10m > Revise non-recommended Calcite api calls > > > Key: HIVE-22881 > URL: https://issues.apache.org/jira/browse/HIVE-22881 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch, > HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch > > Time Spent: 10m > Remaining Estimate: 0h > > RexUtil.simplify* methods -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22881) Revise non-recommended Calcite api calls
[ https://issues.apache.org/jira/browse/HIVE-22881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22881: -- Labels: pull-request-available (was: ) > Revise non-recommended Calcite api calls > > > Key: HIVE-22881 > URL: https://issues.apache.org/jira/browse/HIVE-22881 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch, > HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch > > > RexUtil.simplify* methods -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22881) Revise non-recommended Calcite api calls
[
https://issues.apache.org/jira/browse/HIVE-22881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040392#comment-17040392
]
Hive QA commented on HIVE-22881:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993825/HIVE-22881.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18045 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20733/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20733/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20733/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993825 - PreCommit-HIVE-Build
> Revise non-recommended Calcite api calls
>
>
> Key: HIVE-22881
> URL: https://issues.apache.org/jira/browse/HIVE-22881
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch,
> HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch
>
>
> RexUtil.simplify* methods
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=389566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389566
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 19/Feb/20 19:18
Start Date: 19/Feb/20 19:18
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r381440128
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java
##
@@ -221,6 +227,9 @@ public void debugDumpShort(StringBuilder sb) {
metadataCache, dataCache, bufferManagerOrc, conf, cacheMetrics,
ioMetrics, tracePool);
this.genericCvp = isEncodeEnabled ? new GenericColumnVectorProducer(
serdeCache, bufferManagerGeneric, conf, cacheMetrics, ioMetrics,
tracePool) : null;
+proactiveEvictionExecutor = Executors.newSingleThreadExecutor(
Review comment:
@szlta i do not think we need this, the actual eviction will run on the RPC
thread anyway so that is not touching the execution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389566)
Time Spent: 20m (was: 10m)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition from a table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[ https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=389567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389567 ] ASF GitHub Bot logged work on HIVE-22821: - Author: ASF GitHub Bot Created on: 19/Feb/20 19:18 Start Date: 19/Feb/20 19:18 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #909: HIVE-22821 URL: https://github.com/apache/hive/pull/909#discussion_r381451935 ## File path: llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java ## @@ -75,4 +78,6 @@ * @return amount (bytes) of memory evicted. */ long purge(); + Review comment: can we add some docs about how the predicate is used, is it evicting when true or false ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 389567) Time Spent: 0.5h (was: 20m) > Add necessary endpoints for proactive cache eviction > > > Key: HIVE-22821 > URL: https://issues.apache.org/jira/browse/HIVE-22821 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch, > HIVE-22821.2.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Implement the parts required for iHS2 -> LLAP daemons communication: > * protobuf message schema and endpoints > * Hive configuration > * for use cases: > ** dropping db > ** dropping table > ** dropping partition from a table -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=389570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389570
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 19/Feb/20 19:18
Start Date: 19/Feb/20 19:18
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r381452847
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
##
@@ -333,6 +333,23 @@ public GetTokenResponseProto
getDelegationToken(RpcController controller,
}
}
+ @Override
+ public LlapDaemonProtocolProtos.EvictEntityResponseProto evictEntity(
+ RpcController controller,
LlapDaemonProtocolProtos.EvictEntityRequestProto protoRequest)
+ throws ServiceException {
+LlapDaemonProtocolProtos.EvictEntityResponseProto.Builder
responseProtoBuilder =
+LlapDaemonProtocolProtos.EvictEntityResponseProto.newBuilder();
+
+LlapIo llapIo = LlapProxy.getIo();
+if (llapIo != null) {
+ llapIo.evictEntity(protoRequest);
+ responseProtoBuilder.setAck(true);
Review comment:
how about we do like for purge we return the amount of data purged ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389570)
Time Spent: 40m (was: 0.5h)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition from a table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=389568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389568
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 19/Feb/20 19:18
Start Date: 19/Feb/20 19:18
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r381442289
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4215,6 +4215,12 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_IO_CVB_BUFFERED_SIZE("hive.llap.io.cvb.memory.consumption.", 1L << 30,
"The amount of bytes used to buffer CVB between IO and Processor
Threads default to 1GB, "
+ "this will be used to compute a best effort queue size for VRBs
produced by a LLAP IO thread."),
+
LLAP_IO_PROACTIVE_EVICTION_ENABLED("hive.llap.io.proactive.eviction.enabled",
true,
+"If true proactive cache eviction is enabled, thus LLAP will
proactively evict buffers" +
+ " that belong to dropped Hive entities (DBs, tables, partitions, or
temp tables."),
+LLAP_IO_PROACTIVE_EVICTION_ASYNC("hive.llap.io.proactive.eviction.async",
true,
Review comment:
Am not sure am getting this, it is async in what sense ? the request is
handled by the rpc framework that is running stuff asynch, i think this is not
needed, or please help me understand which thread you are thinking about
avoiding ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389568)
Time Spent: 0.5h (was: 20m)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition from a table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=389565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389565
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 19/Feb/20 19:18
Start Date: 19/Feb/20 19:18
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r381441375
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4215,6 +4215,12 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_IO_CVB_BUFFERED_SIZE("hive.llap.io.cvb.memory.consumption.", 1L << 30,
"The amount of bytes used to buffer CVB between IO and Processor
Threads default to 1GB, "
+ "this will be used to compute a best effort queue size for VRBs
produced by a LLAP IO thread."),
+
LLAP_IO_PROACTIVE_EVICTION_ENABLED("hive.llap.io.proactive.eviction.enabled",
true,
Review comment:
i do not think this is needed we have lot of configs, let it on by default,
unless you see a reason to turn it off from operation perspective ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389565)
Time Spent: 20m (was: 10m)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition from a table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=389569=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389569
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 19/Feb/20 19:18
Start Date: 19/Feb/20 19:18
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r381452125
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
##
@@ -71,6 +72,11 @@ public long purge() {
return evicted;
}
+ @Override
+ public long evictEntity(Predicate predicate) {
+return 0;
Review comment:
this is missing thought ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389569)
Time Spent: 40m (was: 0.5h)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition from a table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22881) Revise non-recommended Calcite api calls
[
https://issues.apache.org/jira/browse/HIVE-22881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040352#comment-17040352
]
Hive QA commented on HIVE-22881:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 1534 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 69 unchanged - 31
fixed = 71 total (was 100) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 726 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
11s{color} | {color:red} The patch 1852 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} ql generated 0 new + 1531 unchanged - 3 fixed = 1531
total (was 1534) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20733/dev-support/hive-personality.sh
|
| git revision | master / 335c2b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20733/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20733/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20733/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20733/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20733/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Revise non-recommended Calcite api calls
>
>
> Key: HIVE-22881
> URL: https://issues.apache.org/jira/browse/HIVE-22881
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22881.01.patch, HIVE-22881.02.patch,
> HIVE-22881.03.patch, HIVE-22881.03.patch, HIVE-22881.03.patch
>
>
> RexUtil.simplify* methods
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22744) TezTask for the vertex with more than one outedge should have proportional sort memory
[ https://issues.apache.org/jira/browse/HIVE-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22744: Attachment: HIVE-22744.3.patch Status: Patch Available (was: Open) > TezTask for the vertex with more than one outedge should have proportional > sort memory > -- > > Key: HIVE-22744 > URL: https://issues.apache.org/jira/browse/HIVE-22744 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22744.1.patch, HIVE-22744.2.patch, > HIVE-22744.3.patch > > > TezTask for the vertex with more than one outedge should have proportional > sort memory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22744) TezTask for the vertex with more than one outedge should have proportional sort memory
[ https://issues.apache.org/jira/browse/HIVE-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22744: Status: Open (was: Patch Available) > TezTask for the vertex with more than one outedge should have proportional > sort memory > -- > > Key: HIVE-22744 > URL: https://issues.apache.org/jira/browse/HIVE-22744 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22744.1.patch, HIVE-22744.2.patch > > > TezTask for the vertex with more than one outedge should have proportional > sort memory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22359) LLAP: when a node restarts with the exact same host/port in kubernetes it is not detected as a task failure
[
https://issues.apache.org/jira/browse/HIVE-22359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040326#comment-17040326
]
Hive QA commented on HIVE-22359:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993816/HIVE-22359.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18026 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.testAuthorization1
(batchId=305)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20732/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20732/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20732/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993816 - PreCommit-HIVE-Build
> LLAP: when a node restarts with the exact same host/port in kubernetes it is
> not detected as a task failure
> ---
>
> Key: HIVE-22359
> URL: https://issues.apache.org/jira/browse/HIVE-22359
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal Vijayaraghavan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22359.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> │ <14>1 2019-10-16T22:16:39.233Z
> query-coordinator-0-5.query-coordinator-0-service.compute-1569601454-l2x9.svc.cluster.local
> query-coordinator 1 461e5ad9-f05f-11e9-85f7-06e84765763e [mdc@18060
> class="te │
> │ zplugins.LlapTaskCommunicator" level="INFO" thread="IPC Server handler 4 on
> 3"] The tasks we expected to be on the node are not there:
> attempt_1569601631911__1_04_34_0, attempt_15696016319 │
> │ 11__1_04_71_0, attempt_1569601631911__1_04_000191_0,
> attempt_1569601631911__1_04_000211_0,
> attempt_1569601631911__1_04_000229_0,
> attempt_1569601631911__1_04_000231_0, attempt_1 │
> │ 569601631911__1_04_000235_0, attempt_1569601631911__1_04_000242_0,
> attempt_1569601631911__1_04_000160_1,
> attempt_1569601631911__1_04_12_2,
> attempt_1569601631911__1_04_03_2, │
> │ attempt_1569601631911__1_04_56_2,
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22877) Fix decimal boundary check for casting to Decimal64
[
https://issues.apache.org/jira/browse/HIVE-22877?focusedWorklogId=389552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389552
]
ASF GitHub Bot logged work on HIVE-22877:
-
Author: ASF GitHub Bot
Created on: 19/Feb/20 18:36
Start Date: 19/Feb/20 18:36
Worklog Time Spent: 10m
Work Description: mustafaiman commented on pull request #905: HIVE-22877:
Fix decimal boundary for casting to Decimal64
URL: https://github.com/apache/hive/pull/905
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389552)
Time Spent: 20m (was: 10m)
> Fix decimal boundary check for casting to Decimal64
> ---
>
> Key: HIVE-22877
> URL: https://issues.apache.org/jira/browse/HIVE-22877
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22877.1.patch, HIVE-22877.1.patch,
> HIVE-22877.1.patch, HIVE-22877.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> During vectorization, decimal fields that are obtained via generic udfs are
> cast to Decimal64 in some circumstances. For decimal to decimal64 cast, hive
> compares the source column's `scale + precision` to 18(maximum number of
> digits that can be represented by a long). A decimal can fit in a long as
> long as its `precision` is smaller than or equal to 18. Scale is irrelevant.
> Since vectorized generic udf expression takes scale into account, it computes
> wrong output column vector: Decimal instead of Decimal64. This in turn causes
> ClassCastException down the operator chain.
> Below query fails with class cast exception:
>
> {code:java}
> create table mini_store
> (
> s_store_sk int,
> s_store_id string
> )
> row format delimited fields terminated by '\t'
> STORED AS ORC;
> create table mini_sales
> (
> ss_store_sk int,
> ss_quantity int,
> ss_sales_price decimal(7,2)
> )
> row format delimited fields terminated by '\t'
> STORED AS ORC;
> insert into mini_store values (1, 'store');
> insert into mini_sales values (1, 2, 1.2);
> select s_store_id, coalesce(ss_sales_price*ss_quantity,0) sumsales
> from mini_sales, mini_store where ss_store_sk = s_store_sk
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Description:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
WHERE LBC.HL_LOCK_TYPE='e'
OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' OR LS.HL_LOCK_TYPE='w' AND
LS.HL_LOCK_STATE='w')
LIMIT 1;
{code}
was:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
WHERE LBC.HL_LOCK_TYPE='e'
OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w' and
LS.HL_LOCK_STATE='w')
LIMIT 1;
{code}
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
> HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 14138) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID
> = 14138) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE LBC.HL_LOCK_TYPE='e'
>OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
>OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' OR LS.HL_LOCK_TYPE='w'
> AND LS.HL_LOCK_STATE='w')
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22359) LLAP: when a node restarts with the exact same host/port in kubernetes it is not detected as a task failure
[
https://issues.apache.org/jira/browse/HIVE-22359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040268#comment-17040268
]
Hive QA commented on HIVE-22359:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} llap-common in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} llap-tez in master has 18 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-tez: The patch generated 1 new + 37 unchanged - 0
fixed = 38 total (was 37) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20732/dev-support/hive-personality.sh
|
| git revision | master / 4700e21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20732/yetus/diff-checkstyle-llap-tez.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20732/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-common llap-tez U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20732/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: when a node restarts with the exact same host/port in kubernetes it is
> not detected as a task failure
> ---
>
> Key: HIVE-22359
> URL: https://issues.apache.org/jira/browse/HIVE-22359
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal Vijayaraghavan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22359.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> │ <14>1 2019-10-16T22:16:39.233Z
> query-coordinator-0-5.query-coordinator-0-service.compute-1569601454-l2x9.svc.cluster.local
> query-coordinator 1 461e5ad9-f05f-11e9-85f7-06e84765763e [mdc@18060
> class="te │
> │ zplugins.LlapTaskCommunicator" level="INFO" thread="IPC Server handler 4 on
> 3"] The tasks we
[jira] [Updated] (HIVE-22589) Add storage support for ProlepticCalendar in ORC, Parquet, and Avro
[ https://issues.apache.org/jira/browse/HIVE-22589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22589: --- Fix Version/s: (was: 3.1.3) (was: 3.2.0) Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. > Add storage support for ProlepticCalendar in ORC, Parquet, and Avro > --- > > Key: HIVE-22589 > URL: https://issues.apache.org/jira/browse/HIVE-22589 > Project: Hive > Issue Type: Bug > Components: Avro, ORC, Parquet >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22589.01.patch, HIVE-22589.02.patch, > HIVE-22589.03.patch, HIVE-22589.04.patch, HIVE-22589.05.patch, > HIVE-22589.06.patch, HIVE-22589.07.patch, HIVE-22589.07.patch, > HIVE-22589.07.patch, HIVE-22589.07.patch, HIVE-22589.08.patch, > HIVE-22589.08.patch, HIVE-22589.patch, HIVE-22589.patch > > > Hive recently moved its processing to the proleptic calendar, which has > created some issues for users who have dates before 1580 AD. > HIVE-22405 extended the column vectors for times & dates to encode which > calendar they are using. > This issue is to support proleptic calendar in ORC, Parquet, and Avro, when > files are written/read by Hive. To preserve compatibility with other engines > until they upgrade their readers, files will be written using hybrid calendar > by default. Default behavior when files do not contain calendar information > in their metadata is configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22589) Add storage support for ProlepticCalendar in ORC, Parquet, and Avro
[ https://issues.apache.org/jira/browse/HIVE-22589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040262#comment-17040262 ] Prasanth Jayachandran commented on HIVE-22589: -- +1 > Add storage support for ProlepticCalendar in ORC, Parquet, and Avro > --- > > Key: HIVE-22589 > URL: https://issues.apache.org/jira/browse/HIVE-22589 > Project: Hive > Issue Type: Bug > Components: Avro, ORC, Parquet >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0, 3.2.0, 3.1.3 > > Attachments: HIVE-22589.01.patch, HIVE-22589.02.patch, > HIVE-22589.03.patch, HIVE-22589.04.patch, HIVE-22589.05.patch, > HIVE-22589.06.patch, HIVE-22589.07.patch, HIVE-22589.07.patch, > HIVE-22589.07.patch, HIVE-22589.07.patch, HIVE-22589.08.patch, > HIVE-22589.08.patch, HIVE-22589.patch, HIVE-22589.patch > > > Hive recently moved its processing to the proleptic calendar, which has > created some issues for users who have dates before 1580 AD. > HIVE-22405 extended the column vectors for times & dates to encode which > calendar they are using. > This issue is to support proleptic calendar in ORC, Parquet, and Avro, when > files are written/read by Hive. To preserve compatibility with other engines > until they upgrade their readers, files will be written using hybrid calendar > by default. Default behavior when files do not contain calendar information > in their metadata is configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22589) Add storage support for ProlepticCalendar in ORC, Parquet, and Avro
[
https://issues.apache.org/jira/browse/HIVE-22589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040259#comment-17040259
]
Hive QA commented on HIVE-22589:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} storage-api in master has 58 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
38s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} common: The patch generated 4 new + 369 unchanged - 0
fixed = 373 total (was 369) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} serde: The patch generated 2 new + 104 unchanged - 3
fixed = 106 total (was 107) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} ql: The patch generated 27 new + 1560 unchanged - 11
fixed = 1587 total (was 1571) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} llap-server: The patch generated 3 new + 184 unchanged
- 1 fixed = 187 total (was 185) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
32s{color} | {color:red} root: The patch generated 37 new + 2223 unchanged - 15
fixed = 2260 total (was 2238) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
46s{color} | {color:red} patch/standalone-metastore/metastore-common cannot run
setBugDatabaseInfo from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
35s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} patch/serde
[jira] [Work started] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22888 started by Denys Kuzmenko.
-
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
> HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 14138) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID
> = 14138) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE LBC.HL_LOCK_TYPE='e'
>OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
>OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w'
> and LS.HL_LOCK_STATE='w')
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Status: Open (was: Patch Available)
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
> HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 14138) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID
> = 14138) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE LBC.HL_LOCK_TYPE='e'
>OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
>OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w'
> and LS.HL_LOCK_STATE='w')
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22589) Add storage support for ProlepticCalendar in ORC, Parquet, and Avro
[
https://issues.apache.org/jira/browse/HIVE-22589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040242#comment-17040242
]
Hive QA commented on HIVE-22589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993799/HIVE-22589.08.patch
{color:green}SUCCESS:{color} +1 due to 31 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18045 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20731/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20731/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20731/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993799 - PreCommit-HIVE-Build
> Add storage support for ProlepticCalendar in ORC, Parquet, and Avro
> ---
>
> Key: HIVE-22589
> URL: https://issues.apache.org/jira/browse/HIVE-22589
> Project: Hive
> Issue Type: Bug
> Components: Avro, ORC, Parquet
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0, 3.2.0, 3.1.3
>
> Attachments: HIVE-22589.01.patch, HIVE-22589.02.patch,
> HIVE-22589.03.patch, HIVE-22589.04.patch, HIVE-22589.05.patch,
> HIVE-22589.06.patch, HIVE-22589.07.patch, HIVE-22589.07.patch,
> HIVE-22589.07.patch, HIVE-22589.07.patch, HIVE-22589.08.patch,
> HIVE-22589.08.patch, HIVE-22589.patch, HIVE-22589.patch
>
>
> Hive recently moved its processing to the proleptic calendar, which has
> created some issues for users who have dates before 1580 AD.
> HIVE-22405 extended the column vectors for times & dates to encode which
> calendar they are using.
> This issue is to support proleptic calendar in ORC, Parquet, and Avro, when
> files are written/read by Hive. To preserve compatibility with other engines
> until they upgrade their readers, files will be written using hybrid calendar
> by default. Default behavior when files do not contain calendar information
> in their metadata is configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Description:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
WHERE LBC.HL_LOCK_TYPE='e'
OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w' and
LS.HL_LOCK_STATE='w')
LIMIT 1;
{code}
was:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
WHERE LBC.HL_LOCK_TYPE='e'
OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w' and
LS.HL_LOCK_STATE='w');
{code}
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
> HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 14138) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID
> = 14138) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE LBC.HL_LOCK_TYPE='e'
>OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
>OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w'
> and LS.HL_LOCK_STATE='w')
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Description:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
WHERE LBC.HL_LOCK_TYPE='e'
OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w' and
LS.HL_LOCK_STATE='w');
{code}
was:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
{code}
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
> HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 14138) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID
> = 14138) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE LBC.HL_LOCK_TYPE='e'
>OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
>OR LBC.HL_LOCK_TYPE='r' AND (LS.HL_LOCK_TYPE='e' or LS.HL_LOCK_TYPE='w'
> and LS.HL_LOCK_STATE='w');
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Description:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
{code}
was:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON (LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
{code}
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
> HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 14138) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID
> = 14138) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22891) Skip PartitonDesc Extraction In CombineHiveRecord For Non-LLAP Execution Mode
[
https://issues.apache.org/jira/browse/HIVE-22891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040192#comment-17040192
]
Ádám Szita commented on HIVE-22891:
---
Looks good to me.
Although I would leave the precondition check in HiveInputFormat#wrapForLlap:
I think we could leave the config check as it is (seeing if LLAP IO is enabled)
just for future fool-proofness.
> Skip PartitonDesc Extraction In CombineHiveRecord For Non-LLAP Execution Mode
> -
>
> Key: HIVE-22891
> URL: https://issues.apache.org/jira/browse/HIVE-22891
> Project: Hive
> Issue Type: Task
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22891.01.patch, HIVE-22891.02.patch
>
>
> {code:java}
> try {
> // TODO: refactor this out
> if (pathToPartInfo == null) {
> MapWork mrwork;
> if (HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
> mrwork = (MapWork) Utilities.getMergeWork(jobConf);
> if (mrwork == null) {
> mrwork = Utilities.getMapWork(jobConf);
> }
> } else {
> mrwork = Utilities.getMapWork(jobConf);
> }
> pathToPartInfo = mrwork.getPathToPartitionInfo();
> } PartitionDesc part = extractSinglePartSpec(hsplit);
> inputFormat = HiveInputFormat.wrapForLlap(inputFormat, jobConf, part);
> } catch (HiveException e) {
> throw new IOException(e);
> }
> {code}
> The above piece of code in CombineHiveRecordReader.java was introduced in
> HIVE-15147. This overwrites inputFormat based on the PartitionDesc which is
> not required in non-LLAP mode of execution as the method
> HiveInputFormat.wrapForLlap() simply returns the previously defined
> inputFormat in case of non-LLAP mode. The method call extractSinglePartSpec()
> has some serious performance implications. If there are large no. of small
> files, each call in the method extractSinglePartSpec() takes approx ~ (2 - 3)
> seconds. Hence the same query which runs in Hive 1.x / Hive 2 is way faster
> than the query run on latest hive.
> {code:java}
> 2020-02-11 07:15:04,701 INFO [main]
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl: Reading ORC rows from
> 2020-02-11 07:15:06,468 WARN [main]
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader: Multiple partitions
> found; not going to pass a part spec to LLAP IO: {{logdate=2020-02-03,
> hour=01, event=win}} and {{logdate=2020-02-03, hour=02, event=act}}
> 2020-02-11 07:15:06,468 INFO [main]
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader: succeeded in getting
> org.apache.hadoop.mapred.FileSplit{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Description:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 14138) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
14138) LBC
ON (LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
{code}
was:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 12143) LS
LEFT JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID =
12143) LBC
ON LS.HL_DB = LBC.HL_DB
AND LS.HL_TABLE = LBC.HL_TABLE
AND LS.HL_PARTITION = LBC.HL_PARTITION
WHERE LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL
OR (LBC.HL_TABLE IS NOT NULL
AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NOT NULL)
)
{code}
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_DB, HL_TABLE, HL_PARTITION,
> HL_LOCK_STATE, HL_LOCK_TYPE, HL_TXNID FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 14138) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID
> = 14138) LBC
> ON (LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LS.HL_TABLE = LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LS.HL_PARTITION = LBC.HL_PARTITION))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22906) Redundant checkLock Mutex blocks concurrent Lock requests
[
https://issues.apache.org/jira/browse/HIVE-22906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040175#comment-17040175
]
Hive QA commented on HIVE-22906:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993795/HIVE-22906.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18026 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20730/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20730/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20730/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993795 - PreCommit-HIVE-Build
> Redundant checkLock Mutex blocks concurrent Lock requests
> -
>
> Key: HIVE-22906
> URL: https://issues.apache.org/jira/browse/HIVE-22906
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22906.1.patch
>
>
> enqueueLocks is already serialising locks creation via (SELECT NL_NEXT FROM
> NEXT_LOCK_ID FOR UPDATE). Requested locks would be assigned 'W' state.
> checkLock is iterating over the sorted set of conflicting locks below current
> EXT_LOCK_ID. It does handle the situation when there is conflicting lock with
> lower ID in 'W' state - lock request would be denied and retried later.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22006) Hive parquet timestamp compatibility, part 2
[ https://issues.apache.org/jira/browse/HIVE-22006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040145#comment-17040145 ] H. Vetinari edited comment on HIVE-22006 at 2/19/20 3:11 PM: - [~klcopp] I hadn't seen that document, that's very cool to see, thanks! It hasn't seen more than a typo fix in more than a year though (at least what's noted in the changelog) - is this tracked somewhere in JIRA (umbrella / per project / subtasks)? Nevermind, found the one for Hive at least: HIVE-21348 was (Author: h-vetinari): [~klcopp] I hadn't seen that document, that's very cool to see, thanks! It hasn't seen more than a typo fix in more than a year though (at least what's noted in the changelog) - is this tracked somewhere in JIRA (umbrella / per project / subtasks)? > Hive parquet timestamp compatibility, part 2 > > > Key: HIVE-22006 > URL: https://issues.apache.org/jira/browse/HIVE-22006 > Project: Hive > Issue Type: Bug >Affects Versions: All Versions >Reporter: H. Vetinari >Priority: Major > > The interaction between HIVE / IMPALA / SPARK writing timestamps is a major > source of headaches in every scenario where such interaction cannot be > avoided. > HIVE-9482 added hive.parquet.timestamp.skip.conversion, which *only* affects > the *reading* of timestamps. > It formulates the next steps as: > > Later fix will change the write path to not convert, and stop the > > read-conversion even for files written by Hive itself. > At the very least, HIVE needs a switch to also turn off the conversion on > writes. That would at least allow a setup where all three of HIVE / IMPALA / > SPARK can be configured not to convert on read/write, and can hence safely > work on the same data -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22006) Hive parquet timestamp compatibility, part 2
[ https://issues.apache.org/jira/browse/HIVE-22006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040145#comment-17040145 ] H. Vetinari commented on HIVE-22006: [~klcopp] I hadn't seen that document, that's very cool to see, thanks! It hasn't seen more than a typo fix in more than a year though (at least what's noted in the changelog) - is this tracked somewhere in JIRA (umbrella / per project / subtasks)? > Hive parquet timestamp compatibility, part 2 > > > Key: HIVE-22006 > URL: https://issues.apache.org/jira/browse/HIVE-22006 > Project: Hive > Issue Type: Bug >Affects Versions: All Versions >Reporter: H. Vetinari >Priority: Major > > The interaction between HIVE / IMPALA / SPARK writing timestamps is a major > source of headaches in every scenario where such interaction cannot be > avoided. > HIVE-9482 added hive.parquet.timestamp.skip.conversion, which *only* affects > the *reading* of timestamps. > It formulates the next steps as: > > Later fix will change the write path to not convert, and stop the > > read-conversion even for files written by Hive itself. > At the very least, HIVE needs a switch to also turn off the conversion on > writes. That would at least allow a setup where all three of HIVE / IMPALA / > SPARK can be configured not to convert on read/write, and can hence safely > work on the same data -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22006) Hive parquet timestamp compatibility, part 2
[ https://issues.apache.org/jira/browse/HIVE-22006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040113#comment-17040113 ] Karen Coppage commented on HIVE-22006: -- [~h-vetinari] Have you seen [this document|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q]? Read path for int64 (HIVE-21050, HIVE-21215) is committed to master and write path (HIVE-21216) should be committed soon. > Hive parquet timestamp compatibility, part 2 > > > Key: HIVE-22006 > URL: https://issues.apache.org/jira/browse/HIVE-22006 > Project: Hive > Issue Type: Bug >Affects Versions: All Versions >Reporter: H. Vetinari >Priority: Major > > The interaction between HIVE / IMPALA / SPARK writing timestamps is a major > source of headaches in every scenario where such interaction cannot be > avoided. > HIVE-9482 added hive.parquet.timestamp.skip.conversion, which *only* affects > the *reading* of timestamps. > It formulates the next steps as: > > Later fix will change the write path to not convert, and stop the > > read-conversion even for files written by Hive itself. > At the very least, HIVE needs a switch to also turn off the conversion on > writes. That would at least allow a setup where all three of HIVE / IMPALA / > SPARK can be configured not to convert on read/write, and can hence safely > work on the same data -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.4.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.3.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22906) Redundant checkLock Mutex blocks concurrent Lock requests
[
https://issues.apache.org/jira/browse/HIVE-22906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040099#comment-17040099
]
Hive QA commented on HIVE-22906:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 584 unchanged - 1 fixed = 584 total (was 585) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20730/dev-support/hive-personality.sh
|
| git revision | master / 4700e21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20730/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20730/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Redundant checkLock Mutex blocks concurrent Lock requests
> -
>
> Key: HIVE-22906
> URL: https://issues.apache.org/jira/browse/HIVE-22906
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22906.1.patch
>
>
> enqueueLocks is already serialising locks creation via (SELECT NL_NEXT FROM
> NEXT_LOCK_ID FOR UPDATE). Requested locks would be assigned 'W' state.
> checkLock is iterating over the sorted set of conflicting locks below current
> EXT_LOCK_ID. It does handle the situation when there is conflicting lock with
> lower ID in 'W' state - lock request would be denied and retried later.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040080#comment-17040080
]
Zoltan Chovan edited comment on HIVE-22899 at 2/19/20 2:10 PM:
---
[~pvary] turns out that this specific test (rfc5424_parser_file_pruning.q) is
excluded[1] from the tests (note, it still shows up in the batches, but then
I'm assuming it gets ignored by the TestCliDriver). I'll remove my changes
regarding both the q and out files.
[1]
[https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java#L254|https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java#L254]
was (Author: zchovan):
[~pvary] turns out that this specific test (rfc5424_parser_file_pruning.q) is
excluded from the tests (note, it still shows up in the batches, but then I'm
assumign it gets ignored by the TestCliDriver). I'll remove my changes
regarding both the q and out files.
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22585) Clean up catalog/db/table name usage
[
https://issues.apache.org/jira/browse/HIVE-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040078#comment-17040078
]
Hive QA commented on HIVE-22585:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993796/HIVE-22585.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 18026 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_table]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_after_multiple_inserts_special_characters]
(batchId=83)
org.apache.hadoop.hive.ql.TestTxnCommands.testQuotedIdentifier (batchId=360)
org.apache.hadoop.hive.ql.TestTxnCommands.testQuotedIdentifier2 (batchId=360)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testQuotedIdentifier
(batchId=341)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testQuotedIdentifier2
(batchId=341)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20729/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20729/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20729/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993796 - PreCommit-HIVE-Build
> Clean up catalog/db/table name usage
>
>
> Key: HIVE-22585
> URL: https://issues.apache.org/jira/browse/HIVE-22585
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available, refactor
> Attachments: HIVE-22585.01.patch, HIVE-22585.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a followup to HIVE-21198 to address some additional improvement ideas
> for the TableName object mentioned in
> [https://github.com/apache/hive/pull/550] and attempt to remove all the fishy
> usages of db/tablenames, as a number of places still rely on certain state
> changes/black magic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040080#comment-17040080
]
Zoltan Chovan commented on HIVE-22899:
--
[~pvary] turns out that this specific test (rfc5424_parser_file_pruning.q) is
excluded from the tests (note, it still shows up in the batches, but then I'm
assumign it gets ignored by the TestCliDriver). I'll remove my changes
regarding both the q and out files.
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21961) Update jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HIVE-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-21961: Attachment: HIVE-21961.03.patch > Update jetty version to 9.4.x > - > > Key: HIVE-21961 > URL: https://issues.apache.org/jira/browse/HIVE-21961 > Project: Hive > Issue Type: Task >Reporter: Oleksiy Sayankin >Assignee: László Bodor >Priority: Major > Attachments: HIVE-21961.02.patch, HIVE-21961.03.patch, > HIVE-21961.patch > > > Update jetty version to 9.4.x -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
However most of the entries have {{HL_ACQUIRED_AT=NULL}} in {{HIVE_LOCKS}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
Hot-fix is to remove orphan/lost locks from {{HIVE_LOCKS}}, but this does not
solve the problem in the future.
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE |
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
However most of the entries are {{HL_ACQUIRED_AT=NULL}} in {{HIVE_LOCKS}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
|
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040033#comment-17040033
]
Peter Vary commented on HIVE-22899:
---
you can check any of the green pre-commit job test results for this specific
test :)
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
The delete sql on {{HIVE_LOCKS}} is based on column {{HL_ACQUIRED_AT}}, however
most of the entries are {{NULL}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
So after deleting,
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
The delete sql on {{HIVE_LOCKS}} is based on column {{HL_ACQUIRED_AT}}, however
most of the entries are {{NULL}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 |
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040022#comment-17040022
]
Zoltan Chovan commented on HIVE-22899:
--
[~pvary]
Regarding the lines "TOTAL_TABLE_ROWS_WRITTEN" and "CREATED_FILES" in
rfc5424_parser_file_pruning.q.out I wasn't sure either. I've run the following
in the latest upstream master branch:
{code:java}
mvn test -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=rfc5424_parser_file_pruning.q
{code}
and it fails. Not sure if this shows up in current precommits or not.
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22585) Clean up catalog/db/table name usage
[
https://issues.apache.org/jira/browse/HIVE-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040014#comment-17040014
]
Hive QA commented on HIVE-22585:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 8 new + 639 unchanged - 0
fixed = 647 total (was 639) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20729/dev-support/hive-personality.sh
|
| git revision | master / 4700e21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20729/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20729/yetus/patch-asflicense-problems.txt
|
| modules | C: metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20729/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up catalog/db/table name usage
>
>
> Key: HIVE-22585
> URL: https://issues.apache.org/jira/browse/HIVE-22585
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available, refactor
> Attachments: HIVE-22585.01.patch, HIVE-22585.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a followup to HIVE-21198 to address some additional improvement ideas
> for the TableName object mentioned in
> [https://github.com/apache/hive/pull/550] and attempt to remove all the fishy
> usages of db/tablenames, as a number of places still rely on certain state
> changes/black magic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22900) Predicate Push Down Of Like Filter While Fetching Partition Data From MetaStore
[ https://issues.apache.org/jira/browse/HIVE-22900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22900: - Attachment: HIVE-22900.02.patch > Predicate Push Down Of Like Filter While Fetching Partition Data From > MetaStore > --- > > Key: HIVE-22900 > URL: https://issues.apache.org/jira/browse/HIVE-22900 > Project: Hive > Issue Type: New Feature >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22900.01.patch, HIVE-22900.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently PPD is disabled for like filter while fetching partition data from > metastore. The following patch covers all the test cases mentioned in > HIVE-5134 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22721) Add option for queries to only read from LLAP cache
[ https://issues.apache.org/jira/browse/HIVE-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22721 started by Ádám Szita. - > Add option for queries to only read from LLAP cache > --- > > Key: HIVE-22721 > URL: https://issues.apache.org/jira/browse/HIVE-22721 > Project: Hive > Issue Type: Test >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22721.0.patch > > > Testing features of LLAP cache sometimes requires to validate if e.g. a > particular table/partition is cached, or not. > This is to avoid relying on counters that are dependent on the underlying > (ORC) file format (which may produce different number of bytes among its > different versions). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22721) Add option for queries to only read from LLAP cache
[ https://issues.apache.org/jira/browse/HIVE-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22721: -- Attachment: HIVE-22721.0.patch > Add option for queries to only read from LLAP cache > --- > > Key: HIVE-22721 > URL: https://issues.apache.org/jira/browse/HIVE-22721 > Project: Hive > Issue Type: Test >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22721.0.patch > > > Testing features of LLAP cache sometimes requires to validate if e.g. a > particular table/partition is cached, or not. > This is to avoid relying on counters that are dependent on the underlying > (ORC) file format (which may produce different number of bytes among its > different versions). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22721) Add option for queries to only read from LLAP cache
[ https://issues.apache.org/jira/browse/HIVE-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22721: -- Status: Patch Available (was: In Progress) > Add option for queries to only read from LLAP cache > --- > > Key: HIVE-22721 > URL: https://issues.apache.org/jira/browse/HIVE-22721 > Project: Hive > Issue Type: Test >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22721.0.patch > > > Testing features of LLAP cache sometimes requires to validate if e.g. a > particular table/partition is cached, or not. > This is to avoid relying on counters that are dependent on the underlying > (ORC) file format (which may produce different number of bytes among its > different versions). -- This message was sent by Atlassian Jira (v8.3.4#803005)
