[jira] [Commented] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061445#comment-17061445 ] Hive QA commented on HIVE-22990: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996969/HIVE-22990.25.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 18108 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21153/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21153/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21153/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12996969 - PreCommit-HIVE-Build > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.25.patch, HIVE-22990.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23022) Arrow deserializer should ensure size of hive vector equal to arrow vector
[ https://issues.apache.org/jira/browse/HIVE-23022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-23022: --- Resolution: Fixed Status: Resolved (was: Patch Available) [^HIVE-23022.01.patch] committed to master. Thanks [~ShubhamChaurasia] for fixing it. > Arrow deserializer should ensure size of hive vector equal to arrow vector > -- > > Key: HIVE-23022 > URL: https://issues.apache.org/jira/browse/HIVE-23022 > Project: Hive > Issue Type: Bug > Components: llap, Serializers/Deserializers >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23022.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Arrow deserializer - {{org.apache.hadoop.hive.ql.io.arrow.Deserializer}} in > some cases does not set the size of hive vector correctly. Size of hive > vector should be set at least equal to arrow vector to be able to read > (accommodate) it fully. > Following exception can be seen when we try to read (using > {{LlapArrowRowInputFormat}} ) some table which contains complex types (struct > nested in array to be specific) and number of rows in table is more than > default (1024) batch/vector size. > {code:java} > Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024 > at > org.apache.hadoop.hive.ql.io.arrow.Deserializer.readStruct(Deserializer.java:440) > at > org.apache.hadoop.hive.ql.io.arrow.Deserializer.read(Deserializer.java:143) > at > org.apache.hadoop.hive.ql.io.arrow.Deserializer.readList(Deserializer.java:394) > at > org.apache.hadoop.hive.ql.io.arrow.Deserializer.read(Deserializer.java:137) > at > org.apache.hadoop.hive.ql.io.arrow.Deserializer.deserialize(Deserializer.java:122) > at > org.apache.hadoop.hive.ql.io.arrow.ArrowColumnarBatchSerDe.deserialize(ArrowColumnarBatchSerDe.java:284) > at > org.apache.hadoop.hive.llap.LlapArrowRowRecordReader.next(LlapArrowRowRecordReader.java:75) > ... 23 more > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.10.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.2.patch, > HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch, > HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[ https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22841: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~krisden]! > ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner > IllegalArgumentException on invalid cookie signature > - > > Key: HIVE-22841 > URL: https://issues.apache.org/jira/browse/HIVE-22841 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22841.1.patch, HIVE-22841.2.patch, > HIVE-22841.3.patch > > > Currently CookieSigner throws an IllegalArgumentException if the cookie > signature is invalid. > {code:java} > if (!MessageDigest.isEqual(originalSignature.getBytes(), > currentSignature.getBytes())) { > throw new IllegalArgumentException("Invalid sign, original = " + > originalSignature + > " current = " + currentSignature); > } > {code} > CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie > and doesn't handle the IllegalArgumentException. It is only checking if the > value from the cookie is null or not. > https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295 > {code:java} > currValue = signer.verifyAndExtract(currValue); > // Retrieve the user name, do the final validation step. > if (currValue != null) { > {code} > This should be fixed to either: > a) Have CookieSigner not return an IllegalArgumentException > b) Improve ThriftHttpServlet to handle CookieSigner throwing an > IllegalArgumentException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22901) Variable substitution can lead to OOM on circular references
[ https://issues.apache.org/jira/browse/HIVE-22901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22901: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~dvoros] for improving on this! > Variable substitution can lead to OOM on circular references > > > Key: HIVE-22901 > URL: https://issues.apache.org/jira/browse/HIVE-22901 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.2 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22901.1.patch, HIVE-22901.2.patch, > HIVE-22901.2.patch, HIVE-22901.2.patch, HIVE-22901.3.patch > > > {{SystemVariables#substitute()}} is dealing with circular references between > variables by only doing the substitution 40 times by default. If the > substituted part is sufficiently large though, it's possible that the > substitution will produce a string bigger than the heap size within the 40 > executions. > Take the following test case that fails with OOM in current master (third > round of execution would need 10G heap, while running with only 2G): > {code} > @Test > public void testSubstitute() { > String randomPart = RandomStringUtils.random(100_000); > String reference = "${hiveconf:myTestVariable}"; > StringBuilder longStringWithReferences = new StringBuilder(); > for(int i = 0; i < 10; i ++) { > longStringWithReferences.append(randomPart).append(reference); > } > SystemVariables uut = new SystemVariables(); > HiveConf conf = new HiveConf(); > conf.set("myTestVariable", longStringWithReferences.toString()); > uut.substitute(conf, longStringWithReferences.toString(), 40); > } > {code} > Produces: > {code} > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) > at java.lang.StringBuilder.append(StringBuilder.java:136) > at > org.apache.hadoop.hive.conf.SystemVariables.substitute(SystemVariables.java:110) > at > org.apache.hadoop.hive.conf.SystemVariablesTest.testSubstitute(SystemVariablesTest.java:27) > {code} > We should check the size of the substituted query and bail out earlier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[ https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22539: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~krisden]! > HiveServer2 SPNEGO authentication should skip if authorization header is empty > -- > > Key: HIVE-22539 > URL: https://issues.apache.org/jira/browse/HIVE-22539 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch, > HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch, > HIVE-22539.6.patch, HIVE-22539.7.patch > > > Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos > before checking header. This can be checked up front to avoid doing any > Kerberos related work if the header is empty. This is helpful in a lot of > cases since typically the first request is empty with the client waiting for > a 401 before returning the Authorization header. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061425#comment-17061425 ] Hive QA commented on HIVE-22990: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 44s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 44s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} The patch ql passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 806 unchanged - 3 fixed = 806 total (was 809) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21153/dev-support/hive-personality.sh | | git revision | master / 26cc315 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21153/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.25.patch, HIVE-22990.
[jira] [Updated] (HIVE-23034) Arrow serializer should not keep the reference of arrow offset and validity buffers
[ https://issues.apache.org/jira/browse/HIVE-23034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-23034: - Attachment: HIVE-23034.01.patch Status: Patch Available (was: Open) > Arrow serializer should not keep the reference of arrow offset and validity > buffers > --- > > Key: HIVE-23034 > URL: https://issues.apache.org/jira/browse/HIVE-23034 > Project: Hive > Issue Type: Bug > Components: llap, Serializers/Deserializers >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23034.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, a part of writeList() method in arrow serializer is implemented > like - > {code:java} > final ArrowBuf offsetBuffer = arrowVector.getOffsetBuffer(); > int nextOffset = 0; > for (int rowIndex = 0; rowIndex < size; rowIndex++) { > int selectedIndex = rowIndex; > if (vectorizedRowBatch.selectedInUse) { > selectedIndex = vectorizedRowBatch.selected[rowIndex]; > } > if (hiveVector.isNull[selectedIndex]) { > offsetBuffer.setInt(rowIndex * OFFSET_WIDTH, nextOffset); > } else { > offsetBuffer.setInt(rowIndex * OFFSET_WIDTH, nextOffset); > nextOffset += (int) hiveVector.lengths[selectedIndex]; > arrowVector.setNotNull(rowIndex); > } > } > offsetBuffer.setInt(size * OFFSET_WIDTH, nextOffset); > {code} > 1) Here we obtain a reference to {{final ArrowBuf offsetBuffer = > arrowVector.getOffsetBuffer();}} and keep updating the arrow vector and > offset vector. > Problem - > {{arrowVector.setNotNull(rowIndex)}} keeps checking the index and reallocates > the offset and validity buffers when a threshold is crossed, updates the > references internally and also releases the old buffers (which decrements the > buffer reference count). Now the reference which we obtained in 1) becomes > obsolete. Furthermore if try to read or write old buffer, we see - > {code:java} > Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0 > at > io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1413) > at io.netty.buffer.ArrowBuf.checkIndexD(ArrowBuf.java:131) > at io.netty.buffer.ArrowBuf.chk(ArrowBuf.java:162) > at io.netty.buffer.ArrowBuf.setInt(ArrowBuf.java:656) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:432) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:285) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:352) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:288) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:419) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:285) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:205) > {code} > > Solution - > This can be fixed by getting the buffers each time ( > {{arrowVector.getOffsetBuffer()}} ) we want to update them. > In our internal tests, this is very frequently seen on arrow 0.8.0 but not on > 0.10.0 but should be handled the same way for 0.10.0 too as it does the same > thing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23034) Arrow serializer should not keep the reference of arrow offset and validity buffers
[ https://issues.apache.org/jira/browse/HIVE-23034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23034: -- Labels: pull-request-available (was: ) > Arrow serializer should not keep the reference of arrow offset and validity > buffers > --- > > Key: HIVE-23034 > URL: https://issues.apache.org/jira/browse/HIVE-23034 > Project: Hive > Issue Type: Bug > Components: llap, Serializers/Deserializers >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > > Currently, a part of writeList() method in arrow serializer is implemented > like - > {code:java} > final ArrowBuf offsetBuffer = arrowVector.getOffsetBuffer(); > int nextOffset = 0; > for (int rowIndex = 0; rowIndex < size; rowIndex++) { > int selectedIndex = rowIndex; > if (vectorizedRowBatch.selectedInUse) { > selectedIndex = vectorizedRowBatch.selected[rowIndex]; > } > if (hiveVector.isNull[selectedIndex]) { > offsetBuffer.setInt(rowIndex * OFFSET_WIDTH, nextOffset); > } else { > offsetBuffer.setInt(rowIndex * OFFSET_WIDTH, nextOffset); > nextOffset += (int) hiveVector.lengths[selectedIndex]; > arrowVector.setNotNull(rowIndex); > } > } > offsetBuffer.setInt(size * OFFSET_WIDTH, nextOffset); > {code} > 1) Here we obtain a reference to {{final ArrowBuf offsetBuffer = > arrowVector.getOffsetBuffer();}} and keep updating the arrow vector and > offset vector. > Problem - > {{arrowVector.setNotNull(rowIndex)}} keeps checking the index and reallocates > the offset and validity buffers when a threshold is crossed, updates the > references internally and also releases the old buffers (which decrements the > buffer reference count). Now the reference which we obtained in 1) becomes > obsolete. Furthermore if try to read or write old buffer, we see - > {code:java} > Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0 > at > io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1413) > at io.netty.buffer.ArrowBuf.checkIndexD(ArrowBuf.java:131) > at io.netty.buffer.ArrowBuf.chk(ArrowBuf.java:162) > at io.netty.buffer.ArrowBuf.setInt(ArrowBuf.java:656) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:432) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:285) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:352) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:288) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:419) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:285) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:205) > {code} > > Solution - > This can be fixed by getting the buffers each time ( > {{arrowVector.getOffsetBuffer()}} ) we want to update them. > In our internal tests, this is very frequently seen on arrow 0.8.0 but not on > 0.10.0 but should be handled the same way for 0.10.0 too as it does the same > thing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23034) Arrow serializer should not keep the reference of arrow offset and validity buffers
[ https://issues.apache.org/jira/browse/HIVE-23034?focusedWorklogId=405235&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405235 ] ASF GitHub Bot logged work on HIVE-23034: - Author: ASF GitHub Bot Created on: 18/Mar/20 05:45 Start Date: 18/Mar/20 05:45 Worklog Time Spent: 10m Work Description: ShubhamChaurasia commented on pull request #957: HIVE-23034: Arrow serializer should not keep the reference of arrow offset and validity buffers URL: https://github.com/apache/hive/pull/957 …ffset and validity buffers This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405235) Remaining Estimate: 0h Time Spent: 10m > Arrow serializer should not keep the reference of arrow offset and validity > buffers > --- > > Key: HIVE-23034 > URL: https://issues.apache.org/jira/browse/HIVE-23034 > Project: Hive > Issue Type: Bug > Components: llap, Serializers/Deserializers >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently, a part of writeList() method in arrow serializer is implemented > like - > {code:java} > final ArrowBuf offsetBuffer = arrowVector.getOffsetBuffer(); > int nextOffset = 0; > for (int rowIndex = 0; rowIndex < size; rowIndex++) { > int selectedIndex = rowIndex; > if (vectorizedRowBatch.selectedInUse) { > selectedIndex = vectorizedRowBatch.selected[rowIndex]; > } > if (hiveVector.isNull[selectedIndex]) { > offsetBuffer.setInt(rowIndex * OFFSET_WIDTH, nextOffset); > } else { > offsetBuffer.setInt(rowIndex * OFFSET_WIDTH, nextOffset); > nextOffset += (int) hiveVector.lengths[selectedIndex]; > arrowVector.setNotNull(rowIndex); > } > } > offsetBuffer.setInt(size * OFFSET_WIDTH, nextOffset); > {code} > 1) Here we obtain a reference to {{final ArrowBuf offsetBuffer = > arrowVector.getOffsetBuffer();}} and keep updating the arrow vector and > offset vector. > Problem - > {{arrowVector.setNotNull(rowIndex)}} keeps checking the index and reallocates > the offset and validity buffers when a threshold is crossed, updates the > references internally and also releases the old buffers (which decrements the > buffer reference count). Now the reference which we obtained in 1) becomes > obsolete. Furthermore if try to read or write old buffer, we see - > {code:java} > Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0 > at > io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1413) > at io.netty.buffer.ArrowBuf.checkIndexD(ArrowBuf.java:131) > at io.netty.buffer.ArrowBuf.chk(ArrowBuf.java:162) > at io.netty.buffer.ArrowBuf.setInt(ArrowBuf.java:656) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:432) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:285) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:352) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:288) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:419) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:285) > at > org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:205) > {code} > > Solution - > This can be fixed by getting the buffers each time ( > {{arrowVector.getOffsetBuffer()}} ) we want to update them. > In our internal tests, this is very frequently seen on arrow 0.8.0 but not on > 0.10.0 but should be handled the same way for 0.10.0 too as it does the same > thing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18983) Add support for table properties inheritance in Create table like
[ https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061399#comment-17061399 ] Hive QA commented on HIVE-18983: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12919862/HIVE-18983.12.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21152/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21152/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21152/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2020-03-18 05:14:40.204 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-21152/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2020-03-18 05:14:40.207 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 26cc315 HIVE-23011: Shared work optimizer should check residual predicates when comparing joins (Jesus Camacho Rodriguez, reviewed by Vineet Garg) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 26cc315 HIVE-23011: Shared work optimizer should check residual predicates when comparing joins (Jesus Camacho Rodriguez, reviewed by Vineet Garg) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2020-03-18 05:14:40.958 + rm -rf ../yetus_PreCommit-HIVE-Build-21152 + mkdir ../yetus_PreCommit-HIVE-Build-21152 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-21152 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-21152/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Trying to apply the patch with -p0 error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not exist in index error: a/ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/create_like.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/create_like2.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/explain_ddl.q.out: does not exist in index error: a/standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java: does not exist in index Trying to apply the patch with -p1 error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:13084 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java' with conflicts. error: patch failed: ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out:309 Falling back to three-way merge... Applied patch to 'ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out' with conflicts. error: patch failed: ql/src/test/results/clientpositive/create_like.q.out:441 Falling back to three-way merge... Applied patch to 'ql/src/test/results/clientpositive/create_like.q.out' with conflicts. error: patch failed: ql/src/test/results/clientpositive/create_like2.q.out:41 Falling back to three-way merge... Applied patch to 'ql/src/test/results/clientpositive/create_like2.q.out' with conflicts. error: patch failed: ql/src/test/results/clientpositive/explain_ddl.q.out:448 Falling back to three-way merge... Applied patch to 'ql/src/test/results/clientpositive/explain_ddl.q.out' with conflicts. error: standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.j
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[ https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061398#comment-17061398 ] Hive QA commented on HIVE-23037: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996944/HIVE-23037.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18108 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence (batchId=252) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21151/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21151/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21151/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12996944 - PreCommit-HIVE-Build > Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus > - > > Key: HIVE-23037 > URL: https://issues.apache.org/jira/browse/HIVE-23037 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23037.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.10.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.2.patch, HIVE-23004.4.patch, HIVE-23004.6.patch, > HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, > HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22932) Unable to kill Beeline with Ctrl+C
[ https://issues.apache.org/jira/browse/HIVE-22932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061381#comment-17061381 ] bianqi edited comment on HIVE-22932 at 3/18/20, 4:41 AM: - {quote}Unknown HS2 problem when communicating with Thrift server. Error: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (state=08S01,code=0){quote} h6. your thrift service is not running or your thift service's port cannot communication with beeline. h6. was (Author: bianqi): {quote}Unknown HS2 problem when communicating with Thrift server. Error: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (state=08S01,code=0) your thrift service is not running or your thift service's port cannot communication with beeline. {quote} > Unable to kill Beeline with Ctrl+C > -- > > Key: HIVE-22932 > URL: https://issues.apache.org/jira/browse/HIVE-22932 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Blocker > > Stopped the server and tried to stop the beeline console with "Ctrl+C". But > it unable to kill the process & gets process gets hanged. > Read call got blocked. > Attached the thread dump. > 0: jdbc:hive2://localhost:1> show tables; > Unknown HS2 problem when communicating with Thrift server. > Error: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe (state=08S01,code=0) > 0: jdbc:hive2://localhost:1> Interrupting... Please be patient this may > take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > 2020-02-26 17:40:42 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode): > "NonBlockingInputStreamThread" #16 daemon prio=5 os_prio=0 > tid=0x7f0318c10800 nid=0x258c in Object.wait() [0x7f031c193000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" #11 daemon prio=9 os_prio=0 tid=0x7f032006c000 > nid=0x257b runnable [0x] >java.lang.Thread.State: RUNNABLE > "C1 CompilerThread3" #10 daemon prio=9 os_prio=0 tid=0x7f0320060800 > nid=0x257a waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread2" #9 daemon prio=9 os_prio=0 tid=0x7f0320056000 > nid=0x2579 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" #8 daemon prio=9 os_prio=0 tid=0x7f0320054000 > nid=0x2578 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" #7 daemon prio=9 os_prio=0 tid=0x7f0320051000 > nid=0x2577 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=0 tid=0x7f032004f000 > nid=0x2576 runnable [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=0 > tid=0x7f032004b800 nid=0x2575 runnable [0x] >java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7f0320035800 > nid=0x2574 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x7f0320003800 nid=0x2572 in > Object.wait() [0x7f0324b1c000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on
[jira] [Commented] (HIVE-22932) Unable to kill Beeline with Ctrl+C
[ https://issues.apache.org/jira/browse/HIVE-22932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061381#comment-17061381 ] bianqi commented on HIVE-22932: --- {quote}Unknown HS2 problem when communicating with Thrift server. Error: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (state=08S01,code=0) your thrift service is not running or your thift service's port cannot communication with beeline. {quote} > Unable to kill Beeline with Ctrl+C > -- > > Key: HIVE-22932 > URL: https://issues.apache.org/jira/browse/HIVE-22932 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Blocker > > Stopped the server and tried to stop the beeline console with "Ctrl+C". But > it unable to kill the process & gets process gets hanged. > Read call got blocked. > Attached the thread dump. > 0: jdbc:hive2://localhost:1> show tables; > Unknown HS2 problem when communicating with Thrift server. > Error: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe (state=08S01,code=0) > 0: jdbc:hive2://localhost:1> Interrupting... Please be patient this may > take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > 2020-02-26 17:40:42 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode): > "NonBlockingInputStreamThread" #16 daemon prio=5 os_prio=0 > tid=0x7f0318c10800 nid=0x258c in Object.wait() [0x7f031c193000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" #11 daemon prio=9 os_prio=0 tid=0x7f032006c000 > nid=0x257b runnable [0x] >java.lang.Thread.State: RUNNABLE > "C1 CompilerThread3" #10 daemon prio=9 os_prio=0 tid=0x7f0320060800 > nid=0x257a waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread2" #9 daemon prio=9 os_prio=0 tid=0x7f0320056000 > nid=0x2579 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" #8 daemon prio=9 os_prio=0 tid=0x7f0320054000 > nid=0x2578 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" #7 daemon prio=9 os_prio=0 tid=0x7f0320051000 > nid=0x2577 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=0 tid=0x7f032004f000 > nid=0x2576 runnable [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=0 > tid=0x7f032004b800 nid=0x2575 runnable [0x] >java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7f0320035800 > nid=0x2574 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x7f0320003800 nid=0x2572 in > Object.wait() [0x7f0324b1c000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe930770> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0xfe930770> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Referenc
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[ https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061378#comment-17061378 ] Hive QA commented on HIVE-23037: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 47s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21151/dev-support/hive-personality.sh | | git revision | master / 26cc315 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21151/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus > - > > Key: HIVE-23037 > URL: https://issues.apache.org/jira/browse/HIVE-23037 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23037.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061368#comment-17061368 ] Hive QA commented on HIVE-23004: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996971/HIVE-23004.9.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 18109 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug] (batchId=192) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_tmp_table] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_decimal_vectorized] (batchId=177) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window_2] (batchId=184) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal64_case_when_nvl] (batchId=177) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal64_case_when_nvl_cbo] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal64_multi_vertex] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_aggregate] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_round] (batchId=176) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_trailing] (batchId=176) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf] (batchId=194) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_reference_windowed] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction2] (batchId=179) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_data_types] (batchId=153) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate] (batchId=126) org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence (batchId=252) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21150/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21150/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21150/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 18 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12996971 - PreCommit-HIVE-Build > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, > HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[ https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=405190&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405190 ] ASF GitHub Bot logged work on HIVE-22821: - Author: ASF GitHub Bot Created on: 18/Mar/20 03:36 Start Date: 18/Mar/20 03:36 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #909: HIVE-22821 URL: https://github.com/apache/hive/pull/909#discussion_r394089629 ## File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java ## @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.llap; + +import java.io.IOException; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import javax.net.SocketFactory; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hive.common.io.CacheTag; +import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos; +import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl; +import org.apache.hadoop.hive.llap.registry.LlapServiceInstance; +import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; +import org.apache.hadoop.net.NetUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Through this class the caller (typically HS2) can request eviction of buffers from LLAP cache by specifying a DB, + * table or partition name/(value). Request sending is implemented here. + */ +public final class ProactiveEviction { + + private ProactiveEviction() { +// Not to be used; + } + + /** + * Trigger LLAP cache eviction of buffers related to entities residing in request parameter. + * @param conf + * @param request + */ + public static void evict(Configuration conf, Request request) { +if (!HiveConf.getBoolVar(conf, HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) { + return; +} + +try { + LlapRegistryService llapRegistryService = LlapRegistryService.getClient(conf); + Collection instances = llapRegistryService.getInstances().getAll(); + if (instances.size() == 0) { +// Not in LLAP mode. +return; + } + ExecutorService executorService = Executors.newCachedThreadPool(); Review comment: I hope you are convinced that Asynchronous stuff can be trick to get right and most of the benefit can turn into debugging nightmare. Let me know if you have more questions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405190) Time Spent: 4h (was: 3h 50m) > Add necessary endpoints for proactive cache eviction > > > Key: HIVE-22821 > URL: https://issues.apache.org/jira/browse/HIVE-22821 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch, > HIVE-22821.2.patch > > Time Spent: 4h > Remaining Estimate: 0h > > Implement the parts required for iHS2 -> LLAP daemons communication: > * protobuf message schema and endpoints > * Hive configuration > * for use cases: > ** dropping db > ** dropping table
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[ https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=405188&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405188 ] ASF GitHub Bot logged work on HIVE-22821: - Author: ASF GitHub Bot Created on: 18/Mar/20 03:35 Start Date: 18/Mar/20 03:35 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #909: HIVE-22821 URL: https://github.com/apache/hive/pull/909#discussion_r394089275 ## File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java ## @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.llap; + +import java.io.IOException; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import javax.net.SocketFactory; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hive.common.io.CacheTag; +import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos; +import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl; +import org.apache.hadoop.hive.llap.registry.LlapServiceInstance; +import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; +import org.apache.hadoop.net.NetUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Through this class the caller (typically HS2) can request eviction of buffers from LLAP cache by specifying a DB, + * table or partition name/(value). Request sending is implemented here. + */ +public final class ProactiveEviction { + + private ProactiveEviction() { +// Not to be used; + } + + /** + * Trigger LLAP cache eviction of buffers related to entities residing in request parameter. + * @param conf + * @param request + */ + public static void evict(Configuration conf, Request request) { +if (!HiveConf.getBoolVar(conf, HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) { + return; +} + +try { + LlapRegistryService llapRegistryService = LlapRegistryService.getClient(conf); + Collection instances = llapRegistryService.getInstances().getAll(); + if (instances.size() == 0) { +// Not in LLAP mode. +return; + } + ExecutorService executorService = Executors.newCachedThreadPool(); Review comment: For number 4 you can refer to this spec https://wiki.sei.cmu.edu/confluence/display/java/TPS02-J.+Ensure+that+tasks+submitted+to+a+thread+pool+are+interruptible This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405188) Time Spent: 3h 50m (was: 3h 40m) > Add necessary endpoints for proactive cache eviction > > > Key: HIVE-22821 > URL: https://issues.apache.org/jira/browse/HIVE-22821 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch, > HIVE-22821.2.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Implement the parts required for iHS2 -> LLAP daemons communication: > * protobuf message schema and endpoints > * Hive configuration > * for use cases: > ** dropping db > ** dropping table
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[ https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=405187&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405187 ] ASF GitHub Bot logged work on HIVE-22821: - Author: ASF GitHub Bot Created on: 18/Mar/20 03:34 Start Date: 18/Mar/20 03:34 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #909: HIVE-22821 URL: https://github.com/apache/hive/pull/909#discussion_r394089108 ## File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java ## @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.llap; + +import java.io.IOException; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import javax.net.SocketFactory; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hive.common.io.CacheTag; +import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos; +import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl; +import org.apache.hadoop.hive.llap.registry.LlapServiceInstance; +import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; +import org.apache.hadoop.net.NetUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Through this class the caller (typically HS2) can request eviction of buffers from LLAP cache by specifying a DB, + * table or partition name/(value). Request sending is implemented here. + */ +public final class ProactiveEviction { + + private ProactiveEviction() { +// Not to be used; + } + + /** + * Trigger LLAP cache eviction of buffers related to entities residing in request parameter. + * @param conf + * @param request + */ + public static void evict(Configuration conf, Request request) { +if (!HiveConf.getBoolVar(conf, HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) { + return; +} + +try { + LlapRegistryService llapRegistryService = LlapRegistryService.getClient(conf); + Collection instances = llapRegistryService.getInstances().getAll(); + if (instances.size() == 0) { +// Not in LLAP mode. +return; + } + ExecutorService executorService = Executors.newCachedThreadPool(); Review comment: To be more clear about point one you can read the [java spec](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html) it is always very good point to start **An unused ExecutorService should be shut down to allow reclamation of its resources.** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405187) Time Spent: 3h 40m (was: 3.5h) > Add necessary endpoints for proactive cache eviction > > > Key: HIVE-22821 > URL: https://issues.apache.org/jira/browse/HIVE-22821 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch, > HIVE-22821.2.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Implement the parts required for iHS2 -> LLAP daemons communication: > * protobuf mes
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[ https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=405173&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405173 ] ASF GitHub Bot logged work on HIVE-22821: - Author: ASF GitHub Bot Created on: 18/Mar/20 03:10 Start Date: 18/Mar/20 03:10 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #909: HIVE-22821 URL: https://github.com/apache/hive/pull/909#discussion_r394083474 ## File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java ## @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.llap; + +import java.io.IOException; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import javax.net.SocketFactory; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hive.common.io.CacheTag; +import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos; +import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl; +import org.apache.hadoop.hive.llap.registry.LlapServiceInstance; +import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService; +import org.apache.hadoop.io.retry.RetryPolicies; +import org.apache.hadoop.io.retry.RetryPolicy; +import org.apache.hadoop.net.NetUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Through this class the caller (typically HS2) can request eviction of buffers from LLAP cache by specifying a DB, + * table or partition name/(value). Request sending is implemented here. + */ +public final class ProactiveEviction { + + private ProactiveEviction() { +// Not to be used; + } + + /** + * Trigger LLAP cache eviction of buffers related to entities residing in request parameter. + * @param conf + * @param request + */ + public static void evict(Configuration conf, Request request) { +if (!HiveConf.getBoolVar(conf, HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) { + return; +} + +try { + LlapRegistryService llapRegistryService = LlapRegistryService.getClient(conf); + Collection instances = llapRegistryService.getInstances().getAll(); + if (instances.size() == 0) { +// Not in LLAP mode. +return; + } + ExecutorService executorService = Executors.newCachedThreadPool(); Review comment: This code has multiple issues and it is not safe. 1. First issue, there is no explicit shutdown of the executor and that is a big issue by it self, because some implementations shutdown themselves in a finalizer **BUT JVM spec do not guaranteed to ever be call finalize()**. 2. Second this code is creating a thread pool on each call, that is a huge resource wast. 3. Third point Threads are not named thus makes it hard to debug. 4. Fourth assume some of those tasks hangs on IO what will happen ? will the JVM ever stops ? 5. Nth issue Assume assume we have 500 llap nodes (yes we have cluster with 500 node running), this will fire 500 threads is that okay ? IMO First cut of this pr should avoid all this Async stuff and start with a simple version OR you need to have a static fixed thread pool with a well defined **life cycle** and clear strategy on how to deal with **blocking IO**, **errors** and **shutdown** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405173) Time Spent: 3.5h (was: 3h 20m) > Ad
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061347#comment-17061347 ] Hive QA commented on HIVE-23004: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 45s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 42s{color} | {color:blue} serde in master has 197 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 40s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s{color} | {color:red} serde: The patch generated 6 new + 289 unchanged - 0 fixed = 295 total (was 289) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 28 new + 849 unchanged - 3 fixed = 877 total (was 852) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 2 line(s) with tabs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s{color} | {color:red} serde generated 3 new + 194 unchanged - 3 fixed = 197 total (was 197) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 52s{color} | {color:red} ql generated 2 new + 1530 unchanged - 1 fixed = 1532 total (was 1531) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 29m 22s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:serde | | | new org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[], DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose internal representation by storing an externally mutable object into BinarySortableDeserializeRead.columnNotNullMarker At BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal representation by storing an externally mutable object into BinarySortableDeserializeRead.columnNotNullMarker At BinarySortableDeserializeRead.java:[line 145] | | | new org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[], DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose internal representation by storing an externally mutable object into BinarySortableDeserializeRead.columnNullMarker At BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal representation by storing an externally mutable object into BinarySortableDeserializeRead.columnNullMarker At BinarySortableDeserializeRead.java:[line 144] | | | new org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[], DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose internal representation by storing an externally mutable object into BinarySortableDeserializeRead.columnSortOrderIsDesc At BinarySortableDeserializeRead.jav
[jira] [Assigned] (HIVE-23040) Checkpointing for repl dump incremental phase
[ https://issues.apache.org/jira/browse/HIVE-23040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi reassigned HIVE-23040: -- > Checkpointing for repl dump incremental phase > - > > Key: HIVE-23040 > URL: https://issues.apache.org/jira/browse/HIVE-23040 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi reassigned HIVE-23039: -- > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061322#comment-17061322 ] Hive QA commented on HIVE-22997: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996939/HIVE-22997.9.patch {color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18109 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partition_cluster] (batchId=190) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query28] (batchId=306) org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testExternalTablesReplLoadBootstrapIncr (batchId=270) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21149/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21149/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21149/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12996939 - PreCommit-HIVE-Build > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[ https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22476: -- Attachment: HIVE-22476.9.patch > Hive datediff function provided inconsistent results when > hive.fetch.task.conversion is set to none > --- > > Key: HIVE-22476 > URL: https://issues.apache.org/jira/browse/HIVE-22476 > Project: Hive > Issue Type: Bug >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch, > HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch, > HIVE-22476.7.patch, HIVE-22476.8.patch, HIVE-22476.8.patch, > HIVE-22476.8.patch, HIVE-22476.9.patch > > > The actual issue stems to the different date parser used by various part of > the engine. > Fetch task uses udfdatediff via {code} > org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the > vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}. > This fix is meant to be not very intrusive and will add more support to the > GenericUDFToDate by enhancing the parser. > For the longer term will be better to use one parser for all the operators. > Thanks [~Rajkumar Singh] for the repro example > {code} > create external table testdatediff(datetimecol string) stored as orc; > insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24'); > select datetimecol from testdatediff where datediff(cast(current_timestamp as > string), datetimecol)<183; > set hive.ferch.task.conversion=none; > select datetimecol from testdatediff where datediff(cast(current_timestamp as > string), datetimecol)<183; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.9.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, > HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: (was: HIVE-23004.9.patch) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, > HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.9.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, > HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.25.patch, HIVE-22990.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.25.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.25.patch, HIVE-22990.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=405114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405114 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 18/Mar/20 01:48 Start Date: 18/Mar/20 01:48 Worklog Time Spent: 10m Work Description: aasha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r394056775 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -398,6 +319,21 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep dbName); } + private void createReplLoadCompleteAckTask() { +if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks()) Review comment: The external table tasks are already added before this method is called. But have still added the check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405114) Time Spent: 5h 40m (was: 5.5h) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=405111&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405111 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 18/Mar/20 01:40 Start Date: 18/Mar/20 01:40 Worklog Time Spent: 10m Work Description: aasha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r394056775 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -398,6 +319,21 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep dbName); } + private void createReplLoadCompleteAckTask() { +if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks()) Review comment: The external table tasks are already added before this method is called This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405111) Time Spent: 5.5h (was: 5h 20m) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.patch > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[ https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061298#comment-17061298 ] Jesus Camacho Rodriguez commented on HIVE-22476: [~bslim], could you rebase this patch / create a PR so we can check it in? Thanks > Hive datediff function provided inconsistent results when > hive.fetch.task.conversion is set to none > --- > > Key: HIVE-22476 > URL: https://issues.apache.org/jira/browse/HIVE-22476 > Project: Hive > Issue Type: Bug >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch, > HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch, > HIVE-22476.7.patch, HIVE-22476.8.patch, HIVE-22476.8.patch, HIVE-22476.8.patch > > > The actual issue stems to the different date parser used by various part of > the engine. > Fetch task uses udfdatediff via {code} > org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the > vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}. > This fix is meant to be not very intrusive and will add more support to the > GenericUDFToDate by enhancing the parser. > For the longer term will be better to use one parser for all the operators. > Thanks [~Rajkumar Singh] for the repro example > {code} > create external table testdatediff(datetimecol string) stored as orc; > insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24'); > select datetimecol from testdatediff where datediff(cast(current_timestamp as > string), datetimecol)<183; > set hive.ferch.task.conversion=none; > select datetimecol from testdatediff where datediff(cast(current_timestamp as > string), datetimecol)<183; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[ https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061298#comment-17061298 ] Jesus Camacho Rodriguez edited comment on HIVE-22476 at 3/18/20, 1:34 AM: -- [~bslim], could you rebase this patch / get a green run so we can check it in? Thanks was (Author: jcamachorodriguez): [~bslim], could you rebase this patch / create a PR so we can check it in? Thanks > Hive datediff function provided inconsistent results when > hive.fetch.task.conversion is set to none > --- > > Key: HIVE-22476 > URL: https://issues.apache.org/jira/browse/HIVE-22476 > Project: Hive > Issue Type: Bug >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch, > HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch, > HIVE-22476.7.patch, HIVE-22476.8.patch, HIVE-22476.8.patch, HIVE-22476.8.patch > > > The actual issue stems to the different date parser used by various part of > the engine. > Fetch task uses udfdatediff via {code} > org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the > vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}. > This fix is meant to be not very intrusive and will add more support to the > GenericUDFToDate by enhancing the parser. > For the longer term will be better to use one parser for all the operators. > Thanks [~Rajkumar Singh] for the repro example > {code} > create external table testdatediff(datetimecol string) stored as orc; > insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24'); > select datetimecol from testdatediff where datediff(cast(current_timestamp as > string), datetimecol)<183; > set hive.ferch.task.conversion=none; > select datetimecol from testdatediff where datediff(cast(current_timestamp as > string), datetimecol)<183; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061294#comment-17061294 ] Hive QA commented on HIVE-22997: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 2s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 48s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} ql: The patch generated 0 new + 76 unchanged - 1 fixed = 76 total (was 77) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 649 unchanged - 1 fixed = 649 total (was 650) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 54s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531 total (was 1531) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 55s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.exec.repl.ReplDumpWork is Serializable; consider declaring a serialVersionUID At ReplDumpWork.java:a serialVersionUID At ReplDumpWork.java:[lines 39-119] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21149/dev-support/hive-personality.sh | | git revision | master / 26cc315 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-21149/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21149/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, >
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=405100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405100 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 18/Mar/20 01:19 Start Date: 18/Mar/20 01:19 Worklog Time Spent: 10m Work Description: aasha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r394056775 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -398,6 +319,21 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep dbName); } + private void createReplLoadCompleteAckTask() { +if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks()) Review comment: The check is already present before the method is being called This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405100) Time Spent: 5h 20m (was: 5h 10m) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.patch > > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.24.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.24.patch, HIVE-22990.patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23035) Scheduled query executor may hang in case TezAMs are launched on-demand
[ https://issues.apache.org/jira/browse/HIVE-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061278#comment-17061278 ] Hive QA commented on HIVE-23035: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996931/HIVE-23035.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 18108 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21148/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21148/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21148/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12996931 - PreCommit-HIVE-Build > Scheduled query executor may hang in case TezAMs are launched on-demand > --- > > Key: HIVE-23035 > URL: https://issues.apache.org/jira/browse/HIVE-23035 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23035.01.patch > > > Right now the schq executor hangs during session initialization - because it > tries to open the tez session while it initializes the SessionState -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23035) Scheduled query executor may hang in case TezAMs are launched on-demand
[ https://issues.apache.org/jira/browse/HIVE-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061260#comment-17061260 ] Hive QA commented on HIVE-23035: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 46s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21148/dev-support/hive-personality.sh | | git revision | master / 26cc315 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21148/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Scheduled query executor may hang in case TezAMs are launched on-demand > --- > > Key: HIVE-23035 > URL: https://issues.apache.org/jira/browse/HIVE-23035 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23035.01.patch > > > Right now the schq executor hangs during session initialization - because it > tries to open the tez session while it initializes the SessionState -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22842) Timestamp/date vectors in Arrow serializer should use correct calendar for value representation
[ https://issues.apache.org/jira/browse/HIVE-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061252#comment-17061252 ] Hive QA commented on HIVE-22842: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996920/HIVE-22842.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 18117 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21147/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21147/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21147/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12996920 - PreCommit-HIVE-Build > Timestamp/date vectors in Arrow serializer should use correct calendar for > value representation > --- > > Key: HIVE-22842 > URL: https://issues.apache.org/jira/browse/HIVE-22842 > Project: Hive > Issue Type: Improvement >Reporter: Jesus Camacho Rodriguez >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22842.01.patch, HIVE-22842.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22842) Timestamp/date vectors in Arrow serializer should use correct calendar for value representation
[ https://issues.apache.org/jira/browse/HIVE-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061221#comment-17061221 ] Hive QA commented on HIVE-22842: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 34s{color} | {color:blue} common in master has 63 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 47s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s{color} | {color:red} ql: The patch generated 1 new + 210 unchanged - 2 fixed = 211 total (was 212) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 33s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21147/dev-support/hive-personality.sh | | git revision | master / 26cc315 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-21147/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-21147/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: common ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21147/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Timestamp/date vectors in Arrow serializer should use correct calendar for > value representation > --- > > Key: HIVE-22842 > URL: https://issues.apache.org/jira/browse/HIVE-22842 > Project: Hive > Issue Type: Improvement >Reporter: Jesus Camacho Rodriguez >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22842.01.patch, HIVE-22842.02
[jira] [Commented] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061210#comment-17061210 ] Hive QA commented on HIVE-22990: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996945/HIVE-22990.23.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18108 tests executed *Failed tests:* {noformat} org.apache.hive.beeline.TestBeeLineWithArgs.testRowsAffected (batchId=286) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21146/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21146/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21146/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12996945 - PreCommit-HIVE-Build > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22842) Timestamp/date vectors in Arrow serializer should use correct calendar for value representation
[ https://issues.apache.org/jira/browse/HIVE-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061203#comment-17061203 ] Jesus Camacho Rodriguez commented on HIVE-22842: +1 (pending additional tests as discussed) > Timestamp/date vectors in Arrow serializer should use correct calendar for > value representation > --- > > Key: HIVE-22842 > URL: https://issues.apache.org/jira/browse/HIVE-22842 > Project: Hive > Issue Type: Improvement >Reporter: Jesus Camacho Rodriguez >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22842.01.patch, HIVE-22842.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[ https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061197#comment-17061197 ] David Mollitor commented on HIVE-22098: --- I linked this case to [HIVE-18983], because when a table is created with {{CREATE TABLE LIKE}}, then the bucket_version information is missing and it later triggers this issue. > Data loss occurs when multiple tables are join with different bucket_version > > > Key: HIVE-22098 > URL: https://issues.apache.org/jira/browse/HIVE-22098 > Project: Hive > Issue Type: Bug > Components: Operators >Affects Versions: 3.1.0, 3.1.2 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Blocker > Labels: data-loss, wrongresults > Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png, > join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc > > > When different bucketVersion of tables do join and no of reducers is greater > than 2, the result is incorrect (*data loss*). > *Scenario 1*: Three tables join. The temporary result data of table_a in the > first table and table_b in the second table joins result is recorded as > tmp_a_b, When it joins with the third table, the bucket_version=2 of the > table created by default after hive-3.0.0, temporary data tmp_a_b initialized > the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In > the init method, the hash algorithm of selecting join column is selected > according to bucketVersion. If bucketVersion = 2 and is not an acid > operation, it will acquired the new algorithm of hash. Otherwise, the old > algorithm of hash is acquired. Because of the inconsistency of the algorithm > of hash, the partition of data allocation caused are different. At stage of > Reducer, Data with the same key can not be paired resulting in data loss. > *Scenario 2*: create two test tables, create table > table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES > ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string) > TBLPROPERTIES ('bucketing_version'='2'); > when use table_bucketversion_1 to join table_bucketversion_2, partial result > data will be loss due to bucketVerison is different. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061185#comment-17061185 ] Hive QA commented on HIVE-22990: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 43s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 46s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} The patch ql passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 806 unchanged - 3 fixed = 806 total (was 809) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 41s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21146/dev-support/hive-personality.sh | | git revision | master / 26cc315 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21146/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.patch > > Time Spent: 5h 10m > R
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404945&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404945 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 20:29 Start Date: 17/Mar/20 20:29 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393762700 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java ## @@ -436,16 +438,29 @@ public void externalTableIncrementalReplication() throws Throwable { } List loadWithClause = externalTableBasePathWithClause(); -replica.load(replicatedDbName, primaryDbName, loadWithClause) +replica.load(replicatedDbName, primaryDbName, withClause) .run("use " + replicatedDbName) .run("show tables like 't1'") .verifyResult("t1") .run("show partitions t1") .verifyResults(new String[] { "country=india", "country=us" }) .run("select place from t1 order by place") -.verifyResults(new String[] { "bangalore", "mumbai", "pune" }) +.verifyResults(new String[] {}) Review comment: Because copy was happening during load. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404945) Time Spent: 4h 50m (was: 4h 40m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404941&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404941 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 20:22 Start Date: 17/Mar/20 20:22 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393848763 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java ## @@ -429,36 +428,19 @@ private Path getCurrentLoadPath() throws IOException, SemanticException { } FileStatus[] statuses = loadPathBase.getFileSystem(conf).listStatus(loadPathBase); if (statuses.length > 0) { - //sort based on last modified. Recent one is at the end - Arrays.sort(statuses, new Comparator() { -public int compare(FileStatus f1, FileStatus f2) { - return Long.compare(f1.getModificationTime(), f2.getModificationTime()); + //sort based on last modified. Recent one is at the beginning + FileStatus latestUpdatedStatus = statuses[0]; + for (FileStatus status : statuses) { +if (status.getModificationTime() > latestUpdatedStatus.getModificationTime()) { + latestUpdatedStatus = status; } - }); - if (replScope.getDbName() != null) { -String currentReplStatusOfTarget -= getReplStatus(replScope.getDbName()); -if (currentReplStatusOfTarget == null) { //bootstrap - return new Path(statuses[0].getPath(), ReplUtils.REPL_HIVE_BASE_DIR); -} else { - DumpMetaData latestDump = new DumpMetaData( - new Path(statuses[statuses.length - 1].getPath(), ReplUtils.REPL_HIVE_BASE_DIR), conf); - if (Long.parseLong(currentReplStatusOfTarget.trim()) >= latestDump.getEventTo()) { -isTargetAlreadyLoaded = true; - } else { -for (FileStatus status : statuses) { - Path hiveLoadPath = new Path(status.getPath(), ReplUtils.REPL_HIVE_BASE_DIR); - DumpMetaData dmd = new DumpMetaData(hiveLoadPath, conf); - if (dmd.isIncrementalDump() - && Long.parseLong(currentReplStatusOfTarget.trim()) < dmd.getEventTo()) { -return hiveLoadPath; - } -} - } + } + Path hiveDumpPath = new Path(latestUpdatedStatus.getPath(), ReplUtils.REPL_HIVE_BASE_DIR); + if (loadPathBase.getFileSystem(conf).exists(hiveDumpPath)) { Review comment: Why do we need this check? Aren't other two checks below good enough? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404941) Time Spent: 5h 10m (was: 5h) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404942&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404942 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 20:22 Start Date: 17/Mar/20 20:22 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393945947 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -398,6 +319,21 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep dbName); } + private void createReplLoadCompleteAckTask() { +if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks()) Review comment: During incremental, when we have no events in dump and only external tables are present, will the event folder still be created? If not the check work.incrementalLoadTasksBuilder().hasMoreWork() might not suffice? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404942) Time Spent: 5h 10m (was: 5h) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[ https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061165#comment-17061165 ] Hive QA commented on HIVE-22888: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996915/HIVE-22888.11.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 18108 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21145/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21145/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21145/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12996915 - PreCommit-HIVE-Build > Rewrite checkLock inner select with JOIN operator > - > > Key: HIVE-22888 > URL: https://issues.apache.org/jira/browse/HIVE-22888 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22888.1.patch, HIVE-22888.10.patch, > HIVE-22888.11.patch, HIVE-22888.2.patch, HIVE-22888.3.patch, > HIVE-22888.4.patch, HIVE-22888.5.patch, HIVE-22888.6.patch, > HIVE-22888.8.patch, HIVE-22888.9.patch, acid-lock-perf-test.pdf > > > - Created extra (db, tbl, part) index on HIVE_LOCKS table; > - Replaced inner select under checkLocks using multiple IN statements with > JOIN operator; > generated query looks like : > {code} > SELECT LS.* FROM ( > SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, > HL_LOCK_TYPE FROM HIVE_LOCKS > WHERE HL_LOCK_EXT_ID < 333) LS > INNER JOIN ( > SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE > HL_LOCK_EXT_ID = 333) LBC > ON LS.HL_DB = LBC.HL_DB > AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE = > LBC.HL_TABLE > AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR > LS.HL_PARTITION = LBC.HL_PARTITION)) > WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID) > AND (LBC.HL_LOCK_TYPE='e' >AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE > IS NOT NULL ) > OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e') > OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e' >AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL)) > LIMIT 1; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[ https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061130#comment-17061130 ] Hive QA commented on HIVE-22888: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 18s{color} | {color:blue} standalone-metastore/metastore-server in master has 186 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s{color} | {color:red} standalone-metastore/metastore-server: The patch generated 12 new + 564 unchanged - 38 fixed = 576 total (was 602) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 19s{color} | {color:red} standalone-metastore/metastore-server generated 1 new + 185 unchanged - 1 fixed = 186 total (was 186) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(Connection, long, long) passes a nonconstant String to an execute or addBatch method on an SQL statement At TxnHandler.java:nonconstant String to an execute or addBatch method on an SQL statement At TxnHandler.java:[line 4435] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21145/dev-support/hive-personality.sh | | git revision | master / 26cc315 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-21145/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-21145/yetus/new-findbugs-standalone-metastore_metastore-server.html | | modules | C: standalone-metastore/metastore-server U: standalone-metastore/metastore-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21145/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Rewrite checkLock inner select with JOIN operator > - > > Key: HIVE-22888 > URL: https://issues.apache.org/jira/browse/HIVE-22888 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22888.1.patch, HIVE-22888.10.patch, > HIVE-22888.11.patch, HIVE-22888.2.patch, HIVE-22888.3.patch, > HIVE-22888.4.patch, HIVE-22888.5.patch, HIVE-22888.6.patch, > HIVE-22888.8.patch, HIVE-22888.9.patch, acid-lock-perf-test.pdf > > > - Created extra (db, tbl, part) index on HIVE_LOCKS table; > - Replaced inner select under checkLocks
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[ https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061109#comment-17061109 ] Hive QA commented on HIVE-22995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996903/HIVE-22995.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 154 failed/errored test(s), 18112 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key2] (batchId=298) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=298) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=298) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=298) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments] (batchId=306) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_database] (batchId=309) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_change_db_location] (batchId=9) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_db_owner] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_owner_actions_db] (batchId=50) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[database_location] (batchId=102) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[database_properties] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[db_ddl_explain] (batchId=101) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_database] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_merge] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[escape_comments] (batchId=88) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_10_external_managed] (batchId=80) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_discovery] (batchId=27) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[unicode_comments] (batchId=45) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join_constants] (batchId=33) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_distinct_gby] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[view_alias] (batchId=97) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_binary_storage_queries] (batchId=113) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key3] (batchId=110) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key] (batchId=110) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_handler_bulk] (batchId=114) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_joins] (batchId=115) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_join] (batchId=112) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_key_range] (batchId=111) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_pushdown] (batchId=111) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries] (batchId=112) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_scan_params] (batchId=115) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_single_sourced_multi_insert] (batchId=114) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_timestamp] (batchId=113) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_timestamp_format] (batchId=114) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbasestats] (batchId=111) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[ppd_key_ranges] (batchId=110) org.apache.hadoop.hive.cli.TestKuduCliDriver.testCliDriver[kudu_queries] (batchId=297) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions] (batchId=205) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter] (batchId=205) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert] (batchId=205) org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic] (batchId=306) org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler] (batchId=306) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[whroot_external1] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[convert_decimal64_to_decimal] (batchId=181) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[csv_llap] (batchId=182) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_4] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalC
[jira] [Updated] (HIVE-23011) Shared work optimizer should check residual predicates when comparing joins
[ https://issues.apache.org/jira/browse/HIVE-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23011: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Shared work optimizer should check residual predicates when comparing joins > --- > > Key: HIVE-23011 > URL: https://issues.apache.org/jira/browse/HIVE-23011 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch, > HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch, > HIVE-23011.patch, HIVE-23011.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[ https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061071#comment-17061071 ] Hive QA commented on HIVE-22995: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 14s{color} | {color:red} /data/hiveptest/logs/PreCommit-HIVE-Build-21144/patches/PreCommit-HIVE-Build-21144.patch does not apply to master. Rebase required? Wrong Branch? See http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21144/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Add support for location for managed tables on database > --- > > Key: HIVE-22995 > URL: https://issues.apache.org/jira/browse/HIVE-22995 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch, Hive Metastore > Support for Tenant-based storage heirarchy.pdf > > > I have attached the initial spec to this jira. > Default location for database would be the external table base directory. > Managed location can be optionally specified. > {code} > CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name > [COMMENT database_comment] > [LOCATION hdfs_path] > [MANAGEDLOCATION hdfs_path] > [WITH DBPROPERTIES (property_name=property_value, ...)]; > ALTER (DATABASE|SCHEMA) database_name SET > MANAGEDLOCATION > hdfs_path; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22940) Make the datasketches functions available as predefined functions
[ https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061067#comment-17061067 ] Hive QA commented on HIVE-22940: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996896/HIVE-22940.04.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 18099 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver (batchId=298) org.apache.hadoop.hive.cli.TestKuduCliDriver.org.apache.hadoop.hive.cli.TestKuduCliDriver (batchId=297) org.apache.hadoop.hive.cli.TestKuduNegativeCliDriver.org.apache.hadoop.hive.cli.TestKuduNegativeCliDriver (batchId=297) org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark (batchId=295) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21143/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21143/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21143/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12996896 - PreCommit-HIVE-Build > Make the datasketches functions available as predefined functions > -- > > Key: HIVE-22940 > URL: https://issues.apache.org/jira/browse/HIVE-22940 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch, > HIVE-22940.03.patch, HIVE-22940.04.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22940) Make the datasketches functions available as predefined functions
[ https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061058#comment-17061058 ] Hive QA commented on HIVE-22940: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 36s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 45s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 43s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} ql: The patch generated 0 new + 82 unchanged - 2 fixed = 82 total (was 84) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 4s{color} | {color:green} root: The patch generated 0 new + 82 unchanged - 2 fixed = 82 total (was 84) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 53s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 63m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc xml compile findbugs checkstyle | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21143/dev-support/hive-personality.sh | | git revision | master / 4daa57c | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql . itests U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21143/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Make the datasketches functions available as predefined functions > -- > > Key: HIVE-22940 > URL: https://issues.apache.org/jira/browse/HIVE-22940 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch, > HIVE-22940.03.patch, HIVE-22940.04.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.23.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.patch > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.23.patch, > HIVE-22990.patch > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404840 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 16:25 Start Date: 17/Mar/20 16:25 Worklog Time Spent: 10m Work Description: aasha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393806962 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -398,6 +319,21 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep dbName); } + private void createReplLoadCompleteAckTask() { +if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks()) +|| (!work.isIncrementalLoad() && !work.hasBootstrapLoadTasks())) { + //All repl load tasks are executed and status is 0, create the task to add the acknowledgement + ReplLoadCompleteAckWork replLoadCompleteAckWork = new ReplLoadCompleteAckWork(work.dumpDirectory); + Task loadCompleteAckWorkTask = TaskFactory.get(replLoadCompleteAckWork, conf); + if (this.childTasks.isEmpty()) { Review comment: No. Its initialised at line 440 for incremental and 86 for bootstrap before this method is called This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404840) Time Spent: 5h (was: 4h 50m) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[ https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23037: -- Status: Patch Available (was: Open) > Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus > - > > Key: HIVE-23037 > URL: https://issues.apache.org/jira/browse/HIVE-23037 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23037.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[ https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23037: -- Attachment: HIVE-23037.1.patch > Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus > - > > Key: HIVE-23037 > URL: https://issues.apache.org/jira/browse/HIVE-23037 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23037.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[ https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23037: - > Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus > - > > Key: HIVE-23037 > URL: https://issues.apache.org/jira/browse/HIVE-23037 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23037.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404838&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404838 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 16:21 Start Date: 17/Mar/20 16:21 Worklog Time Spent: 10m Work Description: aasha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393803832 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java ## @@ -146,7 +146,7 @@ public int execute() { } prepareReturnValues(Arrays.asList(currentDumpPath.toUri().toString(), String.valueOf(lastReplId))); writeDumpCompleteAck(hiveDumpRoot); -deletePreviousDumpMeta(previousDumpMetaPath); +deleteAllPreviousDumpMeta(dumpRoot, currentDumpPath); Review comment: status is a cached copy. So there won't be a problem This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404838) Time Spent: 4h 40m (was: 4.5h) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404839 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 16:22 Start Date: 17/Mar/20 16:22 Worklog Time Spent: 10m Work Description: aasha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393804594 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -286,6 +277,7 @@ a database ( directory ) // Populate the driver context with the scratch dir info from the repl context, so that the temp dirs will be cleaned up later context.getFsScratchDirs().putAll(loadContext.pathInfo.getFsScratchDirs()); + createReplLoadCompleteAckTask(); Review comment: From executeBootStrapLoad the ack task will be added. We need this ack to be added at the end before we return a status. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404839) Time Spent: 4h 50m (was: 4h 40m) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23035) Scheduled query executor may hang in case TezAMs are launched on-demand
[ https://issues.apache.org/jira/browse/HIVE-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061018#comment-17061018 ] László Bodor commented on HIVE-23035: - +1, pending tests > Scheduled query executor may hang in case TezAMs are launched on-demand > --- > > Key: HIVE-23035 > URL: https://issues.apache.org/jira/browse/HIVE-23035 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23035.01.patch > > > Right now the schq executor hangs during session initialization - because it > tries to open the tez session while it initializes the SessionState -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404820&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404820 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 15:56 Start Date: 17/Mar/20 15:56 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393780241 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -286,6 +277,7 @@ a database ( directory ) // Populate the driver context with the scratch dir info from the repl context, so that the temp dirs will be cleaned up later context.getFsScratchDirs().putAll(loadContext.pathInfo.getFsScratchDirs()); + createReplLoadCompleteAckTask(); Review comment: 1) During incremental when there is only bootstrap task and no events, this might not reach here, it will go back right from Line 436: return executeBootStrapLoad(); 2) How does it make sure that the ack task is added after the Repl event ID update task? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404820) Time Spent: 4h 20m (was: 4h 10m) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404821&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404821 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 15:56 Start Date: 17/Mar/20 15:56 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393782909 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java ## @@ -312,7 +312,22 @@ public void testBasic() throws IOException { verifySetup("SELECT * from " + dbName + ".unptned_empty", empty, driver); String replicatedDbName = dbName + "_dupe"; -bootstrapLoadAndVerify(dbName, replicatedDbName); +Tuple bootstrapDump = bootstrapLoadAndVerify(dbName, replicatedDbName); + +FileSystem fs = new Path(bootstrapDump.dumpLocation).getFileSystem(hconf); +Path dumpPath = new Path(bootstrapDump.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR); +boolean dumpAckFound = false; +boolean loadAckFound = false; +for (FileStatus status : fs.listStatus(dumpPath)) { + if (status.getPath().getName().equalsIgnoreCase(ReplUtils.DUMP_ACKNOWLEDGEMENT)) { +dumpAckFound = true; + } + if (status.getPath().getName().equalsIgnoreCase(ReplUtils.LOAD_ACKNOWLEDGEMENT)) { +loadAckFound = true; + } +} +assertTrue(dumpAckFound); +assertTrue(loadAckFound); Review comment: Why not just use: assertTrue(fs.exists(new Path(dumpPath, ReplUtils.DUMP_ACKNOWLEDGEMENT))); assertTrue(fs.exists(new Path(dumpPath, ReplUtils.LOAD_ACKNOWLEDGEMENT))); This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404821) Time Spent: 4.5h (was: 4h 20m) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404822 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 15:56 Start Date: 17/Mar/20 15:56 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393783338 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java ## @@ -838,7 +859,26 @@ public void testIncrementalAdds() throws IOException { verifySetup("SELECT a from " + dbName + ".ptned_late WHERE b=2", ptn_data_2, driver); // Perform REPL-DUMP/LOAD -incrementalLoadAndVerify(dbName, replDbName); +Tuple incrementalDump = incrementalLoadAndVerify(dbName, replDbName); +FileSystem fs = new Path(bootstrapDump.dumpLocation).getFileSystem(hconf); +boolean dumpAckFound = false; +boolean loadAckFound = false; +assertFalse(fs.exists(new Path(bootstrapDump.dumpLocation))); +fs = new Path(incrementalDump.dumpLocation).getFileSystem(hconf); +Path dumpPath = new Path(incrementalDump.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR); +dumpAckFound = false; +loadAckFound = false; +for (FileStatus status : fs.listStatus(dumpPath)) { + if (status.getPath().getName().equalsIgnoreCase(ReplUtils.DUMP_ACKNOWLEDGEMENT)) { +dumpAckFound = true; + } + if (status.getPath().getName().equalsIgnoreCase(ReplUtils.LOAD_ACKNOWLEDGEMENT)) { +loadAckFound = true; + } +} + +assertTrue(dumpAckFound); +assertTrue(loadAckFound); Review comment: Why not just use: assertTrue(fs.exists(new Path(dumpPath, ReplUtils.DUMP_ACKNOWLEDGEMENT))); assertTrue(fs.exists(new Path(dumpPath, ReplUtils.LOAD_ACKNOWLEDGEMENT))); This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404822) Time Spent: 4.5h (was: 4h 20m) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404819 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 17/Mar/20 15:56 Start Date: 17/Mar/20 15:56 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949#discussion_r393780762 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java ## @@ -398,6 +319,21 @@ private void dropTablesExcludedInReplScope(ReplScope replScope) throws HiveExcep dbName); } + private void createReplLoadCompleteAckTask() { +if ((work.isIncrementalLoad() && !work.incrementalLoadTasksBuilder().hasMoreWork() && !work.hasBootstrapLoadTasks()) +|| (!work.isIncrementalLoad() && !work.hasBootstrapLoadTasks())) { + //All repl load tasks are executed and status is 0, create the task to add the acknowledgement + ReplLoadCompleteAckWork replLoadCompleteAckWork = new ReplLoadCompleteAckWork(work.dumpDirectory); + Task loadCompleteAckWorkTask = TaskFactory.get(replLoadCompleteAckWork, conf); + if (this.childTasks.isEmpty()) { Review comment: Isn't childTask null sometime? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404819) Time Spent: 4h 10m (was: 4h) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061008#comment-17061008 ] Hive QA commented on HIVE-22997: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996894/HIVE-22997.8.patch {color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18108 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_materialized] (batchId=184) org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testExternalTablesReplLoadBootstrapIncr (batchId=270) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21142/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21142/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21142/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12996894 - PreCommit-HIVE-Build > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[ https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061002#comment-17061002 ] Denys Kuzmenko edited comment on HIVE-22888 at 3/17/20, 3:35 PM: - MySql: v5.7.23, v5.1.46 Postgres: v9.3, Oracle: XE 11g MsSql: 2017-GA {code} (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='e' AND NOT (EX.HL_TABLE IS NULL AND EX.HL_LOCK_TYPE='r' AND REQ.HL_TABLE IS NOT NULL) limit 1) UNION ALL (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='w' AND EX.HL_LOCK_TYPE IN ('w','e') limit 1) UNION ALL (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='r' AND EX.HL_LOCK_TYPE='e' AND NOT (EX.HL_TABLE IS NOT NULL AND REQ.HL_TABLE IS NULL) limit 1) {code} was (Author: dkuzmenko): MySql: v5.7.23, v5.1.46 Postgres: v9.3, Oracle: XE 11g MsSql: 2017-GA {code} (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='e' AND NOT (EX.HL_TABLE IS NULL AND EX.HL_LOCK_TYPE='r' AND REQ.HL_TABLE IS NOT NULL) limit 1) UNION ALL (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='w' AND EX.HL_LOCK_TYPE IN ('w','e') limit 1) UNION ALL (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='r' AND EX.HL_LOCK_TYPE='e' AND NOT (EX.HL_TABLE IS NOT NULL AND REQ.HL_TABLE IS NULL) limit 1) {code} > Rewrite checkLock inner select with JOIN operator > ---
[jira] [Commented] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[ https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061002#comment-17061002 ] Denys Kuzmenko commented on HIVE-22888: --- MySql: v5.7.23, v5.1.46 Postgres: v9.3, Oracle: XE 11g MsSql: 2017-GA {code} (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='e' AND NOT (EX.HL_TABLE IS NULL AND EX.HL_LOCK_TYPE='r' AND REQ.HL_TABLE IS NOT NULL) limit 1) UNION ALL (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='w' AND EX.HL_LOCK_TYPE IN ('w','e') limit 1) UNION ALL (select EX.*, REQ.HL_LOCK_INT_ID AS REQ_LOCK_INT_ID FROM ( SELECT HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID < 78588) EX INNER JOIN ( SELECT HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE HL_LOCK_EXT_ID = 78588) REQ ON EX.HL_DB = REQ.HL_DB AND (EX.HL_TABLE IS NULL OR REQ.HL_TABLE IS NULL OR EX.HL_TABLE = REQ.HL_TABLE AND (EX.HL_PARTITION IS NULL OR REQ.HL_PARTITION IS NULL OR EX.HL_PARTITION = REQ.HL_PARTITION)) WHERE (REQ.HL_TXNID = 0 OR EX.HL_TXNID != REQ.HL_TXNID) AND REQ.HL_LOCK_TYPE='r' AND EX.HL_LOCK_TYPE='e' AND NOT (EX.HL_TABLE IS NOT NULL AND REQ.HL_TABLE IS NULL) limit 1) {code} > Rewrite checkLock inner select with JOIN operator > - > > Key: HIVE-22888 > URL: https://issues.apache.org/jira/browse/HIVE-22888 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22888.1.patch, HIVE-22888.10.patch, > HIVE-22888.11.patch, HIVE-22888.2.patch, HIVE-22888.3.patch, > HIVE-22888.4.patch, HIVE-22888.5.patch, HIVE-22888.6.patch, > HIVE-22888.8.patch, HIVE-22888.9.patch, acid-lock-perf-test.pdf > > > - Created extra (db, tbl, part) index on HIVE_LOCKS table; > - Replaced inner select under checkLocks using multiple IN statements with > JOIN operator; > generated query looks like : > {code} > SELECT LS.* FROM ( > SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, > HL_LOCK_TYPE FROM HIVE_LOCKS > WHERE HL_LOCK_EXT_ID < 333) LS > INNER JOIN ( > SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE > HL_LOCK_EXT_ID = 333) LBC > ON LS.HL_DB = LBC.HL_DB > AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE = > LBC.HL_TABLE > AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR > LS.HL_PARTITION = LBC.HL_PARTITION)) > WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID) > AND (LBC.HL_LOCK_TYPE='e' >AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE > IS NOT NULL ) > OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e') > OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e' >AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL)) > LIMIT 1; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404796&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404796 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 15:31 Start Date: 17/Mar/20 15:31 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393723769 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java ## @@ -918,7 +922,9 @@ public void testRenameTableScenariosWithReplaceExternalTable() throws Throwable String newPolicy = primaryDbName + ".'(in[0-9]+)|(out1500)|(in2)'"; dumpWithClause = Arrays.asList( "'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'", -"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'" +"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'", Review comment: Missed it here, will fix This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404796) Time Spent: 4h 40m (was: 4.5h) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404793&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404793 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 15:28 Start Date: 17/Mar/20 15:28 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393723769 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java ## @@ -918,7 +922,9 @@ public void testRenameTableScenariosWithReplaceExternalTable() throws Throwable String newPolicy = primaryDbName + ".'(in[0-9]+)|(out1500)|(in2)'"; dumpWithClause = Arrays.asList( "'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'", -"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'" +"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'", Review comment: Yes, I was not planning to refactor all the places. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404793) Time Spent: 4.5h (was: 4h 20m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404790 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 15:26 Start Date: 17/Mar/20 15:26 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393762700 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java ## @@ -436,16 +438,29 @@ public void externalTableIncrementalReplication() throws Throwable { } List loadWithClause = externalTableBasePathWithClause(); -replica.load(replicatedDbName, primaryDbName, loadWithClause) +replica.load(replicatedDbName, primaryDbName, withClause) .run("use " + replicatedDbName) .run("show tables like 't1'") .verifyResult("t1") .run("show partitions t1") .verifyResults(new String[] { "country=india", "country=us" }) .run("select place from t1 order by place") -.verifyResults(new String[] { "bangalore", "mumbai", "pune" }) +.verifyResults(new String[] {}) Review comment: Because copy is happening during load. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404790) Time Spent: 4h 20m (was: 4h 10m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.8.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.9.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060976#comment-17060976 ] Hive QA commented on HIVE-22997: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 49s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} ql: The patch generated 0 new + 76 unchanged - 1 fixed = 76 total (was 77) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 649 unchanged - 1 fixed = 649 total (was 650) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 54s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531 total (was 1531) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 42s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.exec.repl.ReplDumpWork is Serializable; consider declaring a serialVersionUID At ReplDumpWork.java:a serialVersionUID At ReplDumpWork.java:[lines 39-119] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21142/dev-support/hive-personality.sh | | git revision | master / 4daa57c | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-21142/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21142/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, >
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404755&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404755 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:34 Start Date: 17/Mar/20 14:34 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393724555 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java ## @@ -102,12 +104,20 @@ public void targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable { put(HiveConf.ConfVars.REPLDIR.varname, primary.repldDir); }}, "test_key123"); +List dumpWithClause = Arrays.asList( +"'hive.repl.add.raw.reserved.namespace'='true'", +"'" + HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname + "'='" ++ replica.externalTableWarehouseRoot + "'", +"'distcp.options.skipcrccheck'=''", Review comment: That was part of existing test, I have used the same set of config in dump time This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404755) Time Spent: 4h 10m (was: 4h) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404754&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404754 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:33 Start Date: 17/Mar/20 14:33 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393723769 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java ## @@ -918,7 +922,9 @@ public void testRenameTableScenariosWithReplaceExternalTable() throws Throwable String newPolicy = primaryDbName + ".'(in[0-9]+)|(out1500)|(in2)'"; dumpWithClause = Arrays.asList( "'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'", -"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'" +"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'", Review comment: Yes, I am not planning to refactor all the places. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404754) Time Spent: 4h (was: 3h 50m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404753&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404753 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:32 Start Date: 17/Mar/20 14:32 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393723340 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java ## @@ -436,16 +438,29 @@ public void externalTableIncrementalReplication() throws Throwable { } List loadWithClause = externalTableBasePathWithClause(); -replica.load(replicatedDbName, primaryDbName, loadWithClause) +replica.load(replicatedDbName, primaryDbName, withClause) .run("use " + replicatedDbName) .run("show tables like 't1'") .verifyResult("t1") .run("show partitions t1") .verifyResults(new String[] { "country=india", "country=us" }) .run("select place from t1 order by place") -.verifyResults(new String[] { "bangalore", "mumbai", "pune" }) +.verifyResults(new String[] {}) Review comment: The copy was happening in repl load, change was prior to that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404753) Time Spent: 3h 50m (was: 3h 40m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404747&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404747 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:29 Start Date: 17/Mar/20 14:29 Worklog Time Spent: 10m Work Description: aasha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393660921 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java ## @@ -713,9 +732,11 @@ public void testExternalTableDataPath() throws Exception { @Test public void testExternalTablesIncReplicationWithConcurrentDropTable() throws Throwable { -List dumpWithClause = Collections.singletonList( -"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'" -); +List dumpWithClause = Arrays.asList( Review comment: The new method is still not used here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404747) Time Spent: 3.5h (was: 3h 20m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404749&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404749 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:29 Start Date: 17/Mar/20 14:29 Worklog Time Spent: 10m Work Description: aasha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393662263 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java ## @@ -918,7 +922,9 @@ public void testRenameTableScenariosWithReplaceExternalTable() throws Throwable String newPolicy = primaryDbName + ".'(in[0-9]+)|(out1500)|(in2)'"; dumpWithClause = Arrays.asList( "'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'", -"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'" +"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'", Review comment: Its still using its own configs. Is this done? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404749) Time Spent: 3h 40m (was: 3.5h) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404748&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404748 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:29 Start Date: 17/Mar/20 14:29 Worklog Time Spent: 10m Work Description: aasha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393660435 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java ## @@ -436,16 +436,29 @@ public void externalTableIncrementalReplication() throws Throwable { } List loadWithClause = externalTableBasePathWithClause(); -replica.load(replicatedDbName, primaryDbName, loadWithClause) +replica.load(replicatedDbName, primaryDbName, withClause) .run("use " + replicatedDbName) .run("show tables like 't1'") .verifyResult("t1") .run("show partitions t1") .verifyResults(new String[] { "country=india", "country=us" }) .run("select place from t1 order by place") -.verifyResults(new String[] { "bangalore", "mumbai", "pune" }) Review comment: How was it getting loaded here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404748) Time Spent: 3h 40m (was: 3.5h) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404746&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404746 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:29 Start Date: 17/Mar/20 14:29 Worklog Time Spent: 10m Work Description: aasha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393658865 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java ## @@ -102,12 +104,20 @@ public void targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable { put(HiveConf.ConfVars.REPLDIR.varname, primary.repldDir); }}, "test_key123"); +List dumpWithClause = Arrays.asList( +"'hive.repl.add.raw.reserved.namespace'='true'", +"'" + HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname + "'='" ++ replica.externalTableWarehouseRoot + "'", +"'distcp.options.skipcrccheck'=''", Review comment: Why do we need these extra configs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404746) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404745&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404745 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:29 Start Date: 17/Mar/20 14:29 Worklog Time Spent: 10m Work Description: aasha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393660304 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java ## @@ -436,16 +438,29 @@ public void externalTableIncrementalReplication() throws Throwable { } List loadWithClause = externalTableBasePathWithClause(); -replica.load(replicatedDbName, primaryDbName, loadWithClause) +replica.load(replicatedDbName, primaryDbName, withClause) .run("use " + replicatedDbName) .run("show tables like 't1'") .verifyResult("t1") .run("show partitions t1") .verifyResults(new String[] { "country=india", "country=us" }) .run("select place from t1 order by place") -.verifyResults(new String[] { "bangalore", "mumbai", "pune" }) +.verifyResults(new String[] {}) Review comment: How was this getting loaded in the older scenario This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404745) Time Spent: 3h 20m (was: 3h 10m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.22.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.20.patch, > HIVE-22990.21.patch, HIVE-22990.22.patch, HIVE-22990.patch > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23033) MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE
[ https://issues.apache.org/jira/browse/HIVE-23033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060946#comment-17060946 ] Hive QA commented on HIVE-23033: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12996859/HIVE-23033.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 18107 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21141/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21141/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21141/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12996859 - PreCommit-HIVE-Build > MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE > --- > > Key: HIVE-23033 > URL: https://issues.apache.org/jira/browse/HIVE-23033 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0, 3.1.1, 3.1.2 >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0, 3.1.3 > > Attachments: HIVE-23033.1.patch > > > * The inital value for this table in the schema scripts was removed in > HIVE-17566: > https://github.com/apache/hive/commit/32b7abac961ca3879d23b074357f211fc7c49131#diff-3d1a4bae0d5d53c8e4ea79951ebf5eceL598 > * This was fixed in a number of scripts in HIVE-18781, but not for mssql: > https://github.com/apache/hive/commit/59483bca262880d3e7ef1b873d3c21176e9294cb#diff-4f43efd5a45cc362cb138287d90dbf82 > * This is as is since then > When using the schematool, the table gets initialized by other means. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23036) Incorrect ORC PPD eval with sub-millisecond timestamps
[ https://issues.apache.org/jira/browse/HIVE-23036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23036: -- Labels: pull-request-available (was: ) > Incorrect ORC PPD eval with sub-millisecond timestamps > -- > > Key: HIVE-23036 > URL: https://issues.apache.org/jira/browse/HIVE-23036 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > > See [ORC-611|https://issues.apache.org/jira/browse/ORC-611] for more details > ORC stores timestamps with: > - nanosecond precision for the data itself > - milliseconds precision for min-max statistics > As both min and max are rounded to the same value, timestamps with ns > precision will not pass the PPD evaluator. > {code:java} > create table tsstat (ts timestamp) stored as orc; > insert into tsstat values ("1970-01-01 00:00:00.0005"); > select * from tsstat where ts = "1970-01-01 00:00:00.0005"; > -- returned 0 rows{code} > ORC PPD evaluation currently happens as part of OrcInputFormat > [https://github.com/apache/hive/blob/7e39a2c13711f9377c9ce1edb4224880421b1ea5/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L2314] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23036) Incorrect ORC PPD eval with sub-millisecond timestamps
[ https://issues.apache.org/jira/browse/HIVE-23036?focusedWorklogId=404739&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404739 ] ASF GitHub Bot logged work on HIVE-23036: - Author: ASF GitHub Bot Created on: 17/Mar/20 14:19 Start Date: 17/Mar/20 14:19 Worklog Time Spent: 10m Work Description: pgaref commented on pull request #956: HIVE-23036 Reproducing ORC Timestamp precision issue with PPD URL: https://github.com/apache/hive/pull/956 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404739) Remaining Estimate: 0h Time Spent: 10m > Incorrect ORC PPD eval with sub-millisecond timestamps > -- > > Key: HIVE-23036 > URL: https://issues.apache.org/jira/browse/HIVE-23036 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > See [ORC-611|https://issues.apache.org/jira/browse/ORC-611] for more details > ORC stores timestamps with: > - nanosecond precision for the data itself > - milliseconds precision for min-max statistics > As both min and max are rounded to the same value, timestamps with ns > precision will not pass the PPD evaluator. > {code:java} > create table tsstat (ts timestamp) stored as orc; > insert into tsstat values ("1970-01-01 00:00:00.0005"); > select * from tsstat where ts = "1970-01-01 00:00:00.0005"; > -- returned 0 rows{code} > ORC PPD evaluation currently happens as part of OrcInputFormat > [https://github.com/apache/hive/blob/7e39a2c13711f9377c9ce1edb4224880421b1ea5/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L2314] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23036) Incorrect ORC PPD eval with sub-millisecond timestamps
[ https://issues.apache.org/jira/browse/HIVE-23036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-23036: - > Incorrect ORC PPD eval with sub-millisecond timestamps > -- > > Key: HIVE-23036 > URL: https://issues.apache.org/jira/browse/HIVE-23036 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > > See [ORC-611|https://issues.apache.org/jira/browse/ORC-611] for more details > ORC stores timestamps with: > - nanosecond precision for the data itself > - milliseconds precision for min-max statistics > As both min and max are rounded to the same value, timestamps with ns > precision will not pass the PPD evaluator. > {code:java} > create table tsstat (ts timestamp) stored as orc; > insert into tsstat values ("1970-01-01 00:00:00.0005"); > select * from tsstat where ts = "1970-01-01 00:00:00.0005"; > -- returned 0 rows{code} > ORC PPD evaluation currently happens as part of OrcInputFormat > [https://github.com/apache/hive/blob/7e39a2c13711f9377c9ce1edb4224880421b1ea5/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L2314] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23035) Scheduled query executor may hang in case TezAMs are launched on-demand
[ https://issues.apache.org/jira/browse/HIVE-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23035: Status: Patch Available (was: Open) > Scheduled query executor may hang in case TezAMs are launched on-demand > --- > > Key: HIVE-23035 > URL: https://issues.apache.org/jira/browse/HIVE-23035 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > Right now the schq executor hangs during session initialization - because it > tries to open the tez session while it initializes the SessionState -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23035) Scheduled query executor may hang in case TezAMs are launched on-demand
[ https://issues.apache.org/jira/browse/HIVE-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23035: Attachment: HIVE-23035.01.patch > Scheduled query executor may hang in case TezAMs are launched on-demand > --- > > Key: HIVE-23035 > URL: https://issues.apache.org/jira/browse/HIVE-23035 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23035.01.patch > > > Right now the schq executor hangs during session initialization - because it > tries to open the tez session while it initializes the SessionState -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23035) Scheduled query executor may hang in case TezAMs are launched on-demand
[ https://issues.apache.org/jira/browse/HIVE-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23035: --- > Scheduled query executor may hang in case TezAMs are launched on-demand > --- > > Key: HIVE-23035 > URL: https://issues.apache.org/jira/browse/HIVE-23035 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > Right now the schq executor hangs during session initialization - because it > tries to open the tez session while it initializes the SessionState -- This message was sent by Atlassian Jira (v8.3.4#803005)