[jira] [Commented] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[
https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048450#comment-17048450
]
Hive QA commented on HIVE-22786:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995034/HIVE-22786.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18094 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=230)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20899/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20899/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20899/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995034 - PreCommit-HIVE-Build
> Vectorization: Agg with distinct can be optimised in HASH mode
> --
>
> Key: HIVE-22786
> URL: https://issues.apache.org/jira/browse/HIVE-22786
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch,
> HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch,
> HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22951) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22951:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks zhangbutao.
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22951
> URL: https://issues.apache.org/jira/browse/HIVE-22951
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22951.01.patch
>
>
> Build hive source code：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[
https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048444#comment-17048444
]
Hive QA commented on HIVE-22786:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 404 unchanged - 0
fixed = 405 total (was 404) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20899/dev-support/hive-personality.sh
|
| git revision | master / 87c88de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20899/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20899/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20899/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Agg with distinct can be optimised in HASH mode
> --
>
> Key: HIVE-22786
> URL: https://issues.apache.org/jira/browse/HIVE-22786
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch,
> HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch,
> HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048433#comment-17048433
]
Hive QA commented on HIVE-22926:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995041/HIVE-22926.11.patch
{color:green}SUCCESS:{color} +1 due to 19 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 18094 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[repl_dump_requires_admin]
(batchId=107)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[repl_load_requires_admin]
(batchId=107)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrapWithConcurrentWrites
(batchId=277)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootStrapDumpOfWarehouse
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalDumpEmptyDumpDirectory
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrapWithConcurrentWrites
(batchId=279)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootStrapDumpOfWarehouse
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalDumpEmptyDumpDirectory
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testExternalTablesBootstrapIncr
(batchId=270)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testReplDumpResultSet (batchId=293)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20898/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20898/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20898/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995041 - PreCommit-HIVE-Build
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048429#comment-17048429
]
Hive QA commented on HIVE-22926:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
31s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 1274
unchanged - 9 fixed = 1275 total (was 1283) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20898/dev-support/hive-personality.sh
|
| git revision | master / 87c88de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20898/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20898/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20898/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22527) Hive on Tez : Job of merging samll files will be submitted into another queue (default queue)
[
https://issues.apache.org/jira/browse/HIVE-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048425#comment-17048425
]
Hive QA commented on HIVE-22527:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995025/HIVE-22527.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18094 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=185)
org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits2.testGenericUDTFOrderBySplitCount1
(batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20897/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20897/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20897/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995025 - PreCommit-HIVE-Build
> Hive on Tez : Job of merging samll files will be submitted into another queue
> (default queue)
> -
>
> Key: HIVE-22527
> URL: https://issues.apache.org/jira/browse/HIVE-22527
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-22527-branch-3.1.0.patch, HIVE-22527.01.patch,
> HIVE-22527.02.patch, explain with merge files.png, file merge job.png, hive
> logs.png
>
>
> Hive on Tez. We enable small file merge configuration with set
> *hive.merge.tezfiles=true*. So , There will be another job launched for
> merging files after sql job. However, the merge file job is submitted into
> another yarn queue, not the queue of current beeline client session. It seems
> that the merging files job start a new tez session with new conf which is
> different the current session conf, leading to the merging file job goes into
> default queue.
>
> Attachment *hive logs.png* shows that current session queue is
> *root.bdoc.production* ( String queueName = session.getQueueName();) incoming
> queue name is *null* ( String confQueueName =
> conf.get(TezConfiguration.TEZ_QUEUE_NAME);). In fact, we log in to the same
> beeline client with *set tez.queue.name=* *root.bdoc.production,* and all
> jobs should be submitted into the same queue including file merge job.
> [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L445]
> [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L446]
>
> Attachment *explain with merge files.png* shows that ** the stage-4 is
> individual merge file job which is submitted into another yarn queue（default
> queue）， not the queue root.bdoc.production.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22527) Hive on Tez : Job of merging samll files will be submitted into another queue (default queue)
[
https://issues.apache.org/jira/browse/HIVE-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048412#comment-17048412
]
Hive QA commented on HIVE-22527:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20897/dev-support/hive-personality.sh
|
| git revision | master / 87c88de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20897/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20897/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive on Tez : Job of merging samll files will be submitted into another queue
> (default queue)
> -
>
> Key: HIVE-22527
> URL: https://issues.apache.org/jira/browse/HIVE-22527
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-22527-branch-3.1.0.patch, HIVE-22527.01.patch,
> HIVE-22527.02.patch, explain with merge files.png, file merge job.png, hive
> logs.png
>
>
> Hive on Tez. We enable small file merge configuration with set
> *hive.merge.tezfiles=true*. So , There will be another job launched for
> merging files after sql job. However, the merge file job is submitted into
> another yarn queue, not the queue of current beeline client session. It seems
> that the merging files job start a new tez session with new conf which is
> different the current session conf, leading to the merging file job goes into
> default queue.
>
> Attachment *hive logs.png* shows that current session queue is
> *root.bdoc.production* ( String queueName = session.getQueueName();) incoming
> queue name is *null* ( String confQueueName =
> conf.get(TezConfiguration.TEZ_QUEUE_NAME);). In fact, we log in to the same
> beeline client with *set tez.queue.name=* *root.bdoc.production,* and all
> jobs should be submitted into the same queue including file merge job.
>
[jira] [Commented] (HIVE-22931) HoS dynamic partitioning fails with blobstore optimizations off
[
https://issues.apache.org/jira/browse/HIVE-22931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048408#comment-17048408
]
Hive QA commented on HIVE-22931:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995024/HIVE-22931.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18095 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20896/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20896/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20896/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995024 - PreCommit-HIVE-Build
> HoS dynamic partitioning fails with blobstore optimizations off
> ---
>
> Key: HIVE-22931
> URL: https://issues.apache.org/jira/browse/HIVE-22931
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22931.0.patch, HIVE-22931.1.patch,
> HIVE-22931.2.patch, HIVE-22931.3.patch
>
>
> Reproduction steps:
> - Create s3a backed table and normal table.
> {code:java}
> CREATE TABLE source (
> a string,
> b int,
> c int);
>
> CREATE TABLE target (
> a string)
> PARTITIONED BY (
> b int,
> c int)
> STORED AS parquet
> LOCATION
> 's3a://somepath';
> {code}
> - Insert values into normal table.
> {code:java}
> INSERT INTO TABLE source VALUES ("a", "1", "1");
> {code}
> - Do an insert overwrite with dynamic partitions:
> {code:java}
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.blobstore.optimizations.enabled=false;
> set hive.execution.engine=spark;
> INSERT OVERWRITE TABLE target partition (b,c)
> SELECT *
> FROM source;{code}
> This fails only with Spark execution engine + blobstorage optimizations being
> turned off with:
> {code}
> 2020-01-16 15:24:56,064 ERROR hive.ql.metadata.Hive:
> [load-dynamic-partitions-5]: Exception when loading partition with parameters
>
> partPath=hdfs://nameservice1/tmp/hive/hive/6bcee075-b637-429e-9bf0-a2658355415e/hive_2020-01-16_15-24-01_156_4299941251929377815-4/-mr-1/.hive-staging_hive_2020-01-16_15-24-01_156_4299941251929377815-4/-ext-10002,
> table=email_click_base, partSpec={b=null, c=null}, replace=true,
> listBucketingEnabled=false, isAcid=false,
> hasFollowingStatsTask=trueorg.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Partition spec is incorrect. {companyid=null,
> eventmonth=null})
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1666)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22931) HoS dynamic partitioning fails with blobstore optimizations off
[
https://issues.apache.org/jira/browse/HIVE-22931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048399#comment-17048399
]
Hive QA commented on HIVE-22931:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20896/dev-support/hive-personality.sh
|
| git revision | master / 87c88de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20896/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20896/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HoS dynamic partitioning fails with blobstore optimizations off
> ---
>
> Key: HIVE-22931
> URL: https://issues.apache.org/jira/browse/HIVE-22931
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22931.0.patch, HIVE-22931.1.patch,
> HIVE-22931.2.patch, HIVE-22931.3.patch
>
>
> Reproduction steps:
> - Create s3a backed table and normal table.
> {code:java}
> CREATE TABLE source (
> a string,
> b int,
> c int);
>
> CREATE TABLE target (
> a string)
> PARTITIONED BY (
> b int,
> c int)
> STORED AS parquet
> LOCATION
> 's3a://somepath';
> {code}
> - Insert values into normal table.
> {code:java}
> INSERT INTO TABLE source VALUES ("a", "1", "1");
> {code}
> - Do an insert overwrite with dynamic partitions:
> {code:java}
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.blobstore.optimizations.enabled=false;
> set hive.execution.engine=spark;
> INSERT OVERWRITE TABLE target partition (b,c)
> SELECT *
> FROM source;{code}
> This fails only with Spark execution engine + blobstorage optimizations being
> turned off with:
> {code}
> 2020-01-16 15:24:56,064 ERROR hive.ql.metadata.Hive:
> [load-dynamic-partitions-5]: Exception when loading partition with parameters
>
>
[jira] [Commented] (HIVE-22951) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048392#comment-17048392
]
Hive QA commented on HIVE-22951:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995023/HIVE-22951.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20895/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20895/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20895/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995023 - PreCommit-HIVE-Build
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22951
> URL: https://issues.apache.org/jira/browse/HIVE-22951
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Attachments: HIVE-22951.01.patch
>
>
> Build hive source code：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22951) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048381#comment-17048381
]
Ashutosh Chauhan commented on HIVE-22951:
-
+1
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22951
> URL: https://issues.apache.org/jira/browse/HIVE-22951
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Attachments: HIVE-22951.01.patch
>
>
> Build hive source code：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22951) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048380#comment-17048380
]
Hive QA commented on HIVE-22951:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20895/dev-support/hive-personality.sh
|
| git revision | master / 87c88de |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20895/yetus/patch-asflicense-problems.txt
|
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20895/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22951
> URL: https://issues.apache.org/jira/browse/HIVE-22951
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Attachments: HIVE-22951.01.patch
>
>
> Build hive source code：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22872) Support multiple executors for scheduled queries
[ https://issues.apache.org/jira/browse/HIVE-22872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048376#comment-17048376 ] Zoltan Haindrich commented on HIVE-22872: - [~jcamachorodriguez] Could you please take a look? > Support multiple executors for scheduled queries > > > Key: HIVE-22872 > URL: https://issues.apache.org/jira/browse/HIVE-22872 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch, > HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch, > HIVE-22872.06.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22948) QueryCache: Treat query cache locations as temporary storage
[
https://issues.apache.org/jira/browse/HIVE-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048372#comment-17048372
]
Hive QA commented on HIVE-22948:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995022/HIVE-22948.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20894/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20894/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20894/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-02-29 18:58:41.203
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20894/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-02-29 18:58:41.206
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 87c88de HIVE-22941 : Empty files are inserted into external
tables after HIVE-21714 - workaround (Laszlo Bodor via Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 87c88de HIVE-22941 : Empty files are inserted into external
tables after HIVE-21714 - workaround (Laszlo Bodor via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-02-29 18:58:42.016
+ rm -rf ../yetus_PreCommit-HIVE-Build-20894
+ mkdir ../yetus_PreCommit-HIVE-Build-20894
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20894
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20894/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc9042831714565477739.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc9042831714565477739.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc7243983023633460124.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/parser/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveLexer.g
Output file
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.11.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22136) Turn on tez.bucket.pruning
[
https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048371#comment-17048371
]
Hive QA commented on HIVE-22136:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982418/HIVE-22136.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20893/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20893/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20893/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982418 - PreCommit-HIVE-Build
> Turn on tez.bucket.pruning
> ---
>
> Key: HIVE-22136
> URL: https://issues.apache.org/jira/browse/HIVE-22136
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch,
> HIVE-22136.3.patch, HIVE-22136.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.10.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[
https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395487=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395487
]
ASF GitHub Bot logged work on HIVE-22865:
-
Author: ASF GitHub Bot
Created on: 29/Feb/20 18:08
Start Date: 29/Feb/20 18:08
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #911: HIVE-22865
Include data in replication staging directory
URL: https://github.com/apache/hive/pull/911#discussion_r386045024
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java
##
@@ -53,9 +53,7 @@ public int execute() {
work.acidPostProcess(db);
TableExport tableExport = new TableExport(exportPaths,
work.getTableSpec(),
work.getReplicationSpec(), db, null, conf, work.getMmContext());
- if (!tableExport.write()) {
Review comment:
Have refactored the code and same behavior is achieved in
TableExport.export() now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395487)
Time Spent: 2h 20m (was: 2h 10m)
> Include data in replication staging directory
> -
>
> Key: HIVE-22865
> URL: https://issues.apache.org/jira/browse/HIVE-22865
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch,
> HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22136) Turn on tez.bucket.pruning
[
https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048363#comment-17048363
]
Hive QA commented on HIVE-22136:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20893/dev-support/hive-personality.sh
|
| git revision | master / 87c88de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20893/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20893/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Turn on tez.bucket.pruning
> ---
>
> Key: HIVE-22136
> URL: https://issues.apache.org/jira/browse/HIVE-22136
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch,
> HIVE-22136.3.patch, HIVE-22136.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22872) Support multiple executors for scheduled queries
[
https://issues.apache.org/jira/browse/HIVE-22872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048358#comment-17048358
]
Hive QA commented on HIVE-22872:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995019/HIVE-22872.06.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18093 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20892/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20892/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20892/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995019 - PreCommit-HIVE-Build
> Support multiple executors for scheduled queries
>
>
> Key: HIVE-22872
> URL: https://issues.apache.org/jira/browse/HIVE-22872
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch,
> HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch,
> HIVE-22872.06.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[ https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22786: Status: Patch Available (was: Open) > Vectorization: Agg with distinct can be optimised in HASH mode > -- > > Key: HIVE-22786 > URL: https://issues.apache.org/jira/browse/HIVE-22786 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch, > HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch, > HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[ https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22786: Status: Open (was: Patch Available) > Vectorization: Agg with distinct can be optimised in HASH mode > -- > > Key: HIVE-22786 > URL: https://issues.apache.org/jira/browse/HIVE-22786 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch, > HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch, > HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[ https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22786: Attachment: HIVE-22786.8.patch > Vectorization: Agg with distinct can be optimised in HASH mode > -- > > Key: HIVE-22786 > URL: https://issues.apache.org/jira/browse/HIVE-22786 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch, > HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch, > HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22872) Support multiple executors for scheduled queries
[
https://issues.apache.org/jira/browse/HIVE-22872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048350#comment-17048350
]
Hive QA commented on HIVE-22872:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
52s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 295 unchanged - 0 fixed = 296 total (was 295) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 3 new + 15 unchanged - 1 fixed
= 18 total (was 16) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 42 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20892/dev-support/hive-personality.sh
|
| git revision | master / 2a7344a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20892/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20892/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20892/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20892/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20892/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-common common
[jira] [Updated] (HIVE-22941) Empty files are inserted into external tables after HIVE-21714 - workaround
[
https://issues.apache.org/jira/browse/HIVE-22941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22941:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Laszlo!
> Empty files are inserted into external tables after HIVE-21714 - workaround
> ---
>
> Key: HIVE-22941
> URL: https://issues.apache.org/jira/browse/HIVE-22941
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22941.01.patch, HIVE-22941.02.patch,
> HIVE-22941.02.patch
>
>
> There were multiple patches targeting an issue when INSERT OVERWRITE was
> ineffective if the input is empty:
> HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before
> overwriting
> HIVE-21714: Insert overwrite on an acid/mm table is ineffective if the input
> is empty
> HIVE-21784: Insert overwrite on an acid (not mm) table is ineffective if the
> input is empty
> From these patches, HIVE-21714 seems to have a bad effect on external tables,
> because of this part:
> https://github.com/apache/hive/commit/9a10bc28bee5250c0f667c94a295706a44ed4d7e#diff-9bea2581a1fba611f2c10904857b8823R1268
> The original issue before HIVE-21714 was that the original files in the table
> survived an insert overwrite, and select(*)>0 was after that. HIVE-21714
> seems to enable writing empty files regardless of execution engine / table
> type, which is not the proper way, as the proper solution would be to
> completely avoid writing empty files for Tez (this is what HIVE-14014 was
> about). I found that changing condition to...
> {code}
> if (!isTez && (isStreaming || this.isInsertOverwrite))
> {code}
> (which could be an easy solution for external tables) breaks some test cases
> (both full ACID and MM) in insert_overwrite.q, which could mean they rely
> somehow on the empty generated file. We need to find a proper solution which
> is applicable for all table types without polluting external tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[ https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395480 ] ASF GitHub Bot logged work on HIVE-22865: - Author: ASF GitHub Bot Created on: 29/Feb/20 16:40 Start Date: 29/Feb/20 16:40 Worklog Time Spent: 10m Work Description: aasha commented on pull request #911: HIVE-22865 Include data in replication staging directory URL: https://github.com/apache/hive/pull/911#discussion_r386039609 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java ## @@ -305,7 +309,7 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive NotificationEvent ev = evIter.next(); lastReplId = ev.getEventId(); Path evRoot = new Path(dumpRoot, String.valueOf(lastReplId)); - dumpEvent(ev, evRoot, cmRoot, hiveDb); + dumpEvent(ev, evRoot, dumpRoot, cmRoot, hiveDb); Review comment: Why is this dumpRoot and not hiveDumpRoot This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 395480) Time Spent: 2h 10m (was: 2h) > Include data in replication staging directory > - > > Key: HIVE-22865 > URL: https://issues.apache.org/jira/browse/HIVE-22865 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch, > HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[
https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395479
]
ASF GitHub Bot logged work on HIVE-22865:
-
Author: ASF GitHub Bot
Created on: 29/Feb/20 16:35
Start Date: 29/Feb/20 16:35
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #911: HIVE-22865
Include data in replication staging directory
URL: https://github.com/apache/hive/pull/911#discussion_r386039239
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java
##
@@ -53,9 +53,7 @@ public int execute() {
work.acidPostProcess(db);
TableExport tableExport = new TableExport(exportPaths,
work.getTableSpec(),
work.getReplicationSpec(), db, null, conf, work.getMmContext());
- if (!tableExport.write()) {
Review comment:
Why is the check removed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395479)
Time Spent: 2h (was: 1h 50m)
> Include data in replication staging directory
> -
>
> Key: HIVE-22865
> URL: https://issues.apache.org/jira/browse/HIVE-22865
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch,
> HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.09.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22941) Empty files are inserted into external tables after HIVE-21714 - workaround
[
https://issues.apache.org/jira/browse/HIVE-22941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048337#comment-17048337
]
Hive QA commented on HIVE-22941:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995020/HIVE-22941.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20891/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20891/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20891/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995020 - PreCommit-HIVE-Build
> Empty files are inserted into external tables after HIVE-21714 - workaround
> ---
>
> Key: HIVE-22941
> URL: https://issues.apache.org/jira/browse/HIVE-22941
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22941.01.patch, HIVE-22941.02.patch,
> HIVE-22941.02.patch
>
>
> There were multiple patches targeting an issue when INSERT OVERWRITE was
> ineffective if the input is empty:
> HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before
> overwriting
> HIVE-21714: Insert overwrite on an acid/mm table is ineffective if the input
> is empty
> HIVE-21784: Insert overwrite on an acid (not mm) table is ineffective if the
> input is empty
> From these patches, HIVE-21714 seems to have a bad effect on external tables,
> because of this part:
> https://github.com/apache/hive/commit/9a10bc28bee5250c0f667c94a295706a44ed4d7e#diff-9bea2581a1fba611f2c10904857b8823R1268
> The original issue before HIVE-21714 was that the original files in the table
> survived an insert overwrite, and select(*)>0 was after that. HIVE-21714
> seems to enable writing empty files regardless of execution engine / table
> type, which is not the proper way, as the proper solution would be to
> completely avoid writing empty files for Tez (this is what HIVE-14014 was
> about). I found that changing condition to...
> {code}
> if (!isTez && (isStreaming || this.isInsertOverwrite))
> {code}
> (which could be an easy solution for external tables) breaks some test cases
> (both full ACID and MM) in insert_overwrite.q, which could mean they rely
> somehow on the empty generated file. We need to find a proper solution which
> is applicable for all table types without polluting external tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22952) Use LinkedHashMap in TestStandardObjectInspectors.java
[
https://issues.apache.org/jira/browse/HIVE-22952?focusedWorklogId=395462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395462
]
ASF GitHub Bot logged work on HIVE-22952:
-
Author: ASF GitHub Bot
Created on: 29/Feb/20 15:11
Start Date: 29/Feb/20 15:11
Worklog Time Spent: 10m
Work Description: cpugputpu commented on pull request #929: HIVE-22952
Use LinkedHashMap in TestStandardObjectInspectors.java
URL: https://github.com/apache/hive/pull/929
The background information is presented here:
https://issues.apache.org/jira/browse/HIVE-22952
This PR aims to fix the issue without hurting any performance at all. It
uses LinkedHashMap instead of HashMap in the test so that the test is no more
flaky.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395462)
Remaining Estimate: 0h
Time Spent: 10m
> Use LinkedHashMap in TestStandardObjectInspectors.java
> --
>
> Key: HIVE-22952
> URL: https://issues.apache.org/jira/browse/HIVE-22952
> Project: Hive
> Issue Type: Bug
> Components: Test, Tests
>Reporter: cpugputpu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The test in
> _org.apache.hadoop.hive.serde2.objectinspector.TestStandardObjectInspectors#testStandardUnionObjectInspector_
> can fail due to a different iteration order of HashMap. The failure is
> presented as follows.
> org.junit.ComparisonFailure:
> expected:<\{4:{6:"six",7:"seven",8:"eight"}}>
> but was:<\{4:{6:"six",8:"eight",7:"seven"}}>
> The reason is that the assertion
> _assertEquals("\{4:{6:\"six\",7:\"seven\",8:\"eight\"}}",
> SerDeUtils.getJSONString(union, uoi1));_ compares a hard-coded string against
> the string representation of a JSON object, which is implemented by a
> HashMap. To get the string, the HashMap is iterated here at
> _serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java:343_
> _for (Object entry : omap.entrySet())_
> The specification about HashMap says that "this class makes no guarantees as
> to the order of the map; in particular, it does not guarantee that the order
> will remain constant over time". The documentation is here for your
> reference: [https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html]
>
> The fix is to use LinkedHashMap instead of HashMap. In this way, the
> non-deterministic behaviour is eliminated and the test will become more
> stable.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22952) Use LinkedHashMap in TestStandardObjectInspectors.java
[
https://issues.apache.org/jira/browse/HIVE-22952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-22952:
--
Labels: pull-request-available (was: )
> Use LinkedHashMap in TestStandardObjectInspectors.java
> --
>
> Key: HIVE-22952
> URL: https://issues.apache.org/jira/browse/HIVE-22952
> Project: Hive
> Issue Type: Bug
> Components: Test, Tests
>Reporter: cpugputpu
>Priority: Minor
> Labels: pull-request-available
>
> The test in
> _org.apache.hadoop.hive.serde2.objectinspector.TestStandardObjectInspectors#testStandardUnionObjectInspector_
> can fail due to a different iteration order of HashMap. The failure is
> presented as follows.
> org.junit.ComparisonFailure:
> expected:<\{4:{6:"six",7:"seven",8:"eight"}}>
> but was:<\{4:{6:"six",8:"eight",7:"seven"}}>
> The reason is that the assertion
> _assertEquals("\{4:{6:\"six\",7:\"seven\",8:\"eight\"}}",
> SerDeUtils.getJSONString(union, uoi1));_ compares a hard-coded string against
> the string representation of a JSON object, which is implemented by a
> HashMap. To get the string, the HashMap is iterated here at
> _serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java:343_
> _for (Object entry : omap.entrySet())_
> The specification about HashMap says that "this class makes no guarantees as
> to the order of the map; in particular, it does not guarantee that the order
> will remain constant over time". The documentation is here for your
> reference: [https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html]
>
> The fix is to use LinkedHashMap instead of HashMap. In this way, the
> non-deterministic behaviour is eliminated and the test will become more
> stable.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22941) Empty files are inserted into external tables after HIVE-21714 - workaround
[
https://issues.apache.org/jira/browse/HIVE-22941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048323#comment-17048323
]
Hive QA commented on HIVE-22941:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20891/dev-support/hive-personality.sh
|
| git revision | master / 2a7344a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20891/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20891/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Empty files are inserted into external tables after HIVE-21714 - workaround
> ---
>
> Key: HIVE-22941
> URL: https://issues.apache.org/jira/browse/HIVE-22941
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22941.01.patch, HIVE-22941.02.patch,
> HIVE-22941.02.patch
>
>
> There were multiple patches targeting an issue when INSERT OVERWRITE was
> ineffective if the input is empty:
> HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before
> overwriting
> HIVE-21714: Insert overwrite on an acid/mm table is ineffective if the input
> is empty
> HIVE-21784: Insert overwrite on an acid (not mm) table is ineffective if the
> input is empty
> From these patches, HIVE-21714 seems to have a bad effect on external tables,
> because of this part:
> https://github.com/apache/hive/commit/9a10bc28bee5250c0f667c94a295706a44ed4d7e#diff-9bea2581a1fba611f2c10904857b8823R1268
> The original issue before HIVE-21714 was that the original files in the table
> survived an insert overwrite, and select(*)>0 was after that. HIVE-21714
> seems to
[jira] [Commented] (HIVE-21439) Provide an option to reduce lookup overhead for bucketed tables
[
https://issues.apache.org/jira/browse/HIVE-21439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048320#comment-17048320
]
Hive QA commented on HIVE-21439:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995016/HIVE-21439.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18093 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20890/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20890/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20890/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995016 - PreCommit-HIVE-Build
> Provide an option to reduce lookup overhead for bucketed tables
> ---
>
> Key: HIVE-21439
> URL: https://issues.apache.org/jira/browse/HIVE-21439
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-21439.1.patch, HIVE-21439.2.patch,
> HIVE-21439.3.patch
>
>
> If a table is bucketed, `OpTraitsRulesProcFactory::TableScanRule` ends up
> verifying if the partitions have got the same number of files as the number
> of buckets in table.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java#L185
> In large tables, this turns out to be very time consuming (100s of seconds)
> operation. It would be good to have an option to by pass this depending on
> need basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21439) Provide an option to reduce lookup overhead for bucketed tables
[
https://issues.apache.org/jira/browse/HIVE-21439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048302#comment-17048302
]
Hive QA commented on HIVE-21439:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 2 new + 12 unchanged - 0 fixed
= 14 total (was 12) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20890/dev-support/hive-personality.sh
|
| git revision | master / 2a7344a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20890/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20890/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20890/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Provide an option to reduce lookup overhead for bucketed tables
> ---
>
> Key: HIVE-21439
> URL: https://issues.apache.org/jira/browse/HIVE-21439
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-21439.1.patch, HIVE-21439.2.patch,
> HIVE-21439.3.patch
>
>
> If a table is bucketed, `OpTraitsRulesProcFactory::TableScanRule` ends up
> verifying if the partitions have got the same number of files as the number
> of buckets in table.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java#L185
> In large tables, this turns out to be very time consuming (100s of seconds)
> operation. It would be good to have an option to by pass this depending on
> need basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.08.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.07.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048298#comment-17048298
]
Hive QA commented on HIVE-21971:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995015/HIVE-21971.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18095 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[topnkey_grouping_sets]
(batchId=1)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20889/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20889/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20889/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995015 - PreCommit-HIVE-Build
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048286#comment-17048286
]
Hive QA commented on HIVE-21971:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 72 unchanged - 0 fixed
= 73 total (was 72) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20889/dev-support/hive-personality.sh
|
| git revision | master / 2a7344a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20889/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20889/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20889/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
>
[jira] [Commented] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048273#comment-17048273
]
Hive QA commented on HIVE-21778:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12994997/HIVE-21778.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18093 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20887/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20887/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20887/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12994997 - PreCommit-HIVE-Build
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0, 2.3.5
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, test_null.q, test_null.q.out
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-22950) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao resolved HIVE-22950.
---
Resolution: Duplicate
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22950
> URL: https://issues.apache.org/jira/browse/HIVE-22950
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>
> Build hive code source：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22950) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao reassigned HIVE-22950:
-
Assignee: zhangbutao
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22950
> URL: https://issues.apache.org/jira/browse/HIVE-22950
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>
> Build hive code source：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048264#comment-17048264
]
Hive QA commented on HIVE-21778:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20887/dev-support/hive-personality.sh
|
| git revision | master / 2a7344a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20887/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20887/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0, 2.3.5
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, test_null.q, test_null.q.out
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set
[jira] [Commented] (HIVE-22527) Hive on Tez : Job of merging samll files will be submitted into another queue (default queue)
[ https://issues.apache.org/jira/browse/HIVE-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048263#comment-17048263 ] zhangbutao commented on HIVE-22527: --- Thanks for review [~ngangam] . A new patch is uploaded to fix whitespace and findbugs. What else do I need to do to mege the patch into master ? Thanks. > Hive on Tez : Job of merging samll files will be submitted into another queue > (default queue) > - > > Key: HIVE-22527 > URL: https://issues.apache.org/jira/browse/HIVE-22527 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HIVE-22527-branch-3.1.0.patch, HIVE-22527.01.patch, > HIVE-22527.02.patch, explain with merge files.png, file merge job.png, hive > logs.png > > > Hive on Tez. We enable small file merge configuration with set > *hive.merge.tezfiles=true*. So , There will be another job launched for > merging files after sql job. However, the merge file job is submitted into > another yarn queue, not the queue of current beeline client session. It seems > that the merging files job start a new tez session with new conf which is > different the current session conf, leading to the merging file job goes into > default queue. > > Attachment *hive logs.png* shows that current session queue is > *root.bdoc.production* ( String queueName = session.getQueueName();) incoming > queue name is *null* ( String confQueueName = > conf.get(TezConfiguration.TEZ_QUEUE_NAME);). In fact, we log in to the same > beeline client with *set tez.queue.name=* *root.bdoc.production,* and all > jobs should be submitted into the same queue including file merge job. > [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L445] > [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L446] > > Attachment *explain with merge files.png* shows that ** the stage-4 is > individual merge file job which is submitted into another yarn queue（default > queue）， not the queue root.bdoc.production. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22527) Hive on Tez : Job of merging samll files will be submitted into another queue (default queue)
[ https://issues.apache.org/jira/browse/HIVE-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangbutao updated HIVE-22527: -- Attachment: HIVE-22527.02.patch > Hive on Tez : Job of merging samll files will be submitted into another queue > (default queue) > - > > Key: HIVE-22527 > URL: https://issues.apache.org/jira/browse/HIVE-22527 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HIVE-22527-branch-3.1.0.patch, HIVE-22527.01.patch, > HIVE-22527.02.patch, explain with merge files.png, file merge job.png, hive > logs.png > > > Hive on Tez. We enable small file merge configuration with set > *hive.merge.tezfiles=true*. So , There will be another job launched for > merging files after sql job. However, the merge file job is submitted into > another yarn queue, not the queue of current beeline client session. It seems > that the merging files job start a new tez session with new conf which is > different the current session conf, leading to the merging file job goes into > default queue. > > Attachment *hive logs.png* shows that current session queue is > *root.bdoc.production* ( String queueName = session.getQueueName();) incoming > queue name is *null* ( String confQueueName = > conf.get(TezConfiguration.TEZ_QUEUE_NAME);). In fact, we log in to the same > beeline client with *set tez.queue.name=* *root.bdoc.production,* and all > jobs should be submitted into the same queue including file merge job. > [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L445] > [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L446] > > Attachment *explain with merge files.png* shows that ** the stage-4 is > individual merge file job which is submitted into another yarn queue（default > queue）， not the queue root.bdoc.production. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048260#comment-17048260
]
Hive QA commented on HIVE-21304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12994994/HIVE-21304.18.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 18093 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[reduce_deduplicate]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket2]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=190)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testVersioning
(batchId=343)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20886/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20886/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20886/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12994994 - PreCommit-HIVE-Build
> Show Bucketing version for ReduceSinkOp in explain extended plan
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch,
> HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch,
> HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch,
> HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch,
> HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch,
> HIVE-21304.18.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>
> cc [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22951) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-22951:
--
Description:
Build hive source code：
{code:java}
mvn clean install -DskipTests -Pdist{code}
Some xml file or jar would be asked from invalid repository. Some logs of maven
as follow:
{code:java}
Downloading from druid-apache-rc-testing:
https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
We shoule fix the invalid repo url in pom.
was:
Build hive code source：
{code:java}
mvn clean install -DskipTests -Pdist{code}
Some xml file or jar would be asked from invalid repository. Some logs of maven
as follow:
{code:java}
Downloading from druid-apache-rc-testing:
https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
We shoule fix the invalid repo url in pom.
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22951
> URL: https://issues.apache.org/jira/browse/HIVE-22951
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Attachments: HIVE-22951.01.patch
>
>
> Build hive source code：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22931) HoS dynamic partitioning fails with blobstore optimizations off
[
https://issues.apache.org/jira/browse/HIVE-22931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-22931:
--
Attachment: HIVE-22931.3.patch
> HoS dynamic partitioning fails with blobstore optimizations off
> ---
>
> Key: HIVE-22931
> URL: https://issues.apache.org/jira/browse/HIVE-22931
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22931.0.patch, HIVE-22931.1.patch,
> HIVE-22931.2.patch, HIVE-22931.3.patch
>
>
> Reproduction steps:
> - Create s3a backed table and normal table.
> {code:java}
> CREATE TABLE source (
> a string,
> b int,
> c int);
>
> CREATE TABLE target (
> a string)
> PARTITIONED BY (
> b int,
> c int)
> STORED AS parquet
> LOCATION
> 's3a://somepath';
> {code}
> - Insert values into normal table.
> {code:java}
> INSERT INTO TABLE source VALUES ("a", "1", "1");
> {code}
> - Do an insert overwrite with dynamic partitions:
> {code:java}
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.blobstore.optimizations.enabled=false;
> set hive.execution.engine=spark;
> INSERT OVERWRITE TABLE target partition (b,c)
> SELECT *
> FROM source;{code}
> This fails only with Spark execution engine + blobstorage optimizations being
> turned off with:
> {code}
> 2020-01-16 15:24:56,064 ERROR hive.ql.metadata.Hive:
> [load-dynamic-partitions-5]: Exception when loading partition with parameters
>
> partPath=hdfs://nameservice1/tmp/hive/hive/6bcee075-b637-429e-9bf0-a2658355415e/hive_2020-01-16_15-24-01_156_4299941251929377815-4/-mr-1/.hive-staging_hive_2020-01-16_15-24-01_156_4299941251929377815-4/-ext-10002,
> table=email_click_base, partSpec={b=null, c=null}, replace=true,
> listBucketingEnabled=false, isAcid=false,
> hasFollowingStatsTask=trueorg.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Partition spec is incorrect. {companyid=null,
> eventmonth=null})
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1666)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22951) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-22951:
--
Attachment: HIVE-22951.01.patch
Status: Patch Available (was: Open)
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22951
> URL: https://issues.apache.org/jira/browse/HIVE-22951
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Attachments: HIVE-22951.01.patch
>
>
> Build hive code source：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22951) Fix invalid repo url in pom
[
https://issues.apache.org/jira/browse/HIVE-22951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao reassigned HIVE-22951:
-
Assignee: zhangbutao
> Fix invalid repo url in pom
> ---
>
> Key: HIVE-22951
> URL: https://issues.apache.org/jira/browse/HIVE-22951
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>
> Build hive code source：
> {code:java}
> mvn clean install -DskipTests -Pdist{code}
> Some xml file or jar would be asked from invalid repository. Some logs of
> maven as follow:
> {code:java}
> Downloading from druid-apache-rc-testing:
> https://repository.apache.org/content/repositories/orgapachedruid-1001/org/apache/orc/orc-core/1.5.9/orc-core-1.5.9.jar{code}
> We shoule fix the invalid repo url in pom.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048254#comment-17048254
]
Hive QA commented on HIVE-21304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 2 new + 891 unchanged - 6
fixed = 893 total (was 897) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20886/dev-support/hive-personality.sh
|
| git revision | master / 2a7344a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20886/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20886/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20886/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Show Bucketing version for ReduceSinkOp in explain extended plan
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch,
> HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch,
> HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch,
> HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch,
> HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch,
> HIVE-21304.18.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>
> cc [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22948) QueryCache: Treat query cache locations as temporary storage
[
https://issues.apache.org/jira/browse/HIVE-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal Vijayaraghavan updated HIVE-22948:
Status: Patch Available (was: Open)
> QueryCache: Treat query cache locations as temporary storage
>
>
> Key: HIVE-22948
> URL: https://issues.apache.org/jira/browse/HIVE-22948
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Attachments: HIVE-22948.1.patch
>
>
> The WriteEntity with a query cache query is considered for user authorization
> without having direct access for users.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/command/CommandAuthorizerV2.java#L111
> {code}
> if (privObject instanceof WriteEntity &&
> ((WriteEntity)privObject).isTempURI()) {
> // do not authorize temporary uris
> continue;
> }
> {code}
> is not satisfied by the queries qualifying for the query cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22948) QueryCache: Treat query cache locations as temporary storage
[
https://issues.apache.org/jira/browse/HIVE-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal Vijayaraghavan updated HIVE-22948:
Attachment: HIVE-22948.1.patch
> QueryCache: Treat query cache locations as temporary storage
>
>
> Key: HIVE-22948
> URL: https://issues.apache.org/jira/browse/HIVE-22948
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Attachments: HIVE-22948.1.patch
>
>
> The WriteEntity with a query cache query is considered for user authorization
> without having direct access for users.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/command/CommandAuthorizerV2.java#L111
> {code}
> if (privObject instanceof WriteEntity &&
> ((WriteEntity)privObject).isTempURI()) {
> // do not authorize temporary uris
> continue;
> }
> {code}
> is not satisfied by the queries qualifying for the query cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22941) Empty files are inserted into external tables after HIVE-21714 - workaround
[
https://issues.apache.org/jira/browse/HIVE-22941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048243#comment-17048243
]
László Bodor commented on HIVE-22941:
-
thanks for the review [~ashutoshc], HIVE-22949 has been created for following up
waiting for a green run
> Empty files are inserted into external tables after HIVE-21714 - workaround
> ---
>
> Key: HIVE-22941
> URL: https://issues.apache.org/jira/browse/HIVE-22941
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22941.01.patch, HIVE-22941.02.patch,
> HIVE-22941.02.patch
>
>
> There were multiple patches targeting an issue when INSERT OVERWRITE was
> ineffective if the input is empty:
> HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before
> overwriting
> HIVE-21714: Insert overwrite on an acid/mm table is ineffective if the input
> is empty
> HIVE-21784: Insert overwrite on an acid (not mm) table is ineffective if the
> input is empty
> From these patches, HIVE-21714 seems to have a bad effect on external tables,
> because of this part:
> https://github.com/apache/hive/commit/9a10bc28bee5250c0f667c94a295706a44ed4d7e#diff-9bea2581a1fba611f2c10904857b8823R1268
> The original issue before HIVE-21714 was that the original files in the table
> survived an insert overwrite, and select(*)>0 was after that. HIVE-21714
> seems to enable writing empty files regardless of execution engine / table
> type, which is not the proper way, as the proper solution would be to
> completely avoid writing empty files for Tez (this is what HIVE-14014 was
> about). I found that changing condition to...
> {code}
> if (!isTez && (isStreaming || this.isInsertOverwrite))
> {code}
> (which could be an easy solution for external tables) breaks some test cases
> (both full ACID and MM) in insert_overwrite.q, which could mean they rely
> somehow on the empty generated file. We need to find a proper solution which
> is applicable for all table types without polluting external tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22949) Handle empty insert overwrites without inserting an empty file in case of acid/mm tables
[ https://issues.apache.org/jira/browse/HIVE-22949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22949: Description: HIVE-22941 was a quick workaround for empty files in case of external tables. An optimal solution would to completely prevent writing empty files just for having the table contents cleared on an empty INSERT OVERWRITE. There are other tickets about a similar topic, for example, if we need empty files at all: HIVE-22918, HIVE-22938 > Handle empty insert overwrites without inserting an empty file in case of > acid/mm tables > > > Key: HIVE-22949 > URL: https://issues.apache.org/jira/browse/HIVE-22949 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > > HIVE-22941 was a quick workaround for empty files in case of external tables. > An optimal solution would to completely prevent writing empty files just for > having the table contents cleared on an empty INSERT OVERWRITE. > There are other tickets about a similar topic, for example, if we need empty > files at all: HIVE-22918, HIVE-22938 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[
https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048242#comment-17048242
]
Hive QA commented on HIVE-22786:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12994992/HIVE-22786.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 18090 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=306)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=306)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20885/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20885/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20885/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12994992 - PreCommit-HIVE-Build
> Vectorization: Agg with distinct can be optimised in HASH mode
> --
>
> Key: HIVE-22786
> URL: https://issues.apache.org/jira/browse/HIVE-22786
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch,
> HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch,
> HIVE-22786.6.patch, HIVE-22786.7.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22941) Empty files are inserted into external tables after HIVE-21714 - workaround
[
https://issues.apache.org/jira/browse/HIVE-22941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22941:
Summary: Empty files are inserted into external tables after HIVE-21714 -
workaround (was: Empty files are inserted into external tables after
HIVE-21714)
> Empty files are inserted into external tables after HIVE-21714 - workaround
> ---
>
> Key: HIVE-22941
> URL: https://issues.apache.org/jira/browse/HIVE-22941
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22941.01.patch, HIVE-22941.02.patch,
> HIVE-22941.02.patch
>
>
> There were multiple patches targeting an issue when INSERT OVERWRITE was
> ineffective if the input is empty:
> HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before
> overwriting
> HIVE-21714: Insert overwrite on an acid/mm table is ineffective if the input
> is empty
> HIVE-21784: Insert overwrite on an acid (not mm) table is ineffective if the
> input is empty
> From these patches, HIVE-21714 seems to have a bad effect on external tables,
> because of this part:
> https://github.com/apache/hive/commit/9a10bc28bee5250c0f667c94a295706a44ed4d7e#diff-9bea2581a1fba611f2c10904857b8823R1268
> The original issue before HIVE-21714 was that the original files in the table
> survived an insert overwrite, and select(*)>0 was after that. HIVE-21714
> seems to enable writing empty files regardless of execution engine / table
> type, which is not the proper way, as the proper solution would be to
> completely avoid writing empty files for Tez (this is what HIVE-14014 was
> about). I found that changing condition to...
> {code}
> if (!isTez && (isStreaming || this.isInsertOverwrite))
> {code}
> (which could be an easy solution for external tables) breaks some test cases
> (both full ACID and MM) in insert_overwrite.q, which could mean they rely
> somehow on the empty generated file. We need to find a proper solution which
> is applicable for all table types without polluting external tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22136: Status: Patch Available (was: Open) > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22136: Status: Open (was: Patch Available) > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22827) Update Flatbuffer version
[ https://issues.apache.org/jira/browse/HIVE-22827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22827: Resolution: Fixed Status: Resolved (was: Patch Available) > Update Flatbuffer version > - > > Key: HIVE-22827 > URL: https://issues.apache.org/jira/browse/HIVE-22827 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22827.99.patch, HIVE-22827.991.patch, > HIVE-22827.patch > > > Hive currently uses Flatbuffer 1.2.0. Other Apache projects use a more > up-to-date version, e.g. 1.6.0.1. Upgrade to that version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
