[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Attachment: HIVE-21868.08.patch Status: Patch Available (was: Open) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch, > HIVE-21868.08.patch, HIVE-21868.08.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Status: Open (was: Patch Available) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch, > HIVE-21868.08.patch, HIVE-21868.08.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21957) Create temporary table like should omit transactional properties
[
https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878982#comment-16878982
]
Hive QA commented on HIVE-21957:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973692/HIVE-21957.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16332 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17864/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17864/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17864/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973692 - PreCommit-HIVE-Build
> Create temporary table like should omit transactional properties
>
>
> Key: HIVE-21957
> URL: https://issues.apache.org/jira/browse/HIVE-21957
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21957.01.patch
>
>
> In case of create temporary table like queries, where the source table is
> transactional, the transactional properties should not be copied over to the
> new table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21880:
--
Attachment: HIVE-21880.06.patch
Status: Patch Available (was: In Progress)
The build isn't failing locally for me. So, re-attaching 05 renamed as 06.
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21880.01.patch, HIVE-21880.02.patch,
> HIVE-21880.03.patch, HIVE-21880.04.patch, HIVE-21880.05.patch,
> HIVE-21880.06.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Need tp enable
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
> which is disabled as it is flaky and randomly failing with below error.
> {code}
> Error Message
> Notification events are missing in the meta store.
> Stacktrace
> java.lang.IllegalStateException: Notification events are missing in the meta
> store.
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:189)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:231)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:227)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:282)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:265)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:289)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites(TestReplicationScenariosAcidTablesBootstrap.java:328)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Updated] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21637: -- Attachment: HIVE-21637.16.patch > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.2.patch, HIVE-21637.3.patch, HIVE-21637.4.patch, > HIVE-21637.5.patch, HIVE-21637.6.patch, HIVE-21637.7.patch, > HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21880:
--
Status: In Progress (was: Patch Available)
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21880.01.patch, HIVE-21880.02.patch,
> HIVE-21880.03.patch, HIVE-21880.04.patch, HIVE-21880.05.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Need tp enable
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
> which is disabled as it is flaky and randomly failing with below error.
> {code}
> Error Message
> Notification events are missing in the meta store.
> Stacktrace
> java.lang.IllegalStateException: Notification events are missing in the meta
> store.
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:189)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:231)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:227)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:282)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:265)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:289)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites(TestReplicationScenariosAcidTablesBootstrap.java:328)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
[jira] [Commented] (HIVE-21957) Create temporary table like should omit transactional properties
[
https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878974#comment-16878974
]
Hive QA commented on HIVE-21957:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17864/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17864/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create temporary table like should omit transactional properties
>
>
> Key: HIVE-21957
> URL: https://issues.apache.org/jira/browse/HIVE-21957
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21957.01.patch
>
>
> In case of create temporary table like queries, where the source table is
> transactional, the transactional properties should not be copied over to the
> new table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Status: Open (was: Patch Available) > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Status: Patch Available (was: Open) > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Attachment: HIVE-21938.3.patch > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878961#comment-16878961
]
Hive QA commented on HIVE-21880:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973672/HIVE-21880.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17862/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17862/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17862/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-07-05 04:22:54.408
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-17862/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-07-05 04:22:54.411
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 3223bda HIVE-21949: Revert HIVE-21232 LLAP: Add a cache-miss
friendly split affinity provider (Antal Sinkovits reviewed by Gopal V, Oliver
Draese and Peter Vary)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 3223bda HIVE-21949: Revert HIVE-21232 LLAP: Add a cache-miss
friendly split affinity provider (Antal Sinkovits reviewed by Gopal V, Oliver
Draese and Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-07-05 04:22:55.052
+ rm -rf ../yetus_PreCommit-HIVE-Build-17862
+ mkdir ../yetus_PreCommit-HIVE-Build-17862
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-17862
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17862/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTablesBootstrap.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc4480269874805435176.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc4480269874805435176.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims-0.23: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
[jira] [Commented] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[
https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878962#comment-16878962
]
Hive QA commented on HIVE-21955:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973674/HIVE-21955.1.branch-1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17863/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17863/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17863/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12973674/HIVE-21955.1.branch-1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973674 - PreCommit-HIVE-Build
> SearchArgumentImpl generates wrong ExpressionTree in some cases which might
> result in loss of data
> ---
>
> Key: HIVE-21955
> URL: https://issues.apache.org/jira/browse/HIVE-21955
> Project: Hive
> Issue Type: Bug
> Components: Hive, storage-api
>Affects Versions: 1.2.1
>Reporter: Zihao Ye
>Assignee: Zihao Ye
>Priority: Critical
> Labels: pushdown
> Attachments: HIVE-21955.1.branch-1.patch, HIVE-21955.branch-1.patch
>
>
> ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`,
> `convertToCNF`, `flatten` and `buildLeafList` in order to form a
> non-normalized expression into a CNF expression with the unique leaves.
> After an expression is converted to CNF, there might be *more than one
> non-leaf nodes which are exactly the same object* in the expression tree. If
> this happens, those non-leaf nodes will be visited more than once in
> `buildLeafList` function. As a result, a wrong ExpressionTree is generated.
> My version is 1.2.1, but it seems that the higher versions are also affected
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21910) Multiple target location generation in HostAffinitySplitLocationProvider
[
https://issues.apache.org/jira/browse/HIVE-21910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878960#comment-16878960
]
Hive QA commented on HIVE-21910:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973668/HIVE-21910.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 452 failed/errored test(s), 16363 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[alter_table_location2]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cte_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_nullscan]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_stats]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llapdecider]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_dp]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge10]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge_diff_fs]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_struct_type_vectorization]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_struct_type_vectorization]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_globallimit]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_meta_columns_decode]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[alter_merge_2_orc]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[alter_merge_orc]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[approx_distinct]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_smb_mapjoin_14]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_11]
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_13]
(batchId=178)
[jira] [Commented] (HIVE-21910) Multiple target location generation in HostAffinitySplitLocationProvider
[
https://issues.apache.org/jira/browse/HIVE-21910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878952#comment-16878952
]
Hive QA commented on HIVE-21910:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch llap-tez passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 37 unchanged - 1
fixed = 37 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
24s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Possible null pointer dereference of knownLocations in new
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider(List, int)
Dereferenced at HostAffinitySplitLocationProvider.java:knownLocations in new
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider(List, int)
Dereferenced at HostAffinitySplitLocationProvider.java:[line 59] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17861/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17861/yetus/new-findbugs-ql.html
|
| modules | C: common llap-tez ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17861/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Multiple target location generation in HostAffinitySplitLocationProvider
>
>
>
[jira] [Comment Edited] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[ https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878943#comment-16878943 ] Zihao Ye edited comment on HIVE-21955 at 7/5/19 3:15 AM: - The failed tests are mostly related to Java version issues rather than this patch. And the newly added UT was passed. See https://builds.apache.org/job/PreCommit-HIVE-Build/17855/testReport/org.apache.hadoop.hive.ql.io.sarg/TestSearchArgumentImpl/ was (Author: zihao.ye): The failed tests are mostly related to Java version issues rather than this patch. > SearchArgumentImpl generates wrong ExpressionTree in some cases which might > result in loss of data > --- > > Key: HIVE-21955 > URL: https://issues.apache.org/jira/browse/HIVE-21955 > Project: Hive > Issue Type: Bug > Components: Hive, storage-api >Affects Versions: 1.2.1 >Reporter: Zihao Ye >Assignee: Zihao Ye >Priority: Critical > Labels: pushdown > Attachments: HIVE-21955.1.branch-1.patch, HIVE-21955.branch-1.patch > > > ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, > `convertToCNF`, `flatten` and `buildLeafList` in order to form a > non-normalized expression into a CNF expression with the unique leaves. > After an expression is converted to CNF, there might be *more than one > non-leaf nodes which are exactly the same object* in the expression tree. If > this happens, those non-leaf nodes will be visited more than once in > `buildLeafList` function. As a result, a wrong ExpressionTree is generated. > My version is 1.2.1, but it seems that the higher versions are also affected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[ https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878943#comment-16878943 ] Zihao Ye commented on HIVE-21955: - The failed tests are mostly related to Java version issues rather than this patch. > SearchArgumentImpl generates wrong ExpressionTree in some cases which might > result in loss of data > --- > > Key: HIVE-21955 > URL: https://issues.apache.org/jira/browse/HIVE-21955 > Project: Hive > Issue Type: Bug > Components: Hive, storage-api >Affects Versions: 1.2.1 >Reporter: Zihao Ye >Assignee: Zihao Ye >Priority: Critical > Labels: pushdown > Attachments: HIVE-21955.1.branch-1.patch, HIVE-21955.branch-1.patch > > > ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, > `convertToCNF`, `flatten` and `buildLeafList` in order to form a > non-normalized expression into a CNF expression with the unique leaves. > After an expression is converted to CNF, there might be *more than one > non-leaf nodes which are exactly the same object* in the expression tree. If > this happens, those non-leaf nodes will be visited more than once in > `buildLeafList` function. As a result, a wrong ExpressionTree is generated. > My version is 1.2.1, but it seems that the higher versions are also affected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[
https://issues.apache.org/jira/browse/HIVE-21908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878940#comment-16878940
]
Hive QA commented on HIVE-21908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973709/HIVE-21908.04.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16305 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=160)
[external_jdbc_table_typeconversion.q,vector_udf_octet_length.q,schema_evol_orc_acidvec_table_update.q,materialized_view_rewrite_part_2.q,vector_decimal_5.q,vector_case_when_conversion.q,escape1.q,schema_evol_orc_acid_table_update_llap_io.q,cte_mat_5.q,acid_meta_columns_decode.q,vector_string_decimal.q,results_cache_lifetime.q,cross_prod_3.q,join46.q,dynpart_sort_optimization2.q,tez_bmj_schema_evolution.q,insert_into_default_keyword.q,bucketmapjoin4.q,vector_orc_null_check.q,semijoin7.q,uber_reduce.q,schema_evol_orc_nonvec_part_all_complex.q,is_distinct_from.q,schema_evol_text_vec_part_all_complex_llap_io.q,auto_sortmerge_join_3.q,vectorization_9.q,materialized_view_create_rewrite.q,merge2.q,join_nulls.q,bucketmapjoin2.q]
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17860/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17860/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17860/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973709 - PreCommit-HIVE-Build
> LlapDaemon node status should be reflected in the metrics
> -
>
> Key: HIVE-21908
> URL: https://issues.apache.org/jira/browse/HIVE-21908
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch,
> HIVE-21908.03.patch, HIVE-21908.04.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When we enable/disable a node it should be reflected in the LlapDaemon
> metrics, so the administrator can act upon the disabled nodes. They can
> manually check the status and either reenable them by restart, or fix the
> existing issues causing the problems
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[
https://issues.apache.org/jira/browse/HIVE-21908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878932#comment-16878932
]
Hive QA commented on HIVE-21908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 2 new + 141 unchanged
- 9 fixed = 143 total (was 150) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red} llap-server generated 1 new + 82 unchanged - 1 fixed =
83 total (was 83) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-server |
| | Should
org.apache.hadoop.hive.llap.daemon.impl.DumpingMetricsCollector$DumpingMetricsRecordBuilder
be a _static_ inner class? At DumpingMetricsCollector.java:inner class? At
DumpingMetricsCollector.java:[lines 41-106] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17860/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17860/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17860/yetus/new-findbugs-llap-server.html
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17860/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LlapDaemon node status should be reflected in the metrics
> -
>
> Key: HIVE-21908
> URL: https://issues.apache.org/jira/browse/HIVE-21908
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch,
> HIVE-21908.03.patch, HIVE-21908.04.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When we enable/disable a node it should be reflected in the LlapDaemon
> metrics, so the administrator can act upon the disabled nodes. They can
> manually check the status and either reenable them by restart, or fix the
> existing issues causing the problems
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21578) Introduce SQL:2016 formats FM, FX, and nested strings
[
https://issues.apache.org/jira/browse/HIVE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878926#comment-16878926
]
Hive QA commented on HIVE-21578:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973658/HIVE-21578.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16331 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=232)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17859/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17859/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17859/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973658 - PreCommit-HIVE-Build
> Introduce SQL:2016 formats FM, FX, and nested strings
> -
>
> Key: HIVE-21578
> URL: https://issues.apache.org/jira/browse/HIVE-21578
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21578.01.patch, HIVE-21578.02.patch,
> HIVE-21578.02.patch, HIVE-21578.03.patch, HIVE-21578.04.patch
>
>
> Enable Hive to parse the following datetime formats when any combination or
> subset of these or previously implemented formats is provided in one string.
> * "text" (nested strings)
> * FM
> * FX
> [Definitions
> here|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21578) Introduce SQL:2016 formats FM, FX, and nested strings
[
https://issues.apache.org/jira/browse/HIVE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878920#comment-16878920
]
Hive QA commented on HIVE-21578:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17859/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17859/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce SQL:2016 formats FM, FX, and nested strings
> -
>
> Key: HIVE-21578
> URL: https://issues.apache.org/jira/browse/HIVE-21578
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21578.01.patch, HIVE-21578.02.patch,
> HIVE-21578.02.patch, HIVE-21578.03.patch, HIVE-21578.04.patch
>
>
> Enable Hive to parse the following datetime formats when any combination or
> subset of these or previously implemented formats is provided in one string.
> * "text" (nested strings)
> * FM
> * FX
> [Definitions
> here|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878912#comment-16878912
]
Hive QA commented on HIVE-21912:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973682/HIVE-21912.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16368 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17858/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17858/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17858/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973682 - PreCommit-HIVE-Build
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.2.patch, HIVE-21912.3.patch,
> HIVE-21912.patch, HIVE-21912.wip-2.patch, HIVE-21912.wip.patch
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21571) SHOW COMPACTIONS shows column names as its first output row
[ https://issues.apache.org/jira/browse/HIVE-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21571: -- Attachment: HIVE-21571.07.patch Status: Patch Available (was: Open) > SHOW COMPACTIONS shows column names as its first output row > --- > > Key: HIVE-21571 > URL: https://issues.apache.org/jira/browse/HIVE-21571 > Project: Hive > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21571.01.patch, HIVE-21571.02.patch, > HIVE-21571.03.patch, HIVE-21571.04.patch, HIVE-21571.05.patch, > HIVE-21571.06.patch, HIVE-21571.07.patch, HIVE-21571.patch > > > SHOW COMPACTIONS yields a resultset with nice column names, and then the > first row of data is a repetition of those column names. This is somewhat > confusing and hard to read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21571) SHOW COMPACTIONS shows column names as its first output row
[ https://issues.apache.org/jira/browse/HIVE-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21571: -- Status: Open (was: Patch Available) > SHOW COMPACTIONS shows column names as its first output row > --- > > Key: HIVE-21571 > URL: https://issues.apache.org/jira/browse/HIVE-21571 > Project: Hive > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21571.01.patch, HIVE-21571.02.patch, > HIVE-21571.03.patch, HIVE-21571.04.patch, HIVE-21571.05.patch, > HIVE-21571.06.patch, HIVE-21571.07.patch, HIVE-21571.patch > > > SHOW COMPACTIONS yields a resultset with nice column names, and then the > first row of data is a repetition of those column names. This is somewhat > confusing and hard to read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878905#comment-16878905
]
Hive QA commented on HIVE-21912:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} llap-common in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} llap-client in master has 26 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
21s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} llap-common: The patch generated 7 new + 0 unchanged -
0 fixed = 7 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-client: The patch generated 1 new + 16 unchanged
- 3 fixed = 17 total (was 19) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 1 new + 0 unchanged -
7 fixed = 1 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} llap-tez generated 1 new + 17 unchanged - 0 fixed = 18
total (was 17) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} llap-tez generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-tez |
| |
org.apache.hadoop.hive.llap.tezplugins.metrics.BlacklistingLlapMetricsListener.newClusterMetrics(Map)
makes inefficient use of keySet iterator instead of entrySet iterator At
BlacklistingLlapMetricsListener.java:keySet iterator instead of entrySet
iterator At BlacklistingLlapMetricsListener.java:[line 112] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878904#comment-16878904
]
Hive QA commented on HIVE-21909:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973655/HIVE-21909.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16335 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17857/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17857/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17857/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973655 - PreCommit-HIVE-Build
> Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity
> when scheduling new tasks
> ---
>
> Key: HIVE-21909
> URL: https://issues.apache.org/jira/browse/HIVE-21909
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21909.01.patch, HIVE-21909.02.patch,
> HIVE-21909.03.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The changes in LlapDaemonNode status (or executor number) should be reflected
> in the ZooKeeper information, and we should make sure TezAMs are reading and
> honor the changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878891#comment-16878891
]
Hive QA commented on HIVE-21909:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} llap-client in master has 26 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
19s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-client: The patch generated 2 new + 29 unchanged
- 2 fixed = 31 total (was 31) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 1 new + 50 unchanged
- 0 fixed = 51 total (was 50) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17857/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17857/yetus/patch-mvninstall-llap-server.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17857/yetus/patch-mvninstall-service.txt
|
| compile |
[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Attachment: HIVE-21868.08.patch Status: Patch Available (was: Open) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch, > HIVE-21868.08.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Status: Open (was: Patch Available) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch, > HIVE-21868.08.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT
[
https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878887#comment-16878887
]
Hive QA commented on HIVE-21868:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973644/HIVE-21868.07.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16368 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17856/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17856/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17856/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973644 - PreCommit-HIVE-Build
> Vectorize CAST...FORMAT
> ---
>
> Key: HIVE-21868
> URL: https://issues.apache.org/jira/browse/HIVE-21868
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch,
> HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch,
> HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch
>
>
> Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT
> ) and CAST ( AS TIMESTAMP/DATE FORMAT ).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[ https://issues.apache.org/jira/browse/HIVE-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21959: -- Status: Patch Available (was: Open) > Clean up Concatenate and Msck DDL commands > -- > > Key: HIVE-21959 > URL: https://issues.apache.org/jira/browse/HIVE-21959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21959.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Concatenate and Msck DDL use basic data structure implementations instead of > their interfaces for variable declaration, like ArrayList instead of List, > LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in > the code, which needs to be cleaned. > Concatenate also had a non-immutable Desc class, that needs to be transformed > into an immutable one. Concatenate operation code should be cut into smaller > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT
[
https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878878#comment-16878878
]
Hive QA commented on HIVE-21868:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 2 new + 412 unchanged - 1
fixed = 414 total (was 413) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
23s{color} | {color:red} ql generated 2 new + 2253 unchanged - 0 fixed = 2255
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector,
long[], int, HiveSqlDateTimeFormatter):in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector,
long[], int, HiveSqlDateTimeFormatter): String.getBytes() At
CastDateToString.java:[line 70] |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector,
TimestampColumnVector, int, HiveSqlDateTimeFormatter):in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector,
TimestampColumnVector, int, HiveSqlDateTimeFormatter): String.getBytes() At
CastTimestampToString.java:[line 79] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17856/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17856/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17856/yetus/new-findbugs-ql.html
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17856/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize CAST...FORMAT
> ---
>
> Key: HIVE-21868
>
[jira] [Commented] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[ https://issues.apache.org/jira/browse/HIVE-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878874#comment-16878874 ] Miklos Gergely commented on HIVE-21959: --- [~jcamachorodriguez] could you please review: [https://github.com/apache/hive/pull/703] > Clean up Concatenate and Msck DDL commands > -- > > Key: HIVE-21959 > URL: https://issues.apache.org/jira/browse/HIVE-21959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21959.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Concatenate and Msck DDL use basic data structure implementations instead of > their interfaces for variable declaration, like ArrayList instead of List, > LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in > the code, which needs to be cleaned. > Concatenate also had a non-immutable Desc class, that needs to be transformed > into an immutable one. Concatenate operation code should be cut into smaller > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[ https://issues.apache.org/jira/browse/HIVE-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21959: -- Labels: pull-request-available refactor-ddl (was: refactor-ddl) > Clean up Concatenate and Msck DDL commands > -- > > Key: HIVE-21959 > URL: https://issues.apache.org/jira/browse/HIVE-21959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21959.01.patch > > > Concatenate and Msck DDL use basic data structure implementations instead of > their interfaces for variable declaration, like ArrayList instead of List, > LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in > the code, which needs to be cleaned. > Concatenate also had a non-immutable Desc class, that needs to be transformed > into an immutable one. Concatenate operation code should be cut into smaller > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[ https://issues.apache.org/jira/browse/HIVE-21959?focusedWorklogId=272352=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272352 ] ASF GitHub Bot logged work on HIVE-21959: - Author: ASF GitHub Bot Created on: 04/Jul/19 20:15 Start Date: 04/Jul/19 20:15 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #703: HIVE-21959 Clean up Concatenate and Msck DDL commands URL: https://github.com/apache/hive/pull/703 Concatenate and Msck DDL use basic data structure implementations instead of their interfaces for variable declaration, like ArrayList instead of List, LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in the code, which needs to be cleaned. Concatenate also had a non-immutable Desc class, that needs to be transformed into an immutable one. Concatenate operation code should be cut into smaller functions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272352) Time Spent: 10m Remaining Estimate: 0h > Clean up Concatenate and Msck DDL commands > -- > > Key: HIVE-21959 > URL: https://issues.apache.org/jira/browse/HIVE-21959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21959.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Concatenate and Msck DDL use basic data structure implementations instead of > their interfaces for variable declaration, like ArrayList instead of List, > LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in > the code, which needs to be cleaned. > Concatenate also had a non-immutable Desc class, that needs to be transformed > into an immutable one. Concatenate operation code should be cut into smaller > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[ https://issues.apache.org/jira/browse/HIVE-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21959: -- Attachment: HIVE-21959.01.patch > Clean up Concatenate and Msck DDL commands > -- > > Key: HIVE-21959 > URL: https://issues.apache.org/jira/browse/HIVE-21959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21959.01.patch > > > Concatenate and Msck DDL use basic data structure implementations instead of > their interfaces for variable declaration, like ArrayList instead of List, > LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in > the code, which needs to be cleaned. > Concatenate also had a non-immutable Desc class, that needs to be transformed > into an immutable one. Concatenate operation code should be cut into smaller > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[ https://issues.apache.org/jira/browse/HIVE-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21959: - > Clean up Concatenate and Msck DDL commands > -- > > Key: HIVE-21959 > URL: https://issues.apache.org/jira/browse/HIVE-21959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > > Concatenate and Msck DDL use basic data structure implementations instead of > their interfaces for variable declaration, like ArrayList instead of List, > LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in > the code, which needs to be cleaned. > Concatenate also had a non-immutable Desc class, that needs to be transformed > into an immutable one. Concatenate operation code should be cut into smaller > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[
https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878873#comment-16878873
]
Hive QA commented on HIVE-21955:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973674/HIVE-21955.1.branch-1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 99 failed/errored test(s), 9122 tests
executed
*Failed tests:*
{noformat}
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=452)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=449)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=448)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=446)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=451)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=202)
[multi_insert_mixed.q,smb_mapjoin_4.q,timestamp_comparison.q,join_cond_pushdown_3.q,insert1.q,union_remove_10.q,mapreduce2.q,bucketmapjoin_negative.q,udf_in_file.q,skewjoinopt5.q,auto_join12.q,skewjoin.q,vector_count_distinct.q,smb_mapjoin_3.q,stats10.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=205)
[auto_join30.q,union_remove_1.q,ppd_outer_join2.q,date_udf.q,join16.q,smb_mapjoin_13.q,bucketmapjoin7.q,smb_mapjoin_18.q,join19.q,metadata_only_queries.q,union6.q,cbo_subq_in.q,vectorization_part.q,groupby3_map_multi_distinct.q,vectorized_timestamp_funcs.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
[bucketmapjoin12.q,auto_join10.q,ptf_rcfile.q,join20.q,vector_elt.q,multi_insert.q,groupby_rollup1.q,ppd_join5.q,join_filters_overlap.q,vector_string_concat.q,join_empty.q,smb_mapjoin_6.q,auto_sortmerge_join_12.q,groupby_bigdata.q,innerjoin.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=207)
[tez_joins_explain.q,input17.q,ppd_gby_join.q,vectorized_rcfile_columnar.q,ppd_join.q,join_cond_pushdown_1.q,union_remove_6_subq.q,timestamp_3.q,load_dyn_part6.q,load_dyn_part9.q,multi_insert_gby2.q,vectorization_11.q,avro_compression_enabled_native.q,stats_noscan_2.q,transform1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=211)
[escape_distributeby1.q,join9.q,groupby2.q,vectorization_pushdown.q,union_date.q,join_cond_pushdown_unqual3.q,join8.q,sample10.q,ppd_outer_join3.q,cross_product_check_1.q,statsfs.q,auto_sortmerge_join_2.q,auto_join_stats.q,input_part2.q,groupby_multi_single_reducer3.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
[ppd_join4.q,load_dyn_part2.q,smb_mapjoin_7.q,vectorization_5.q,smb_mapjoin_2.q,ppd_join_filter.q,column_access_stats.q,stats0.q,vector_between_in.q,vectorized_string_funcs.q,vectorization_1.q,bucket_map_join_2.q,temp_table_join1.q,vectorized_case.q,stats_noscan_1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=217)
[stats12.q,groupby4.q,union_top_level.q,groupby10.q,subquery_in.q,mapjoin_filter_on_outerjoin.q,stats14.q,auto_sortmerge_join_4.q,limit_partition_metadataonly.q,load_dyn_part4.q,union3.q,smb_mapjoin_14.q,groupby3_noskew_multi_distinct.q,stats18.q,union_remove_21.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
[table_access_keys_stats.q,bucketmapjoin11.q,union_remove_9.q,mergejoins_mixed.q,join_nullsafe.q,stats8.q,auto_join28.q,skewjoinopt14.q,union17.q,vectorized_shufflejoin.q,groupby8_noskew.q,auto_sortmerge_join_10.q,groupby11.q,skewjoinopt11.q,load_dyn_part11.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=222)
[script_pipe.q,auto_join24.q,ptf_seqfile.q,union_remove_23.q,filter_join_breaktask.q,parallel_join0.q,join_thrift.q,vectorized_mapjoin.q,groupby8.q,union4.q,auto_join5.q,smb_mapjoin_20.q,groupby_multi_insert_common_distinct.q,join6.q,union_remove_16.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
[join36.q,union_remove_15.q,smb_mapjoin_10.q,bucket_map_join_tez1.q,temp_table.q,union_remove_13.q,auto_join8.q,auto_join6.q,auto_join0.q,vectorization_17.q,auto_join_stats2.q,skewjoin_union_remove_1.q,union16.q,auto_join32.q,union_remove_20.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_mapjoin (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_database_removes_partition_dirs
(batchId=98)
[jira] [Commented] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[
https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878854#comment-16878854
]
Hive QA commented on HIVE-21938:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973639/HIVE-21938.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16363 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17854/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17854/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17854/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973639 - PreCommit-HIVE-Build
> Add database and table filter options to PreUpgradeTool
> ---
>
> Key: HIVE-21938
> URL: https://issues.apache.org/jira/browse/HIVE-21938
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch
>
>
> By default pre upgrade tool scans all databases and tables in the warehouse.
> Add database and table filter options to run the tool for a specific subset
> of databases and tables only.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[
https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878832#comment-16878832
]
Hive QA commented on HIVE-21938:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 17 new +
59 unchanged - 9 fixed = 76 total (was 68) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17854/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17854/yetus/diff-checkstyle-upgrade-acid_pre-upgrade.txt
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17854/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add database and table filter options to PreUpgradeTool
> ---
>
> Key: HIVE-21938
> URL: https://issues.apache.org/jira/browse/HIVE-21938
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch
>
>
> By default pre upgrade tool scans all databases and tables in the warehouse.
> Add database and table filter options to run the tool for a specific subset
> of databases and tables only.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878825#comment-16878825
]
Hive QA commented on HIVE-21637:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
31s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} beeline in master has 44 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue}
standalone-metastore/metastore-tools/metastore-benchmarks in master has 3
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/util in master has 44 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} storage-api: The patch generated 1 new + 5 unchanged -
0 fixed = 6 total (was 5) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 9 new + 498 unchanged - 2 fixed = 507 total (was 500) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 164 new + 2193 unchanged - 65 fixed = 2357 total (was 2258) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 25 new + 962 unchanged - 10
fixed = 987 total (was 972) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} standalone-metastore/metastore-tools/tools-common: The
patch generated 5 new + 31 unchanged - 0 fixed = 36 total (was 31) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/hcatalog-unit: The patch generated 2 new + 24
unchanged - 3 fixed = 26 total (was 27) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 163
unchanged - 1 fixed = 166 total (was 164) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} |
[jira] [Work logged] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[
https://issues.apache.org/jira/browse/HIVE-21908?focusedWorklogId=272291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272291
]
ASF GitHub Bot logged work on HIVE-21908:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 16:37
Start Date: 04/Jul/19 16:37
Worklog Time Spent: 10m
Work Description: asinkovits commented on pull request #699: HIVE-21908:
LlapDaemon node status should be reflected in the metrics
URL: https://github.com/apache/hive/pull/699#discussion_r300462045
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/metrics/LlapDaemonExecutorInfo.java
##
@@ -64,6 +64,8 @@
ExecutorFallOffNumCompletedFragments("Number of completed fragments w.r.t
falloff values"),
AverageQueueTime("Average queue time for tasks"),
AverageResponseTime("Average response time for successful tasks"),
+ NumExecutorsEnabled("The number of executor threads enabled"),
+ WaitQueueSizeEnabled("Size of wait queue enabled"),
Review comment:
I've done some renaming, let me know what do you think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272291)
Time Spent: 1h (was: 50m)
> LlapDaemon node status should be reflected in the metrics
> -
>
> Key: HIVE-21908
> URL: https://issues.apache.org/jira/browse/HIVE-21908
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch,
> HIVE-21908.03.patch, HIVE-21908.04.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When we enable/disable a node it should be reflected in the LlapDaemon
> metrics, so the administrator can act upon the disabled nodes. They can
> manually check the status and either reenable them by restart, or fix the
> existing issues causing the problems
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[ https://issues.apache.org/jira/browse/HIVE-21908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-21908: --- Attachment: HIVE-21908.04.patch > LlapDaemon node status should be reflected in the metrics > - > > Key: HIVE-21908 > URL: https://issues.apache.org/jira/browse/HIVE-21908 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch, > HIVE-21908.03.patch, HIVE-21908.04.patch > > Time Spent: 50m > Remaining Estimate: 0h > > When we enable/disable a node it should be reflected in the LlapDaemon > metrics, so the administrator can act upon the disabled nodes. They can > manually check the status and either reenable them by restart, or fix the > existing issues causing the problems -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera reassigned HIVE-21958:
--
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21940) Metastore: Postgres text <-> clob mismatch for PARTITION_PARAMS/PARAM_VALUE
[
https://issues.apache.org/jira/browse/HIVE-21940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878772#comment-16878772
]
Hive QA commented on HIVE-21940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973630/HIVE-21940.01.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16362 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17852/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17852/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17852/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973630 - PreCommit-HIVE-Build
> Metastore: Postgres text <-> clob mismatch for PARTITION_PARAMS/PARAM_VALUE
> ---
>
> Key: HIVE-21940
> URL: https://issues.apache.org/jira/browse/HIVE-21940
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21940.01.patch, HIVE-21940.01.patch,
> HIVE-21940.repro.patch
>
>
> Summary: this is a fix for a regression introduced by HIVE-20833/HIVE-20221,
> fixed in the same way as HIVE-16667 earlier
> This issue was found while running sqoop/hive tests on a cluster with hive
> with postgres metastore, and it turned out the problem is that PARAM_VALUE is
> handled as it was CLOB but it's a text, so extractSqlClob returns it as is.
> It's reproducible on cluster by the following statements:
> {code}
> USE default;
> drop table if exists my_table;
> create external table my_table (col1 int, col3 int) partitioned by (col2
> string) STORED AS TEXTFILE;
> insert into my_table VALUES(11,201,"F");
> SELECT pp.* FROM sys.partition_params pp join sys.partitions p on p.part_id =
> pp.part_id join sys.tbls t on t.tbl_id = p.tbl_id where t.tbl_name =
> "my_table";
> {code}
> sys query results in:
> {code}
> +-++-+
> | pp.part_id | pp.param_key | pp.param_value |
> +-++-+
> | 151 | rawDataSize| 28629 |
> | 151 | numRows| 28628 |
> | 151 | transient_lastDdlTime | 28627 |
> | 151 | COLUMN_STATS_ACCURATE | 28626 |
> | 151 | numFiles | 28625 |
> | 151 | totalSize | 28622 |
> +-++-+
> {code}
> Seems like (propably) since HIVE-20833/HIVE-20221 there is an inconvenience
> while using PARTITION_PARAMS/PARAM_VALUE, because in postgres there is no
> such type as CLOB, and metastore simply saves large object ids into this
> field.
> In case of a direct metastore query (from hive's sys schema, but the same
> result for direct postgres), it shows the result above (see sys query
> output). This is an issue when hive treats these ids as they were real
> values, but they are obviously not correct, and this causes various failures
> (e.g. using serde parameter serialization.format=28392)
> param_value values above are large object ids, according to pg_dump
> | 151 | COLUMN_STATS_ACCURATE | 28626 |
> {code}
> SELECT pg_catalog.lo_open('28626', 131072);
> SELECT pg_catalog.lowrite(0,
> '\x7b2242415349435f5354415453223a2274727565222c22434f4c554d4e5f5354415453223a7b22636f6c31223a2274727565222c22636f6c33223a2274727565227d7d');
> SELECT pg_catalog.lo_close(0);
> {code}
> decoded large object value:
> {code}
> {"BASIC_STATS":"true","COLUMN_STATS":{"col1":"true","col3":"true"}}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21940) Metastore: Postgres text <-> clob mismatch for PARTITION_PARAMS/PARAM_VALUE
[
https://issues.apache.org/jira/browse/HIVE-21940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878748#comment-16878748
]
Hive QA commented on HIVE-21940:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 44 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17852/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore standalone-metastore/metastore-server ql
itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17852/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore: Postgres text <-> clob mismatch for PARTITION_PARAMS/PARAM_VALUE
> ---
>
> Key: HIVE-21940
> URL: https://issues.apache.org/jira/browse/HIVE-21940
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21940.01.patch, HIVE-21940.01.patch,
> HIVE-21940.repro.patch
>
>
> Summary: this is a fix for a regression introduced by HIVE-20833/HIVE-20221,
> fixed in the same way as HIVE-16667 earlier
> This issue was found while running sqoop/hive tests on a cluster with hive
> with postgres metastore, and it turned out the problem is that PARAM_VALUE is
> handled as it was CLOB but it's a text, so extractSqlClob returns it as is.
> It's reproducible on cluster by the following statements:
>
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: HIVE-21947.02.patch > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch, HIVE-21947.02.patch, > HIVE-21947.02.patch > > Time Spent: 40m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: (was: HIVE-21947.02.patch) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch, HIVE-21947.02.patch > > Time Spent: 40m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878721#comment-16878721
]
Hive QA commented on HIVE-18842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973629/HIVE-18842.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17851/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17851/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17851/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12973629/HIVE-18842.03.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973629 - PreCommit-HIVE-Build
> CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available, todoc4.0
> Attachments: HIVE-18842.01.patch, HIVE-18842.01.patch,
> HIVE-18842.02.patch, HIVE-18842.03.patch, HIVE-18842.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON+SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | DISTRIBUTED ON (col_name, ...) SORTED ON
> (col_name, ...)] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21947) Move Materialized View Update under DDL
[
https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878719#comment-16878719
]
Hive QA commented on HIVE-21947:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973624/HIVE-21947.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16332 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_transactional]
(batchId=164)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers2
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17850/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17850/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17850/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973624 - PreCommit-HIVE-Build
> Move Materialized View Update under DDL
> ---
>
> Key: HIVE-21947
> URL: https://issues.apache.org/jira/browse/HIVE-21947
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21947.01.patch, HIVE-21947.02.patch,
> HIVE-21947.02.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> MaterializedViewDesc is not part of the DDL framework, it has it's own task,
> which defines itself as DDL. They should be moved to the framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21951) Llap query on external table with header or footer returns incorrect row count.
[
https://issues.apache.org/jira/browse/HIVE-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21951:
Status: Patch Available (was: Open)
> Llap query on external table with header or footer returns incorrect row
> count.
> ---
>
> Key: HIVE-21951
> URL: https://issues.apache.org/jira/browse/HIVE-21951
> Project: Hive
> Issue Type: Bug
> Components: llap, Query Processor
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Attachments: HIVE-21951.01.patch, HIVE-21951.02.patch
>
>
> Create a table with header and footer as follows.
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS externaltableOpenCSV (eid int, name
> String, salary String, destination String)
> COMMENT 'Employee details'
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> STORED AS TEXTFILE
> LOCATION '/externaltableOpenCSV'
> tblproperties ("skip.header.line.count"="1", "skip.footer.line.count"="2");
> {code}
> Now, query on this table returns incorrect row count as header/footer are not
> skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21947) Move Materialized View Update under DDL
[
https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878673#comment-16878673
]
Hive QA commented on HIVE-21947:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 502 unchanged - 7
fixed = 502 total (was 509) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17850/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17850/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move Materialized View Update under DDL
> ---
>
> Key: HIVE-21947
> URL: https://issues.apache.org/jira/browse/HIVE-21947
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21947.01.patch, HIVE-21947.02.patch,
> HIVE-21947.02.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> MaterializedViewDesc is not part of the DDL framework, it has it's own task,
> which defines itself as DDL. They should be moved to the framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21957) Create temporary table like should omit transactional properties
[ https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21957: - Status: Patch Available (was: Open) > Create temporary table like should omit transactional properties > > > Key: HIVE-21957 > URL: https://issues.apache.org/jira/browse/HIVE-21957 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21957.01.patch > > > In case of create temporary table like queries, where the source table is > transactional, the transactional properties should not be copied over to the > new table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21957) Create temporary table like should omit transactional properties
[ https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21957: - Attachment: HIVE-21957.01.patch > Create temporary table like should omit transactional properties > > > Key: HIVE-21957 > URL: https://issues.apache.org/jira/browse/HIVE-21957 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21957.01.patch > > > In case of create temporary table like queries, where the source table is > transactional, the transactional properties should not be copied over to the > new table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21951) Llap query on external table with header or footer returns incorrect row count.
[
https://issues.apache.org/jira/browse/HIVE-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21951:
Status: Open (was: Patch Available)
> Llap query on external table with header or footer returns incorrect row
> count.
> ---
>
> Key: HIVE-21951
> URL: https://issues.apache.org/jira/browse/HIVE-21951
> Project: Hive
> Issue Type: Bug
> Components: llap, Query Processor
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Attachments: HIVE-21951.01.patch, HIVE-21951.02.patch
>
>
> Create a table with header and footer as follows.
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS externaltableOpenCSV (eid int, name
> String, salary String, destination String)
> COMMENT 'Employee details'
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> STORED AS TEXTFILE
> LOCATION '/externaltableOpenCSV'
> tblproperties ("skip.header.line.count"="1", "skip.footer.line.count"="2");
> {code}
> Now, query on this table returns incorrect row count as header/footer are not
> skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21951) Llap query on external table with header or footer returns incorrect row count.
[
https://issues.apache.org/jira/browse/HIVE-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21951:
Attachment: HIVE-21951.02.patch
> Llap query on external table with header or footer returns incorrect row
> count.
> ---
>
> Key: HIVE-21951
> URL: https://issues.apache.org/jira/browse/HIVE-21951
> Project: Hive
> Issue Type: Bug
> Components: llap, Query Processor
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Attachments: HIVE-21951.01.patch, HIVE-21951.02.patch
>
>
> Create a table with header and footer as follows.
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS externaltableOpenCSV (eid int, name
> String, salary String, destination String)
> COMMENT 'Employee details'
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> STORED AS TEXTFILE
> LOCATION '/externaltableOpenCSV'
> tblproperties ("skip.header.line.count"="1", "skip.footer.line.count"="2");
> {code}
> Now, query on this table returns incorrect row count as header/footer are not
> skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878650#comment-16878650
]
Hive QA commented on HIVE-21225:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973614/HIVE-21225.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 59 failed/errored test(s), 16361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug]
(batchId=89)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim]
(batchId=186)
org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341)
org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01
(batchId=341)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion
(batchId=298)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298)
org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket
(batchId=322)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize
(batchId=313)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas
(batchId=313)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize
(batchId=313)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas
(batchId=313)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps
(batchId=313)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache
(batchId=313)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen
(batchId=313)
[jira] [Updated] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[ https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21912: -- Attachment: HIVE-21912.3.patch > Implement BlacklistingLlapMetricsListener > - > > Key: HIVE-21912 > URL: https://issues.apache.org/jira/browse/HIVE-21912 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21912.2.patch, HIVE-21912.3.patch, > HIVE-21912.patch, HIVE-21912.wip-2.patch, HIVE-21912.wip.patch > > Time Spent: 6h > Remaining Estimate: 0h > > We should implement a DaemonStatisticsHandler which: > * If a node average response time is bigger than 150% (configurable) of the > other nodes > * If the other nodes has enough empty executors to handle the requests > Then disables the limping node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21957) Create temporary table like should omit transactional properties
[ https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-21957: > Create temporary table like should omit transactional properties > > > Key: HIVE-21957 > URL: https://issues.apache.org/jira/browse/HIVE-21957 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > In case of create temporary table like queries, where the source table is > transactional, the transactional properties should not be copied over to the > new table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878620#comment-16878620
]
Hive QA commented on HIVE-21225:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 16 new + 169 unchanged - 1
fixed = 185 total (was 170) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
12s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Unread field:AcidUtils.java:[line 1402] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17849/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus/new-findbugs-ql.html
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: getAcidState() should cache a recursive dir listing locally
> -
>
> Key: HIVE-21225
> URL: https://issues.apache.org/jira/browse/HIVE-21225
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch,
> HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch,
> HIVE-21225.5.patch, HIVE-21225.6.patch, async-pid-44-2.svg
>
>
> Currently
[jira] [Updated] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[ https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihao Ye updated HIVE-21955: Description: ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, `convertToCNF`, `flatten` and `buildLeafList` in order to form a non-normalized expression into a CNF expression with the unique leaves. After an expression is converted to CNF, there might be *more than one non-leaf nodes which are exactly the same object* in the expression tree. If this happens, those non-leaf nodes will be visited more than once in `buildLeafList` function. As a result, a wrong ExpressionTree is generated. My version is 1.2.1, but it seems that the higher versions are also affected was: ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, `convertToCNF`, `flatten` and `buildLeafList` in order to form a non-normalized expression into a CNF expression with the unique leaves. After an expression is converted to CNF, there might be *more than one non-leaf node which are exactly the same object* in the expression tree. If this happens, those non-leaf node will be visited more than once in `buildLeafList` function. As a result, a wrong ExpressionTree is generated. My version is 1.2.1, but it seems that the higher versions are also affected > SearchArgumentImpl generates wrong ExpressionTree in some cases which might > result in loss of data > --- > > Key: HIVE-21955 > URL: https://issues.apache.org/jira/browse/HIVE-21955 > Project: Hive > Issue Type: Bug > Components: Hive, storage-api >Affects Versions: 1.2.1 >Reporter: Zihao Ye >Assignee: Zihao Ye >Priority: Critical > Labels: pushdown > Attachments: HIVE-21955.1.branch-1.patch, HIVE-21955.branch-1.patch > > > ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, > `convertToCNF`, `flatten` and `buildLeafList` in order to form a > non-normalized expression into a CNF expression with the unique leaves. > After an expression is converted to CNF, there might be *more than one > non-leaf nodes which are exactly the same object* in the expression tree. If > this happens, those non-leaf nodes will be visited more than once in > `buildLeafList` function. As a result, a wrong ExpressionTree is generated. > My version is 1.2.1, but it seems that the higher versions are also affected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[ https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihao Ye updated HIVE-21955: Description: ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, `convertToCNF`, `flatten` and `buildLeafList` in order to form a non-normalized expression into a CNF expression with the unique leaves. After an expression is converted to CNF, there might be *more than one non-leaf node which are exactly the same object* in the expression tree. If this happens, those non-leaf node will be visited more than once in `buildLeafList` function. As a result, a wrong ExpressionTree is generated. My version is 1.2.1, but it seems that the higher versions are also affected was: ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, `convertToCNF`, `flatten` and `buildLeafList` in order to form a non-normalized expression into a CNF expression with the unique leaves. After an expression is converted to CNF, there might be more than one non-leaf node which are exactly the same object in the expression tree. If this happens, those non-leaf node will be visited more than once in `buildLeafList` function. As a result, a wrong ExpressionTree is generated. My version is 1.2.1, but it seems that the higher versions are also affected > SearchArgumentImpl generates wrong ExpressionTree in some cases which might > result in loss of data > --- > > Key: HIVE-21955 > URL: https://issues.apache.org/jira/browse/HIVE-21955 > Project: Hive > Issue Type: Bug > Components: Hive, storage-api >Affects Versions: 1.2.1 >Reporter: Zihao Ye >Assignee: Zihao Ye >Priority: Critical > Labels: pushdown > Attachments: HIVE-21955.1.branch-1.patch, HIVE-21955.branch-1.patch > > > ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, > `convertToCNF`, `flatten` and `buildLeafList` in order to form a > non-normalized expression into a CNF expression with the unique leaves. > After an expression is converted to CNF, there might be *more than one > non-leaf node which are exactly the same object* in the expression tree. If > this happens, those non-leaf node will be visited more than once in > `buildLeafList` function. As a result, a wrong ExpressionTree is generated. > My version is 1.2.1, but it seems that the higher versions are also affected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[ https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihao Ye updated HIVE-21955: Status: Open (was: Patch Available) Didn't attach a UT > SearchArgumentImpl generates wrong ExpressionTree in some cases which might > result in loss of data > --- > > Key: HIVE-21955 > URL: https://issues.apache.org/jira/browse/HIVE-21955 > Project: Hive > Issue Type: Bug > Components: Hive, storage-api >Affects Versions: 1.2.1 >Reporter: Zihao Ye >Assignee: Zihao Ye >Priority: Critical > Labels: pushdown > Attachments: HIVE-21955.branch-1.patch > > > ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, > `convertToCNF`, `flatten` and `buildLeafList` in order to form a > non-normalized expression into a CNF expression with the unique leaves. > After an expression is converted to CNF, there might be more than one > non-leaf node which are exactly the same object in the expression tree. If > this happens, those non-leaf node will be visited more than once in > `buildLeafList` function. As a result, a wrong ExpressionTree is generated. > My version is 1.2.1, but it seems that the higher versions are also affected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21955) SearchArgumentImpl generates wrong ExpressionTree in some cases which might result in loss of data
[ https://issues.apache.org/jira/browse/HIVE-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihao Ye updated HIVE-21955: Attachment: HIVE-21955.1.branch-1.patch Status: Patch Available (was: Open) UT added > SearchArgumentImpl generates wrong ExpressionTree in some cases which might > result in loss of data > --- > > Key: HIVE-21955 > URL: https://issues.apache.org/jira/browse/HIVE-21955 > Project: Hive > Issue Type: Bug > Components: Hive, storage-api >Affects Versions: 1.2.1 >Reporter: Zihao Ye >Assignee: Zihao Ye >Priority: Critical > Labels: pushdown > Attachments: HIVE-21955.1.branch-1.patch, HIVE-21955.branch-1.patch > > > ExpressionBuilder applies `pushDownNot`, `foldMaybe`, `flatten`, > `convertToCNF`, `flatten` and `buildLeafList` in order to form a > non-normalized expression into a CNF expression with the unique leaves. > After an expression is converted to CNF, there might be more than one > non-leaf node which are exactly the same object in the expression tree. If > this happens, those non-leaf node will be visited more than once in > `buildLeafList` function. As a result, a wrong ExpressionTree is generated. > My version is 1.2.1, but it seems that the higher versions are also affected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878591#comment-16878591
]
Hive QA commented on HIVE-18842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973629/HIVE-18842.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17848/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17848/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17848/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12973629/HIVE-18842.03.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973629 - PreCommit-HIVE-Build
> CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available, todoc4.0
> Attachments: HIVE-18842.01.patch, HIVE-18842.01.patch,
> HIVE-18842.02.patch, HIVE-18842.03.patch, HIVE-18842.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON+SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | DISTRIBUTED ON (col_name, ...) SORTED ON
> (col_name, ...)] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21951) Llap query on external table with header or footer returns incorrect row count.
[
https://issues.apache.org/jira/browse/HIVE-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878590#comment-16878590
]
Hive QA commented on HIVE-21951:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973610/HIVE-21951.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[file_with_header_footer]
(batchId=195)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17847/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17847/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17847/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973610 - PreCommit-HIVE-Build
> Llap query on external table with header or footer returns incorrect row
> count.
> ---
>
> Key: HIVE-21951
> URL: https://issues.apache.org/jira/browse/HIVE-21951
> Project: Hive
> Issue Type: Bug
> Components: llap, Query Processor
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Attachments: HIVE-21951.01.patch
>
>
> Create a table with header and footer as follows.
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS externaltableOpenCSV (eid int, name
> String, salary String, destination String)
> COMMENT 'Employee details'
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> STORED AS TEXTFILE
> LOCATION '/externaltableOpenCSV'
> tblproperties ("skip.header.line.count"="1", "skip.footer.line.count"="2");
> {code}
> Now, query on this table returns incorrect row count as header/footer are not
> skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21880:
--
Attachment: HIVE-21880.05.patch
Status: Patch Available (was: In Progress)
Patch removing the new error code as per [~maheshk114]'s suggestion.
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21880.01.patch, HIVE-21880.02.patch,
> HIVE-21880.03.patch, HIVE-21880.04.patch, HIVE-21880.05.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Need tp enable
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
> which is disabled as it is flaky and randomly failing with below error.
> {code}
> Error Message
> Notification events are missing in the meta store.
> Stacktrace
> java.lang.IllegalStateException: Notification events are missing in the meta
> store.
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:189)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:231)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:227)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:282)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:265)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:289)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites(TestReplicationScenariosAcidTablesBootstrap.java:328)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Updated] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21880:
--
Status: In Progress (was: Patch Available)
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21880.01.patch, HIVE-21880.02.patch,
> HIVE-21880.03.patch, HIVE-21880.04.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Need tp enable
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
> which is disabled as it is flaky and randomly failing with below error.
> {code}
> Error Message
> Notification events are missing in the meta store.
> Stacktrace
> java.lang.IllegalStateException: Notification events are missing in the meta
> store.
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:189)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:231)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:227)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:282)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:265)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:289)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites(TestReplicationScenariosAcidTablesBootstrap.java:328)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
[jira] [Assigned] (HIVE-21956) Add the list of table selected by dump in the dump folder.
[ https://issues.apache.org/jira/browse/HIVE-21956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-21956: -- > Add the list of table selected by dump in the dump folder. > -- > > Key: HIVE-21956 > URL: https://issues.apache.org/jira/browse/HIVE-21956 > Project: Hive > Issue Type: Sub-task >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > > The list of tables selected by a dump should be kept in the dump folder as a > _tables file. This will help user to find out the tables replicated and the > list can be used by user for ranger and atlas policy replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21910) Multiple target location generation in HostAffinitySplitLocationProvider
[ https://issues.apache.org/jira/browse/HIVE-21910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878562#comment-16878562 ] Peter Vary commented on HIVE-21910: --- Rebased, removed the HDFS locality based config since that was reverted > Multiple target location generation in HostAffinitySplitLocationProvider > > > Key: HIVE-21910 > URL: https://issues.apache.org/jira/browse/HIVE-21910 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21910.2.patch, HIVE-21910.3.patch, > HIVE-21910.4.patch, HIVE-21910.5.patch, HIVE-21910.6.patch, HIVE-21910.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We need to generate multiple target locations by > HostAffinitySplitLocationProvider, so we will have deterministic fallback > nodes in case the target node is disabled -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21910) Multiple target location generation in HostAffinitySplitLocationProvider
[ https://issues.apache.org/jira/browse/HIVE-21910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21910: -- Attachment: HIVE-21910.6.patch > Multiple target location generation in HostAffinitySplitLocationProvider > > > Key: HIVE-21910 > URL: https://issues.apache.org/jira/browse/HIVE-21910 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21910.2.patch, HIVE-21910.3.patch, > HIVE-21910.4.patch, HIVE-21910.5.patch, HIVE-21910.6.patch, HIVE-21910.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We need to generate multiple target locations by > HostAffinitySplitLocationProvider, so we will have deterministic fallback > nodes in case the target node is disabled -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21951) Llap query on external table with header or footer returns incorrect row count.
[
https://issues.apache.org/jira/browse/HIVE-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878561#comment-16878561
]
Hive QA commented on HIVE-21951:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 146 unchanged - 0
fixed = 149 total (was 146) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17847/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17847/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17847/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Llap query on external table with header or footer returns incorrect row
> count.
> ---
>
> Key: HIVE-21951
> URL: https://issues.apache.org/jira/browse/HIVE-21951
> Project: Hive
> Issue Type: Bug
> Components: llap, Query Processor
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Attachments: HIVE-21951.01.patch
>
>
> Create a table with header and footer as follows.
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS externaltableOpenCSV (eid int, name
> String, salary String, destination String)
> COMMENT 'Employee details'
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> STORED AS TEXTFILE
> LOCATION '/externaltableOpenCSV'
> tblproperties ("skip.header.line.count"="1", "skip.footer.line.count"="2");
> {code}
> Now, query on this table returns incorrect row count as header/footer are not
> skipped.
--
This
[jira] [Work logged] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[
https://issues.apache.org/jira/browse/HIVE-21908?focusedWorklogId=272122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272122
]
ASF GitHub Bot logged work on HIVE-21908:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 10:53
Start Date: 04/Jul/19 10:53
Worklog Time Spent: 10m
Work Description: asinkovits commented on pull request #699: HIVE-21908:
LlapDaemon node status should be reflected in the metrics
URL: https://github.com/apache/hive/pull/699#discussion_r300345174
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -213,6 +215,10 @@ public synchronized void setCapacity(int newNumExecutors,
int newWaitQueueSize)
numSlotsAvailable.addAndGet(newNumExecutors - maxParallelExecutors);
maxParallelExecutors = newNumExecutors;
waitQueue.setWaitQueueSize(newWaitQueueSize);
+if (metrics != null) {
Review comment:
Yes, this makes sense. I think removing all those null checks just makes the
code more readable. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272122)
Time Spent: 50m (was: 40m)
> LlapDaemon node status should be reflected in the metrics
> -
>
> Key: HIVE-21908
> URL: https://issues.apache.org/jira/browse/HIVE-21908
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch,
> HIVE-21908.03.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When we enable/disable a node it should be reflected in the LlapDaemon
> metrics, so the administrator can act upon the disabled nodes. They can
> manually check the status and either reenable them by restart, or fix the
> existing issues causing the problems
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[ https://issues.apache.org/jira/browse/HIVE-21908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-21908: --- Attachment: HIVE-21908.03.patch > LlapDaemon node status should be reflected in the metrics > - > > Key: HIVE-21908 > URL: https://issues.apache.org/jira/browse/HIVE-21908 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch, > HIVE-21908.03.patch > > Time Spent: 40m > Remaining Estimate: 0h > > When we enable/disable a node it should be reflected in the LlapDaemon > metrics, so the administrator can act upon the disabled nodes. They can > manually check the status and either reenable them by restart, or fix the > existing issues causing the problems -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?focusedWorklogId=272117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272117
]
ASF GitHub Bot logged work on HIVE-21880:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 10:40
Start Date: 04/Jul/19 10:40
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #684: HIVE-21880 :
Enable flaky test
TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
URL: https://github.com/apache/hive/pull/684#discussion_r300235355
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
##
@@ -3236,18 +3238,28 @@ public NotificationEventResponse
getNextNotification(long lastEventId, int maxEv
NotificationEventResponse filtered = new NotificationEventResponse();
if (rsp != null && rsp.getEvents() != null) {
long nextEventId = lastEventId + 1;
+ long prevEventId = lastEventId;
for (NotificationEvent e : rsp.getEvents()) {
+LOG.debug("Got event with id : " + e.getEventId());
if (e.getEventId() != nextEventId) {
- LOG.error("Requested events are found missing in NOTIFICATION_LOG
table. Expected: {}, Actual: {}. "
- + "Probably, cleaner would've cleaned it up. "
- + "Try setting higher value for
hive.metastore.event.db.listener.timetolive. "
- + "Also, bootstrap the system again to get back the
consistent replicated state.",
- nextEventId, e.getEventId());
- throw new IllegalStateException(REPL_EVENTS_MISSING_IN_METASTORE);
+ if (e.getEventId() == prevEventId) {
+LOG.error("NOTIFICATION_LOG table has multiple events with the
same event Id {}. " +
+"Something went wrong when inserting notification events.
Bootstrap the system " +
+"again to get back teh consistent replicated state.",
prevEventId);
Review comment:
But this one is also misleading. If the event id comes is less than
prevEventId ...then also it will report event missing ..though that is also not
possible as per code. nextEventId and e.getEventId() are already logged so i
think this new error is not required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272117)
Time Spent: 3h (was: 2h 50m)
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21880.01.patch, HIVE-21880.02.patch,
> HIVE-21880.03.patch, HIVE-21880.04.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Need tp enable
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
> which is disabled as it is flaky and randomly failing with below error.
> {code}
> Error Message
> Notification events are missing in the meta store.
> Stacktrace
> java.lang.IllegalStateException: Notification events are missing in the meta
> store.
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
> at
>
[jira] [Updated] (HIVE-21578) Introduce SQL:2016 formats FM, FX, and nested strings
[ https://issues.apache.org/jira/browse/HIVE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21578: - Attachment: HIVE-21578.04.patch Status: Patch Available (was: Open) > Introduce SQL:2016 formats FM, FX, and nested strings > - > > Key: HIVE-21578 > URL: https://issues.apache.org/jira/browse/HIVE-21578 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21578.01.patch, HIVE-21578.02.patch, > HIVE-21578.02.patch, HIVE-21578.03.patch, HIVE-21578.04.patch > > > Enable Hive to parse the following datetime formats when any combination or > subset of these or previously implemented formats is provided in one string. > * "text" (nested strings) > * FM > * FX > [Definitions > here|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[
https://issues.apache.org/jira/browse/HIVE-21908?focusedWorklogId=272105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272105
]
ASF GitHub Bot logged work on HIVE-21908:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:55
Start Date: 04/Jul/19 09:55
Worklog Time Spent: 10m
Work Description: asinkovits commented on pull request #699: HIVE-21908:
LlapDaemon node status should be reflected in the metrics
URL: https://github.com/apache/hive/pull/699#discussion_r300324355
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/metrics/LlapDaemonExecutorInfo.java
##
@@ -64,6 +64,8 @@
ExecutorFallOffNumCompletedFragments("Number of completed fragments w.r.t
falloff values"),
AverageQueueTime("Average queue time for tasks"),
AverageResponseTime("Average response time for successful tasks"),
+ NumExecutorsEnabled("The number of executor threads enabled"),
+ WaitQueueSizeEnabled("Size of wait queue enabled"),
Review comment:
We already have a ExecutorWaitQueueSize("Size of wait queue per node"),
which is the original wait queue size. (Now it should be something like wait
queue max or something)
Should we rework the naming?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272105)
Time Spent: 40m (was: 0.5h)
> LlapDaemon node status should be reflected in the metrics
> -
>
> Key: HIVE-21908
> URL: https://issues.apache.org/jira/browse/HIVE-21908
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When we enable/disable a node it should be reflected in the LlapDaemon
> metrics, so the administrator can act upon the disabled nodes. They can
> manually check the status and either reenable them by restart, or fix the
> existing issues causing the problems
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[ https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21912: -- Attachment: HIVE-21912.2.patch > Implement BlacklistingLlapMetricsListener > - > > Key: HIVE-21912 > URL: https://issues.apache.org/jira/browse/HIVE-21912 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21912.2.patch, HIVE-21912.patch, > HIVE-21912.wip-2.patch, HIVE-21912.wip.patch > > Time Spent: 6h > Remaining Estimate: 0h > > We should implement a DaemonStatisticsHandler which: > * If a node average response time is bigger than 150% (configurable) of the > other nodes > * If the other nodes has enough empty executors to handle the requests > Then disables the limping node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272104=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272104
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:45
Start Date: 04/Jul/19 09:45
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300320098
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
+"Minimum served tasks should be greater than 0");
+Preconditions.checkArgument(minConfigChangeDelayMs > 0,
+"Minimum config change delay should be greater than 0");
+Preconditions.checkArgument(timeThreshold > 1.0f,
+"The time threshold should be greater than 1");
+Preconditions.checkArgument(maxBlacklistedNodes > 0,
+"The maximum number of blacklisted node should be greater than 1");
+
+LOG.debug("BlacklistingLlapMetricsListener initialized with " +
+ "minServedTasksNumber={}, " +
+ "minConfigChangeDelayMs={}, " +
+ "timeThreshold={}, " +
+ "emptyExecutorsThreshold={}, " +
+ "maxBlacklistedNodes={}",
+minServedTasksNumber, minConfigChangeDelayMs, timeThreshold,
emptyExecutorsThreshold, maxBlacklistedNodes);
+ }
+
+ @Override
+ public void init(Configuration conf, LlapRegistryService registry) {
+init(conf, registry,
[jira] [Updated] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-21909: --- Attachment: HIVE-21909.03.patch > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch, HIVE-21909.02.patch, > HIVE-21909.03.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272098
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:39
Start Date: 04/Jul/19 09:39
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300317487
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
+"Minimum served tasks should be greater than 0");
+Preconditions.checkArgument(minConfigChangeDelayMs > 0,
+"Minimum config change delay should be greater than 0");
+Preconditions.checkArgument(timeThreshold > 1.0f,
+"The time threshold should be greater than 1");
+Preconditions.checkArgument(maxBlacklistedNodes > 0,
+"The maximum number of blacklisted node should be greater than 1");
+
+LOG.debug("BlacklistingLlapMetricsListener initialized with " +
+ "minServedTasksNumber={}, " +
+ "minConfigChangeDelayMs={}, " +
+ "timeThreshold={}, " +
+ "emptyExecutorsThreshold={}, " +
+ "maxBlacklistedNodes={}",
+minServedTasksNumber, minConfigChangeDelayMs, timeThreshold,
emptyExecutorsThreshold, maxBlacklistedNodes);
+ }
+
+ @Override
+ public void init(Configuration conf, LlapRegistryService registry) {
+init(conf, registry,
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272097
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:38
Start Date: 04/Jul/19 09:38
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300317229
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
+"Minimum served tasks should be greater than 0");
+Preconditions.checkArgument(minConfigChangeDelayMs > 0,
+"Minimum config change delay should be greater than 0");
+Preconditions.checkArgument(timeThreshold > 1.0f,
+"The time threshold should be greater than 1");
+Preconditions.checkArgument(maxBlacklistedNodes > 0,
+"The maximum number of blacklisted node should be greater than 1");
+
+LOG.debug("BlacklistingLlapMetricsListener initialized with " +
+ "minServedTasksNumber={}, " +
+ "minConfigChangeDelayMs={}, " +
+ "timeThreshold={}, " +
+ "emptyExecutorsThreshold={}, " +
+ "maxBlacklistedNodes={}",
+minServedTasksNumber, minConfigChangeDelayMs, timeThreshold,
emptyExecutorsThreshold, maxBlacklistedNodes);
+ }
+
+ @Override
+ public void init(Configuration conf, LlapRegistryService registry) {
+init(conf, registry,
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272096
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:38
Start Date: 04/Jul/19 09:38
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300317082
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
+"Minimum served tasks should be greater than 0");
+Preconditions.checkArgument(minConfigChangeDelayMs > 0,
+"Minimum config change delay should be greater than 0");
+Preconditions.checkArgument(timeThreshold > 1.0f,
+"The time threshold should be greater than 1");
+Preconditions.checkArgument(maxBlacklistedNodes > 0,
+"The maximum number of blacklisted node should be greater than 1");
+
+LOG.debug("BlacklistingLlapMetricsListener initialized with " +
+ "minServedTasksNumber={}, " +
+ "minConfigChangeDelayMs={}, " +
+ "timeThreshold={}, " +
+ "emptyExecutorsThreshold={}, " +
+ "maxBlacklistedNodes={}",
+minServedTasksNumber, minConfigChangeDelayMs, timeThreshold,
emptyExecutorsThreshold, maxBlacklistedNodes);
+ }
+
+ @Override
+ public void init(Configuration conf, LlapRegistryService registry) {
+init(conf, registry,
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272093
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:34
Start Date: 04/Jul/19 09:34
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300315241
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
+"Minimum served tasks should be greater than 0");
+Preconditions.checkArgument(minConfigChangeDelayMs > 0,
+"Minimum config change delay should be greater than 0");
+Preconditions.checkArgument(timeThreshold > 1.0f,
+"The time threshold should be greater than 1");
+Preconditions.checkArgument(maxBlacklistedNodes > 0,
+"The maximum number of blacklisted node should be greater than 1");
+
+LOG.debug("BlacklistingLlapMetricsListener initialized with " +
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272093)
Time Spent: 5h 20m (was: 5h 10m)
> Implement BlacklistingLlapMetricsListener
>
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272092
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:33
Start Date: 04/Jul/19 09:33
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300314856
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272092)
Time Spent: 5h 10m (was: 5h)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch,
> HIVE-21912.wip.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272079
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:32
Start Date: 04/Jul/19 09:32
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300314381
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
Review comment:
I usually sticking to check only for technical limits when validating the
values. Different values can be technically valid and might be useful for
different deployments.
All said I am open for discussion
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272079)
Time Spent: 5h (was: 4h 50m)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272075
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:29
Start Date: 04/Jul/19 09:29
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300313040
##
File path:
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
##
@@ -416,4 +423,65 @@ protected String getZkPathUser(Configuration conf) {
// rather than relying on RegistryUtils.currentUser().
return HiveConf.getVar(conf, ConfVars.LLAP_ZK_REGISTRY_USER,
RegistryUtils.currentUser());
}
+
+ /**
+ * Locks the Llap Cluster for configuration change by setting the next
possible configuration
+ * change time. Until this time is reached the configuration should not be
changed.
+ * @param nextMinConfigChangeTime The next time when the cluster can be
reconfigured
+ * @return The result of the change (success if the lock is succeeded, and
the next possible
+ * configuration change time
+ */
+ public ConfigChangeLockResult lockForConfigChange(long
nextMinConfigChangeTime) {
+try {
+ if (nextChangeTime == null) {
+// Create the node with the
/llap-sasl/hiveuser/hostname/config-change/next-change path without retry
+nextChangeTime = new DistributedAtomicLong(zooKeeperClient,
+String.join("/", workersPath.substring(0,
workersPath.lastIndexOf('/')), CONFIG_CHANGE_PATH,
+CONFIG_CHANGE_NODE), (i, j, sleeper) -> false);
+nextChangeTime.initialize(0L);
+ }
+ AtomicValue current = nextChangeTime.get();
+ if (!current.succeeded()) {
+LOG.debug("Can not get the current configuration lock time");
+return new ConfigChangeLockResult(false, -1L);
+ }
+ if (current.postValue() >= nextMinConfigChangeTime) {
+LOG.debug("Can not set {}. Current value is {}.",
nextMinConfigChangeTime, current.postValue());
+return new ConfigChangeLockResult(false, current.postValue());
+ }
+ current = nextChangeTime.compareAndSet(current.postValue(),
nextMinConfigChangeTime);
+ if (!current.succeeded()) {
+LOG.debug("Can not set {}. Current value is changed to {}.",
nextMinConfigChangeTime, current.postValue());
+return new ConfigChangeLockResult(false, current.postValue());
+ }
+ return new ConfigChangeLockResult(true, current.postValue());
+} catch (Throwable t) {
+ LOG.info("Can not reserve configuration change lock", t);
+ return new ConfigChangeLockResult(false, -1L);
+}
+ }
+
+ public static class ConfigChangeLockResult {
+boolean success;
Review comment:
Yeah. Forget to finish my refactor.
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272075)
Time Spent: 4.5h (was: 4h 20m)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch,
> HIVE-21912.wip.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272077
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:29
Start Date: 04/Jul/19 09:29
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300313222
##
File path:
llap-server/src/test/org/apache/hadoop/hive/llap/registry/impl/TestLlapRegistryService.java
##
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.registry.impl;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.MiniLlapCluster;
+import
org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.ConfigChangeLockResult;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.assertFalse;
+
+import java.io.IOException;
+
+public class TestLlapRegistryService {
+ private static MiniLlapCluster cluster = null;
+ private static HiveConf conf = new HiveConf();
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+cluster = MiniLlapCluster.create("llap01", null, 1, 2L, false, false, 1L,
1);
+HiveConf.setVar(conf, HiveConf.ConfVars.LLAP_DAEMON_XMX_HEADROOM, "1");
+cluster.serviceInit(conf);
+cluster.serviceStart();
+ }
+
+ @AfterClass
+ public static void tearDown() throws IOException {
+if (cluster != null) {
+ cluster.serviceStop();
+}
+ }
+
+ @Test
+ public void testLockForConfigChange() {
+LlapRegistryService client1 = new LlapRegistryService(false);
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272077)
Time Spent: 4h 50m (was: 4h 40m)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch,
> HIVE-21912.wip.patch
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272076
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:29
Start Date: 04/Jul/19 09:29
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300313139
##
File path:
llap-server/src/test/org/apache/hadoop/hive/llap/registry/impl/TestLlapRegistryService.java
##
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.registry.impl;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.MiniLlapCluster;
+import
org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.ConfigChangeLockResult;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.assertFalse;
+
+import java.io.IOException;
+
+public class TestLlapRegistryService {
+ private static MiniLlapCluster cluster = null;
+ private static HiveConf conf = new HiveConf();
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+cluster = MiniLlapCluster.create("llap01", null, 1, 2L, false, false, 1L,
1);
+HiveConf.setVar(conf, HiveConf.ConfVars.LLAP_DAEMON_XMX_HEADROOM, "1");
+cluster.serviceInit(conf);
+cluster.serviceStart();
+ }
+
+ @AfterClass
+ public static void tearDown() throws IOException {
+if (cluster != null) {
+ cluster.serviceStop();
+}
+ }
+
+ @Test
+ public void testLockForConfigChange() {
+LlapRegistryService client1 = new LlapRegistryService(false);
+client1.init(conf);
+client1.start();
+
+LlapRegistryService client2 = new LlapRegistryService(false);
+client2.init(conf);
+client2.start();
+ConfigChangeLockResult result = null;
+
+assertTrue(client1.lockForConfigChange(1).success);
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272076)
Time Spent: 4h 40m (was: 4.5h)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch,
> HIVE-21912.wip.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272074=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272074
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:28
Start Date: 04/Jul/19 09:28
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300312540
##
File path:
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
##
@@ -416,4 +423,65 @@ protected String getZkPathUser(Configuration conf) {
// rather than relying on RegistryUtils.currentUser().
return HiveConf.getVar(conf, ConfVars.LLAP_ZK_REGISTRY_USER,
RegistryUtils.currentUser());
}
+
+ /**
+ * Locks the Llap Cluster for configuration change by setting the next
possible configuration
+ * change time. Until this time is reached the configuration should not be
changed.
+ * @param nextMinConfigChangeTime The next time when the cluster can be
reconfigured
+ * @return The result of the change (success if the lock is succeeded, and
the next possible
+ * configuration change time
+ */
+ public ConfigChangeLockResult lockForConfigChange(long
nextMinConfigChangeTime) {
+try {
+ if (nextChangeTime == null) {
+// Create the node with the
/llap-sasl/hiveuser/hostname/config-change/next-change path without retry
+nextChangeTime = new DistributedAtomicLong(zooKeeperClient,
+String.join("/", workersPath.substring(0,
workersPath.lastIndexOf('/')), CONFIG_CHANGE_PATH,
+CONFIG_CHANGE_NODE), (i, j, sleeper) -> false);
Review comment:
My understanding is that Curator keeps the ZooKeeper communication
errors/retries under the hood. If I understand correctly the retries in this
recipe is for incr/decr so we get the correct value. Since we use only
initialize/compareAndSet we should not be affected.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272074)
Time Spent: 4h 20m (was: 4h 10m)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch,
> HIVE-21912.wip.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[ https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272073 ] ASF GitHub Bot logged work on HIVE-21912: - Author: ASF GitHub Bot Created on: 04/Jul/19 09:25 Start Date: 04/Jul/19 09:25 Worklog Time Spent: 10m Work Description: pvary commented on pull request #698: HIVE-21912: Implement DisablingDaemonStatisticsHandler URL: https://github.com/apache/hive/pull/698#discussion_r30033 ## File path: llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java ## @@ -67,13 +70,17 @@ private final static String NAMESPACE_PREFIX = "llap-"; private static final String SLOT_PREFIX = "slot-"; private static final String SASL_LOGIN_CONTEXT_NAME = "LlapZooKeeperClient"; + private static final String CONFIG_CHANGE_PATH = "config-change"; + private static final String CONFIG_CHANGE_NODE = "next-change"; private SlotZnode slotZnode; // to be used by clients of ServiceRegistry TODO: this is unnecessary private DynamicServiceInstanceSet instances; + private DistributedAtomicLong nextChangeTime; Review comment: Renamed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272073) Time Spent: 4h 10m (was: 4h) > Implement BlacklistingLlapMetricsListener > - > > Key: HIVE-21912 > URL: https://issues.apache.org/jira/browse/HIVE-21912 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch, > HIVE-21912.wip.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > We should implement a DaemonStatisticsHandler which: > * If a node average response time is bigger than 150% (configurable) of the > other nodes > * If the other nodes has enough empty executors to handle the requests > Then disables the limping node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272072
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:23
Start Date: 04/Jul/19 09:23
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300310251
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
+"Minimum served tasks should be greater than 0");
+Preconditions.checkArgument(minConfigChangeDelayMs > 0,
+"Minimum config change delay should be greater than 0");
+Preconditions.checkArgument(timeThreshold > 1.0f,
+"The time threshold should be greater than 1");
+Preconditions.checkArgument(maxBlacklistedNodes > 0,
+"The maximum number of blacklisted node should be greater than 1");
+
+LOG.debug("BlacklistingLlapMetricsListener initialized with " +
+ "minServedTasksNumber={}, " +
+ "minConfigChangeDelayMs={}, " +
+ "timeThreshold={}, " +
+ "emptyExecutorsThreshold={}, " +
+ "maxBlacklistedNodes={}",
+minServedTasksNumber, minConfigChangeDelayMs, timeThreshold,
emptyExecutorsThreshold, maxBlacklistedNodes);
+ }
+
+ @Override
+ public void init(Configuration conf, LlapRegistryService registry) {
+init(conf, registry,
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878460#comment-16878460
]
Hive QA commented on HIVE-18842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973629/HIVE-18842.03.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16368 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=350)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17846/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17846/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17846/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973629 - PreCommit-HIVE-Build
> CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available, todoc4.0
> Attachments: HIVE-18842.01.patch, HIVE-18842.01.patch,
> HIVE-18842.02.patch, HIVE-18842.03.patch, HIVE-18842.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON+SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | DISTRIBUTED ON (col_name, ...) SORTED ON
> (col_name, ...)] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272067
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 09:00
Start Date: 04/Jul/19 09:00
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300300551
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapMetricsCollector.java
##
@@ -210,53 +206,30 @@ public LlapMetrics getMetrics(String workerIdentity) {
return Collections.unmodifiableMap(instanceStatisticsMap);
}
- /**
- * Creates a LlapManagementProtocolClientImpl from a given
LlapServiceInstance.
- */
- public static class LlapManagementProtocolClientImplFactory {
-private final Configuration conf;
-private final RetryPolicy retryPolicy;
-private final SocketFactory socketFactory;
-
-public LlapManagementProtocolClientImplFactory(Configuration conf,
RetryPolicy retryPolicy,
- SocketFactory
socketFactory) {
- this.conf = conf;
- this.retryPolicy = retryPolicy;
- this.socketFactory = socketFactory;
-}
-
-private static LlapManagementProtocolClientImplFactory
basicInstance(Configuration conf) {
- return new LlapManagementProtocolClientImplFactory(
- conf,
- RetryPolicies.retryUpToMaximumCountWithFixedSleep(5, 3000L,
TimeUnit.MILLISECONDS),
- NetUtils.getDefaultSocketFactory(conf));
-}
-
-public LlapManagementProtocolClientImpl create(LlapServiceInstance
serviceInstance) {
- LlapManagementProtocolClientImpl client = new
LlapManagementProtocolClientImpl(conf, serviceInstance.getHost(),
- serviceInstance.getManagementPort(), retryPolicy,
- socketFactory);
- return client;
-}
- }
-
/**
* Stores the metrics retrieved from the llap daemons, along with the
retrieval timestamp.
*/
public static class LlapMetrics {
private final long timestamp;
-private final LlapDaemonProtocolProtos.GetDaemonMetricsResponseProto
metrics;
+private final Map metrics;
+
+@VisibleForTesting
+LlapMetrics(long timestamp, Map metrics) {
+ this.timestamp = timestamp;
+ this.metrics = metrics;
+}
public LlapMetrics(LlapDaemonProtocolProtos.GetDaemonMetricsResponseProto
metrics) {
this.timestamp = System.currentTimeMillis();
- this.metrics = metrics;
+ this.metrics = new HashMap(metrics.getMetricsCount());
+ metrics.getMetricsList().forEach(entry ->
this.metrics.put(entry.getKey(), entry.getValue()));
}
public long getTimestamp() {
return timestamp;
}
-public LlapDaemonProtocolProtos.GetDaemonMetricsResponseProto getMetrics()
{
+public Map getMetrics() {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272067)
Time Spent: 3h 50m (was: 3h 40m)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch,
> HIVE-21912.wip.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272065
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 08:58
Start Date: 04/Jul/19 08:58
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300299243
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapManagementProtocolClientImplFactory.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+
+import javax.net.SocketFactory;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Creates a LlapManagementProtocolClientImpl from a given LlapServiceInstance.
+ */
+public class LlapManagementProtocolClientImplFactory {
+ private final Configuration conf;
+ private final RetryPolicy retryPolicy;
+ private final SocketFactory socketFactory;
+
+ public LlapManagementProtocolClientImplFactory(Configuration conf,
RetryPolicy retryPolicy,
+ SocketFactory socketFactory) {
+this.conf = conf;
+this.retryPolicy = retryPolicy;
+this.socketFactory = socketFactory;
+ }
+
+ public static LlapManagementProtocolClientImplFactory
basicInstance(Configuration conf) {
+return new LlapManagementProtocolClientImplFactory(
+conf,
+RetryPolicies.retryUpToMaximumCountWithFixedSleep(5, 3000L,
TimeUnit.MILLISECONDS),
+NetUtils.getDefaultSocketFactory(conf));
+ }
+
+ public LlapManagementProtocolClientImpl create(LlapServiceInstance
serviceInstance) {
+LlapManagementProtocolClientImpl client = new
LlapManagementProtocolClientImpl(conf, serviceInstance.getHost(),
Review comment:
Honestly? Just refactored the class without change, but good point.
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272065)
Time Spent: 3h 40m (was: 3.5h)
> Implement BlacklistingLlapMetricsListener
> -
>
> Key: HIVE-21912
> URL: https://issues.apache.org/jira/browse/HIVE-21912
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21912.patch, HIVE-21912.wip-2.patch,
> HIVE-21912.wip.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> We should implement a DaemonStatisticsHandler which:
> * If a node average response time is bigger than 150% (configurable) of the
> other nodes
> * If the other nodes has enough empty executors to handle the requests
> Then disables the limping node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21912) Implement BlacklistingLlapMetricsListener
[
https://issues.apache.org/jira/browse/HIVE-21912?focusedWorklogId=272062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272062
]
ASF GitHub Bot logged work on HIVE-21912:
-
Author: ASF GitHub Bot
Created on: 04/Jul/19 08:55
Start Date: 04/Jul/19 08:55
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #698: HIVE-21912:
Implement DisablingDaemonStatisticsHandler
URL: https://github.com/apache/hive/pull/698#discussion_r300298150
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/BlacklistingLlapMetricsListener.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.SetCapacityRequestProto;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonExecutorInfo;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of MetricsListener which blacklists slow nodes based on the
statistics.
+ */
+public class BlacklistingLlapMetricsListener implements LlapMetricsListener {
+ private static final Logger LOG =
LoggerFactory.getLogger(BlacklistingLlapMetricsListener.class);
+ private LlapRegistryService registry;
+ private LlapManagementProtocolClientImplFactory clientFactory;
+ private int minServedTasksNumber;
+ private int maxBlacklistedNodes;
+ private long minConfigChangeDelayMs;
+ private float timeThreshold;
+ private float emptyExecutorsThreshold;
+ @VisibleForTesting
+ long nextCheckTime = Long.MIN_VALUE;
+
+ @VisibleForTesting
+ void init(Configuration conf, LlapRegistryService registry,
LlapManagementProtocolClientImplFactory clientFactory) {
+this.registry = registry;
+this.clientFactory = clientFactory;
+this.minServedTasksNumber =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_SERVED_TASKS);
+this.minConfigChangeDelayMs =
+HiveConf.getTimeVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MIN_CHANGE_DELAY,
+TimeUnit.MILLISECONDS);
+this.timeThreshold =
+HiveConf.getFloatVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_TIME_THRESHOLD);
+this.emptyExecutorsThreshold =
+HiveConf.getFloatVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_EMPTY_EXECUTORS);
+this.maxBlacklistedNodes =
+HiveConf.getIntVar(conf,
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_BLACKLISTING_METRICS_LISTENER_MAX_LISTED_NODES);
+
+Preconditions.checkArgument(minServedTasksNumber > 0,
+"Minimum served tasks should be greater than 0");
+Preconditions.checkArgument(minConfigChangeDelayMs > 0,
+"Minimum config change delay should be greater than 0");
+Preconditions.checkArgument(timeThreshold > 1.0f,
+"The time threshold should be greater than 1");
+Preconditions.checkArgument(maxBlacklistedNodes > 0,
+"The maximum number of blacklisted node should be greater than 1");
+
+LOG.debug("BlacklistingLlapMetricsListener initialized with " +
+ "minServedTasksNumber={}, " +
+ "minConfigChangeDelayMs={}, " +
+ "timeThreshold={}, " +
+ "emptyExecutorsThreshold={}, " +
+ "maxBlacklistedNodes={}",
+minServedTasksNumber, minConfigChangeDelayMs, timeThreshold,
emptyExecutorsThreshold, maxBlacklistedNodes);
+ }
+
+ @Override
+ public void init(Configuration conf, LlapRegistryService registry) {
+init(conf, registry,
[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Attachment: HIVE-21868.07.patch Status: Patch Available (was: Open) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Status: Open (was: Patch Available) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Status: Patch Available (was: Open) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21868) Vectorize CAST...FORMAT
[ https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21868: - Status: Open (was: Patch Available) > Vectorize CAST...FORMAT > --- > > Key: HIVE-21868 > URL: https://issues.apache.org/jira/browse/HIVE-21868 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, > HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, > HIVE-21868.05.patch, HIVE-21868.06.patch, HIVE-21868.07.patch > > > Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT > ) and CAST ( AS TIMESTAMP/DATE FORMAT ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878433#comment-16878433
]
Hive QA commented on HIVE-18842:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 11 new + 707 unchanged - 2
fixed = 718 total (was 709) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 2 new + 2252 unchanged - 1 fixed = 2254
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to LA29_5 in
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At
HiveLexer.java:org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At HiveLexer.java:[line 12954] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA240 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
48890-48903] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17846/dev-support/hive-personality.sh
|
| git revision | master / 3223bda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17846/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17846/yetus/new-findbugs-ql.html
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17846/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho
