[jira] [Commented] (HIVE-18277) Use subtraction and addition should be a good tips when type not support with hplsql
[ https://issues.apache.org/jira/browse/HIVE-18277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299670#comment-16299670 ] ZhangBing Lin commented on HIVE-18277: -- [~dmtolpeko],can you plz take a quick review? > Use subtraction and addition should be a good tips when type not support with > hplsql > > > Key: HIVE-18277 > URL: https://issues.apache.org/jira/browse/HIVE-18277 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HIVE-18277.1.patch, HIVE-18277.2.patch > > > When users use subtraction and addition, when type does not support,it should > not return a null value, but should provide a information that Unsupported > data types -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin reassigned HIVE-17630: Assignee: (was: ZhangBing Lin) > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 2.2.0, 3.0.0 >Reporter: ZhangBing Lin >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin reassigned HIVE-17630: Assignee: Dmitry Tolpeko > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 2.2.0, 3.0.0 >Reporter: ZhangBing Lin >Assignee: Dmitry Tolpeko >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin reassigned HIVE-17630: Assignee: ZhangBing Lin (was: Dmitry Tolpeko) > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 2.2.0, 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work
[
https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299656#comment-16299656
]
Hive QA commented on HIVE-11609:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hbase-handler: The patch generated 22 new + 29
unchanged - 34 fixed = 51 total (was 63) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 21e18de |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8349/yetus/diff-checkstyle-hbase-handler.txt
|
| modules | C: ql hbase-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8349/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Capability to add a filter to hbase scan via composite key doesn't work
> ---
>
> Key: HIVE-11609
> URL: https://issues.apache.org/jira/browse/HIVE-11609
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Swarnim Kulkarni
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-11609.08.patch, HIVE-11609.09.patch,
> HIVE-11609.1.patch.txt, HIVE-11609.10.patch, HIVE-11609.2.patch.txt,
> HIVE-11609.3.patch.txt, HIVE-11609.4.patch.txt, HIVE-11609.5.patch,
> HIVE-11609.6.patch.txt, HIVE-11609.7.patch.txt
>
>
> It seems like the capability to add filter to an hbase scan which was added
> as part of HIVE-6411 doesn't work. This is primarily because in the
> HiveHBaseInputFormat, the filter is added in the getsplits instead of
> getrecordreader. This works fine for start and stop keys but not for filter
> because a filter is respected only when an actual scan is performed. This is
> also related to the initial refactoring that was done as part of HIVE-3420.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17829) ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
[
https://issues.apache.org/jira/browse/HIVE-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-17829:
---
Status: Patch Available (was: Open)
> ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
> --
>
> Key: HIVE-17829
> URL: https://issues.apache.org/jira/browse/HIVE-17829
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Chiran Ravani
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-17829.0.patch, HIVE-17829.1.patch
>
>
> Stack
> {code}
> 2017-10-09T09:39:54,804 ERROR [HiveServer2-Background-Pool: Thread-95]:
> metadata.Table (Table.java:getColsInternal(642)) - Unable to get field from
> serde: org.apache.hadoop.hive.hbase.HBaseSerDe
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.Arrays$ArrayList.get(Arrays.java:3841) ~[?:1.8.0_77]
> at
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.init(BaseStructObjectInspector.java:104)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.init(LazySimpleStructObjectInspector.java:97)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.(LazySimpleStructObjectInspector.java:77)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory.getLazySimpleStructObjectInspector(LazyObjectInspectorFactory.java:115)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseLazyObjectFactory.createLazyHBaseStructInspector(HBaseLazyObjectFactory.java:79)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:127)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4228)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
>
[jira] [Commented] (HIVE-18031) Support replication for Alter Database operation.
[ https://issues.apache.org/jira/browse/HIVE-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299645#comment-16299645 ] anishek commented on HIVE-18031: +1 , but failed tests need to be verified, if they are not failing because of the change, specifically "org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints" seems to be related to replication atleast. > Support replication for Alter Database operation. > - > > Key: HIVE-18031 > URL: https://issues.apache.org/jira/browse/HIVE-18031 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-18031.01.patch, HIVE-18031.02.patch > > > Currently alter database operations to alter the database properties or > description are not generating any events due to which it is not getting > replicated. > Need to add an event for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299638#comment-16299638
]
Hive QA commented on HIVE-16821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903124/HIVE-16821.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11536 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_context_ngrams]
(batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_multi_output_select]
(batchId=164)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8348/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8348/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8348/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903124 - PreCommit-HIVE-Build
> Vectorization: support Explain Analyze in vectorized mode
> -
>
> Key: HIVE-16821
> URL: https://issues.apache.org/jira/browse/HIVE-16821
> Project: Hive
> Issue Type: Bug
> Components: Diagnosability, Vectorization
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-16821.1.patch, HIVE-16821.2.patch,
> HIVE-16821.2.patch, HIVE-16821.3.patch, HIVE-16821.7.patch, HIVE-16821.8.patch
>
>
> Currently, to avoid a branch in the operator inner loop - the runtime stats
> are only available in non-vector mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17829) ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
[
https://issues.apache.org/jira/browse/HIVE-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-17829:
---
Attachment: (was: HIVE-17829.1.patch)
> ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
> --
>
> Key: HIVE-17829
> URL: https://issues.apache.org/jira/browse/HIVE-17829
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Chiran Ravani
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-17829.0.patch, HIVE-17829.1.patch
>
>
> Stack
> {code}
> 2017-10-09T09:39:54,804 ERROR [HiveServer2-Background-Pool: Thread-95]:
> metadata.Table (Table.java:getColsInternal(642)) - Unable to get field from
> serde: org.apache.hadoop.hive.hbase.HBaseSerDe
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.Arrays$ArrayList.get(Arrays.java:3841) ~[?:1.8.0_77]
> at
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.init(BaseStructObjectInspector.java:104)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.init(LazySimpleStructObjectInspector.java:97)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.(LazySimpleStructObjectInspector.java:77)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory.getLazySimpleStructObjectInspector(LazyObjectInspectorFactory.java:115)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseLazyObjectFactory.createLazyHBaseStructInspector(HBaseLazyObjectFactory.java:79)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:127)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4228)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
>
[jira] [Updated] (HIVE-17829) ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
[
https://issues.apache.org/jira/browse/HIVE-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-17829:
---
Attachment: HIVE-17829.1.patch
reattaching the same patch again so that hive build system picks up the change
and runs tests.
> ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
> --
>
> Key: HIVE-17829
> URL: https://issues.apache.org/jira/browse/HIVE-17829
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Chiran Ravani
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-17829.0.patch, HIVE-17829.1.patch
>
>
> Stack
> {code}
> 2017-10-09T09:39:54,804 ERROR [HiveServer2-Background-Pool: Thread-95]:
> metadata.Table (Table.java:getColsInternal(642)) - Unable to get field from
> serde: org.apache.hadoop.hive.hbase.HBaseSerDe
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.Arrays$ArrayList.get(Arrays.java:3841) ~[?:1.8.0_77]
> at
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.init(BaseStructObjectInspector.java:104)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.init(LazySimpleStructObjectInspector.java:97)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.(LazySimpleStructObjectInspector.java:77)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory.getLazySimpleStructObjectInspector(LazyObjectInspectorFactory.java:115)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseLazyObjectFactory.createLazyHBaseStructInspector(HBaseLazyObjectFactory.java:79)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:127)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4228)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
[jira] [Commented] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299603#comment-16299603
]
Hive QA commented on HIVE-16821:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 21e18de |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8348/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8348/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: support Explain Analyze in vectorized mode
> -
>
> Key: HIVE-16821
> URL: https://issues.apache.org/jira/browse/HIVE-16821
> Project: Hive
> Issue Type: Bug
> Components: Diagnosability, Vectorization
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-16821.1.patch, HIVE-16821.2.patch,
> HIVE-16821.2.patch, HIVE-16821.3.patch, HIVE-16821.7.patch, HIVE-16821.8.patch
>
>
> Currently, to avoid a branch in the operator inner loop - the runtime stats
> are only available in non-vector mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15393) Update Guava version
[
https://issues.apache.org/jira/browse/HIVE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299595#comment-16299595
]
Hive QA commented on HIVE-15393:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903117/HIVE-15393.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanQpChanges
(batchId=284)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8346/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8346/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8346/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903117 - PreCommit-HIVE-Build
> Update Guava version
>
>
> Key: HIVE-15393
> URL: https://issues.apache.org/jira/browse/HIVE-15393
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.2.0
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-15393.2.patch, HIVE-15393.3.patch,
> HIVE-15393.5.patch, HIVE-15393.6.patch, HIVE-15393.7.patch,
> HIVE-15393.8.patch, HIVE-15393.9.patch, HIVE-15393.patch
>
>
> Druid base code is using newer version of guava 16.0.1 that is not compatible
> with the current version used by Hive.
> FYI Hadoop project is moving to Guava 18 not sure if it is better to move to
> guava 18 or even 19.
> https://issues.apache.org/jira/browse/HADOOP-10101
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15393) Update Guava version
[
https://issues.apache.org/jira/browse/HIVE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299582#comment-16299582
]
Hive QA commented on HIVE-15393:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} common: The patch generated 1 new + 21 unchanged - 0
fixed = 22 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} llap-tez: The patch generated 1 new + 2 unchanged - 0
fixed = 3 total (was 2) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 1 new + 72 unchanged - 0 fixed
= 73 total (was 72) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} llap-server: The patch generated 9 new + 79 unchanged
- 5 fixed = 88 total (was 84) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} druid-handler: The patch generated 2 new + 11 unchanged
- 0 fixed = 13 total (was 11) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
31s{color} | {color:red} root: The patch generated 14 new + 197 unchanged - 5
fixed = 211 total (was 202) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 21e18de |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8346/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8346/yetus/diff-checkstyle-llap-tez.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8346/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8346/yetus/diff-checkstyle-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8346/yetus/diff-checkstyle-druid-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8346/yetus/diff-checkstyle-root.txt
|
| modules | C: common
[jira] [Updated] (HIVE-18317) Improve error messages in TransactionalValidationListerner
[ https://issues.apache.org/jira/browse/HIVE-18317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18317: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) committed to master thanks Jason for the review > Improve error messages in TransactionalValidationListerner > -- > > Key: HIVE-18317 > URL: https://issues.apache.org/jira/browse/HIVE-18317 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 3.0.0 > > Attachments: HIVE-18317.01.patch, HIVE-18317.02.patch, > HIVE-18317.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299530#comment-16299530
]
Hive QA commented on HIVE-18293:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903114/HIVE-18293.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11536 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8345/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8345/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8345/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903114 - PreCommit-HIVE-Build
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Johannes Alberti
>Priority: Critical
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
>
[jira] [Updated] (HIVE-18316) HiveEndPoint should only work with full acid tables
[ https://issues.apache.org/jira/browse/HIVE-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18316: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) committed to master thanks Sergey for the review > HiveEndPoint should only work with full acid tables > --- > > Key: HIVE-18316 > URL: https://issues.apache.org/jira/browse/HIVE-18316 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 3.0.0 > > Attachments: HIVE-18316.01.patch, HIVE-18316.02.patch > > > now that we have full acid and 1/4 acid the check needs to be updated to > check for full acid tables -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek reassigned HIVE-18193: -- Assignee: Sankar Hariappan (was: anishek) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Fix For: 3.0.0 > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299511#comment-16299511 ] anishek commented on HIVE-18193: Hey [~asherman] the design doc is attached in HIVE-18320 which will be the umbrella jira for acid related work. > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18320) Support ACID Tables Replication
[ https://issues.apache.org/jira/browse/HIVE-18320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-18320: --- Attachment: Hive Replication- ACID Tables.pdf Design doc for replicating ACID Tables. > Support ACID Tables Replication > --- > > Key: HIVE-18320 > URL: https://issues.apache.org/jira/browse/HIVE-18320 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, Metastore, repl, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: anishek > Labels: ACID, DR, replication > Attachments: Hive Replication- ACID Tables.pdf > > > Currently, Full ACID and MM (Micro-Managed) tables are not supported for > replication. > Need to support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17829) ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
[
https://issues.apache.org/jira/browse/HIVE-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299503#comment-16299503
]
Thejas M Nair commented on HIVE-17829:
--
+1
> ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
> --
>
> Key: HIVE-17829
> URL: https://issues.apache.org/jira/browse/HIVE-17829
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Chiran Ravani
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-17829.0.patch, HIVE-17829.1.patch
>
>
> Stack
> {code}
> 2017-10-09T09:39:54,804 ERROR [HiveServer2-Background-Pool: Thread-95]:
> metadata.Table (Table.java:getColsInternal(642)) - Unable to get field from
> serde: org.apache.hadoop.hive.hbase.HBaseSerDe
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.Arrays$ArrayList.get(Arrays.java:3841) ~[?:1.8.0_77]
> at
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.init(BaseStructObjectInspector.java:104)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.init(LazySimpleStructObjectInspector.java:97)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.(LazySimpleStructObjectInspector.java:77)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory.getLazySimpleStructObjectInspector(LazyObjectInspectorFactory.java:115)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseLazyObjectFactory.createLazyHBaseStructInspector(HBaseLazyObjectFactory.java:79)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:127)
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261)
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4228)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at
[jira] [Commented] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299494#comment-16299494
]
Hive QA commented on HIVE-18293:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 4e43ec7 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8345/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Johannes Alberti
>Priority: Critical
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
>
[jira] [Updated] (HIVE-18318) LLAP record reader should check interrupt even when not blocking
[ https://issues.apache.org/jira/browse/HIVE-18318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18318: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > LLAP record reader should check interrupt even when not blocking > > > Key: HIVE-18318 > URL: https://issues.apache.org/jira/browse/HIVE-18318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18318.patch > > > Hive operators don't check interrupts, and may not do blocking operations. > LLAP record reader only blocks in IO is slower than processing; so, if IO is > fast enough, it will not ever block (at least not interruptibly, the sync > w/IO on the object does not check interrupts), and thus never catch > interrupts. > So, the task would be impossible to terminate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18318) LLAP record reader should check interrupt even when not blocking
[ https://issues.apache.org/jira/browse/HIVE-18318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299472#comment-16299472 ] Sergey Shelukhin commented on HIVE-18318: - The operator stuff is not really an issue, it's more like a semi deliberate design omission. Abort flag is only checked in some init methods; no operator has a mechanism to check it anywhere, so it's not like there are some problematic operators. > LLAP record reader should check interrupt even when not blocking > > > Key: HIVE-18318 > URL: https://issues.apache.org/jira/browse/HIVE-18318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18318.patch > > > Hive operators don't check interrupts, and may not do blocking operations. > LLAP record reader only blocks in IO is slower than processing; so, if IO is > fast enough, it will not ever block (at least not interruptibly, the sync > w/IO on the object does not check interrupts), and thus never catch > interrupts. > So, the task would be impossible to terminate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18257) implement scheduling policy configuration instead of hardcoding fair scheduling
[ https://issues.apache.org/jira/browse/HIVE-18257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18257: Attachment: HIVE-18257.03.patch Rebased and added the missing generated file. > implement scheduling policy configuration instead of hardcoding fair > scheduling > --- > > Key: HIVE-18257 > URL: https://issues.apache.org/jira/browse/HIVE-18257 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18257.01.patch, HIVE-18257.02.patch, > HIVE-18257.02.patch, HIVE-18257.03.patch, HIVE-18257.patch > > > Not sure it makes sense to actually make it pluggable. At least the standard > ones will be an enum; we don't expect people to implement custom classes - > phase 2 if someone wants to -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[
https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299470#comment-16299470
]
Hive QA commented on HIVE-18190:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903113/HIVE-18190.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 11535 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestAcidUtils.testParsing (batchId=266)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8344/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8344/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8344/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903113 - PreCommit-HIVE-Build
> Consider looking at ORC file schema rather than using _metadata_acid file
> -
>
> Key: HIVE-18190
> URL: https://issues.apache.org/jira/browse/HIVE-18190
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch,
> HIVE-18190.04.patch
>
>
> See if it's possible to just look at the schema of the file in base_ or
> delta_ to see if it has Acid metadata columns. If not, it's an 'original'
> file and needs ROW_IDs generated.
> see more discussion at https://reviews.apache.org/r/64131/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18078) WM getSession needs some retry logic
[ https://issues.apache.org/jira/browse/HIVE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18078: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > WM getSession needs some retry logic > > > Key: HIVE-18078 > URL: https://issues.apache.org/jira/browse/HIVE-18078 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18078.01.patch, HIVE-18078.01.patch, > HIVE-18078.04.patch, HIVE-18078.05.patch, HIVE-18078.05.patch, > HIVE-18078.only.patch, HIVE-18078.patch > > > When we get a bad session (e.g. no registry info because AM has gone > catatonic), the failure by the timeout future fails the getSession call. > The retry model in TezTask is that it would get a session (which in original > model can be completely unusable, but we still get the object), and then > retry (reopen) if it's a lemon. If the reopen fails, we fail. > getSession is not covered by this retry scheme, and should thus do its own > retries (or the retry logic needs to be changed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18095) add a unmanaged flag to triggers (applies to container based sessions)
[ https://issues.apache.org/jira/browse/HIVE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299452#comment-16299452 ] Sergey Shelukhin commented on HIVE-18095: - Cannot update RB because rbt post gets some git API error, whatever that means. The only major changes are the out file. > add a unmanaged flag to triggers (applies to container based sessions) > -- > > Key: HIVE-18095 > URL: https://issues.apache.org/jira/browse/HIVE-18095 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18095.01.patch, HIVE-18095.02.patch, > HIVE-18095.nogen.patch, HIVE-18095.patch > > > cc [~prasanth_j] > It should be impossible to attach global triggers for pools. Setting global > flag should probably automatically remove attachments to pools. > Global triggers would only support actions that Tez supports (for simplicity; > also, for now, move doesn't make a lot of sense because the trigger would > apply again after the move). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18095) add a unmanaged flag to triggers (applies to container based sessions)
[ https://issues.apache.org/jira/browse/HIVE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18095: Attachment: HIVE-18095.02.patch Rebased and updated sysdb > add a unmanaged flag to triggers (applies to container based sessions) > -- > > Key: HIVE-18095 > URL: https://issues.apache.org/jira/browse/HIVE-18095 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18095.01.patch, HIVE-18095.02.patch, > HIVE-18095.nogen.patch, HIVE-18095.patch > > > cc [~prasanth_j] > It should be impossible to attach global triggers for pools. Setting global > flag should probably automatically remove attachments to pools. > Global triggers would only support actions that Tez supports (for simplicity; > also, for now, move doesn't make a lot of sense because the trigger would > apply again after the move). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18230) create plan like plan, and replace plan commands for easy modification
[ https://issues.apache.org/jira/browse/HIVE-18230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18230: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the reviews! > create plan like plan, and replace plan commands for easy modification > -- > > Key: HIVE-18230 > URL: https://issues.apache.org/jira/browse/HIVE-18230 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18230.01.patch, HIVE-18230.only.nogen.patch, > HIVE-18230.patch > > > Given that the plan already on the cluster cannot be altered, it would be > helpful to have create plan like plan, and replace plan commands that would > make a copy to be modified, and then rename+apply the copy in place of an > existing plan, and rename the existing active plan with a versioned name or > drop it altogether. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18203) change the way WM is enabled and allow dropping the last resource plan
[ https://issues.apache.org/jira/browse/HIVE-18203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18203: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. I was not able to repro some test failures, I think we'd have to look at timeouts separately. Thanks for the reviews! > change the way WM is enabled and allow dropping the last resource plan > -- > > Key: HIVE-18203 > URL: https://issues.apache.org/jira/browse/HIVE-18203 > Project: Hive > Issue Type: Sub-task >Reporter: Aswathy Chellammal Sreekumar >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18203.01.patch, HIVE-18203.02.patch, > HIVE-18203.03.patch, HIVE-18203.04.patch, HIVE-18203.patch > > > Currently it's impossible to drop the last active resource plan even if WM is > disabled. It should be possible to deactivate the last resource plan AND > disable WM in the same action. Activating a resource plan should enable WM in > this case. > This should interact with the WM queue config in a sensible manner. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[
https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299416#comment-16299416
]
Hive QA commented on HIVE-18190:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 462 unchanged - 0
fixed = 463 total (was 462) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0ecf649 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8344/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8344/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Consider looking at ORC file schema rather than using _metadata_acid file
> -
>
> Key: HIVE-18190
> URL: https://issues.apache.org/jira/browse/HIVE-18190
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch,
> HIVE-18190.04.patch
>
>
> See if it's possible to just look at the schema of the file in base_ or
> delta_ to see if it has Acid metadata columns. If not, it's an 'original'
> file and needs ROW_IDs generated.
> see more discussion at https://reviews.apache.org/r/64131/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-17755) NPE exception when running TestAcidOnTez#testGetSplitsLocks with "
[ https://issues.apache.org/jira/browse/HIVE-17755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang resolved HIVE-17755. --- Resolution: Not A Problem > NPE exception when running TestAcidOnTez#testGetSplitsLocks with " > -- > > Key: HIVE-17755 > URL: https://issues.apache.org/jira/browse/HIVE-17755 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299400#comment-16299400
]
Eugene Koifman commented on HIVE-18315:
---
no related failures
[~jdere] could you review please
> update tests use non-acid tables
>
>
> Key: HIVE-18315
> URL: https://issues.apache.org/jira/browse/HIVE-18315
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18315.01.patch, HIVE-18315.02.patch,
> HIVE-18315.03.patch
>
>
> some tests like TestTxnLoadData need to create create non-acid table so that
> non-acid to acid conversion can be tested so they need explicit
> tblproperties('transactional'='false').
> HCat doesn't support acid so the tests need to use non-acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18316) HiveEndPoint should only work with full acid tables
[ https://issues.apache.org/jira/browse/HIVE-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299397#comment-16299397 ] Eugene Koifman commented on HIVE-18316: --- HiveEndPoint is what represents what partition of a table you create a connection to when doing Streaming Ingest > HiveEndPoint should only work with full acid tables > --- > > Key: HIVE-18316 > URL: https://issues.apache.org/jira/browse/HIVE-18316 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18316.01.patch, HIVE-18316.02.patch > > > now that we have full acid and 1/4 acid the check needs to be updated to > check for full acid tables -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299388#comment-16299388
]
Hive QA commented on HIVE-18315:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903106/HIVE-18315.03.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11139 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[jira] [Commented] (HIVE-18316) HiveEndPoint should only work with full acid tables
[ https://issues.apache.org/jira/browse/HIVE-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299389#comment-16299389 ] Sergey Shelukhin commented on HIVE-18316: - Not sure what this does. What is HiveEndPoint? +1 for implementation :) > HiveEndPoint should only work with full acid tables > --- > > Key: HIVE-18316 > URL: https://issues.apache.org/jira/browse/HIVE-18316 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18316.01.patch, HIVE-18316.02.patch > > > now that we have full acid and 1/4 acid the check needs to be updated to > check for full acid tables -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18148) NPE in SparkDynamicPartitionPruningResolver
[
https://issues.apache.org/jira/browse/HIVE-18148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299384#comment-16299384
]
liyunzhang commented on HIVE-18148:
---
+1
> NPE in SparkDynamicPartitionPruningResolver
> ---
>
> Key: HIVE-18148
> URL: https://issues.apache.org/jira/browse/HIVE-18148
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-18148.1.patch, HIVE-18148.2.patch
>
>
> The stack trace is:
> {noformat}
> 2017-11-27T10:32:38,752 ERROR [e6c8aab5-ddd2-461d-b185-a7597c3e7519 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver$SparkDynamicPartitionPruningDispatcher.dispatch(SparkDynamicPartitionPruningResolver.java:100)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver.resolve(SparkDynamicPartitionPruningResolver.java:74)
> at
> org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeTaskPlan(SparkCompiler.java:568)
> {noformat}
> At this stage, there shouldn't be a DPP sink whose target map work is null.
> The root cause seems to be a malformed operator tree generated by
> SplitOpTreeForDPP.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299373#comment-16299373
]
Misha Dmitriev commented on HIVE-17684:
---
[~stakiar] How do I run these {{TestSparkCliDriver}} tests? Note the "Spark"
thing - looks like they are different from TestCliDriver tests that I know how
to run. Also in the test report all these test names look the same.
In the mean time, I've just tried the first two of the failed {{TestCliDriver}}
tests locally. The first one passed for me. The second one keeps failing, with
the error below. This looks confusing. Note that the error message mentions
"exhausted memory". However, I cannot find any exception stack traces in the
hive.log file. Please advise.
{code}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hive.cli.TestCliDriver
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 102.789
s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestCliDriver
[ERROR]
testCliDriver[auto_join_without_localtask](org.apache.hadoop.hive.cli.TestCliDriver)
Time elapsed: 21.46 s <<< FAILURE!
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing auto_join_without_localtask.q
1047a1048,1053
> Hive Runtime Error: Map local work exhausted memory
> FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
> ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> Hive Runtime Error: Map local work exhausted memory
> FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
> ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
1054c1060
< RUN: Stage-9:MAPRED
---
> RUN: Stage-1:MAPRED
1057c1063
< RUN: Stage-6:MAPRED
---
> RUN: Stage-2:MAPRED
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hive.ql.QTestUtil.failedDiff(QTestUtil.java:2244)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:183)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
[jira] [Commented] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299346#comment-16299346
]
Hive QA commented on HIVE-18315:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0ecf649 |
| Default Java | 1.8.0_111 |
| modules | C: ql hcatalog/core hcatalog/hcatalog-pig-adapter
hcatalog/streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8343/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> update tests use non-acid tables
>
>
> Key: HIVE-18315
> URL: https://issues.apache.org/jira/browse/HIVE-18315
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18315.01.patch, HIVE-18315.02.patch,
> HIVE-18315.03.patch
>
>
> some tests like TestTxnLoadData need to create create non-acid table so that
> non-acid to acid conversion can be tested so they need explicit
> tblproperties('transactional'='false').
> HCat doesn't support acid so the tests need to use non-acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18203) change the way WM is enabled and allow dropping the last resource plan
[ https://issues.apache.org/jira/browse/HIVE-18203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299340#comment-16299340 ] Prasanth Jayachandran commented on HIVE-18203: -- +1 > change the way WM is enabled and allow dropping the last resource plan > -- > > Key: HIVE-18203 > URL: https://issues.apache.org/jira/browse/HIVE-18203 > Project: Hive > Issue Type: Sub-task >Reporter: Aswathy Chellammal Sreekumar >Assignee: Sergey Shelukhin > Attachments: HIVE-18203.01.patch, HIVE-18203.02.patch, > HIVE-18203.03.patch, HIVE-18203.04.patch, HIVE-18203.patch > > > Currently it's impossible to drop the last active resource plan even if WM is > disabled. It should be possible to deactivate the last resource plan AND > disable WM in the same action. Activating a resource plan should enable WM in > this case. > This should interact with the WM queue config in a sensible manner. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18230) create plan like plan, and replace plan commands for easy modification
[ https://issues.apache.org/jira/browse/HIVE-18230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299343#comment-16299343 ] Prasanth Jayachandran commented on HIVE-18230: -- +1 > create plan like plan, and replace plan commands for easy modification > -- > > Key: HIVE-18230 > URL: https://issues.apache.org/jira/browse/HIVE-18230 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18230.01.patch, HIVE-18230.only.nogen.patch, > HIVE-18230.patch > > > Given that the plan already on the cluster cannot be altered, it would be > helpful to have create plan like plan, and replace plan commands that would > make a copy to be modified, and then rename+apply the copy in place of an > existing plan, and rename the existing active plan with a versioned name or > drop it altogether. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18323:
Summary: Vectorization: add the support of timestamp in
VectorizedPrimitiveColumnReader for parquet (was: Vectorization: add the
support of timestamp in VectorizedPrimitiveColumnReader)
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-18323:
---
Assignee: Aihua Xu
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18316) HiveEndPoint should only work with full acid tables
[ https://issues.apache.org/jira/browse/HIVE-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299323#comment-16299323 ] Eugene Koifman commented on HIVE-18316: --- no related failures [~sershe] could you review please > HiveEndPoint should only work with full acid tables > --- > > Key: HIVE-18316 > URL: https://issues.apache.org/jira/browse/HIVE-18316 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18316.01.patch, HIVE-18316.02.patch > > > now that we have full acid and 1/4 acid the check needs to be updated to > check for full acid tables -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18053) Support different table types for MVs
[
https://issues.apache.org/jira/browse/HIVE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299318#comment-16299318
]
Hive QA commented on HIVE-18053:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903104/HIVE-18053.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11536 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalTasks
(batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8342/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8342/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8342/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903104 - PreCommit-HIVE-Build
> Support different table types for MVs
> -
>
> Key: HIVE-18053
> URL: https://issues.apache.org/jira/browse/HIVE-18053
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-18053.patch
>
>
> MVs backed by MM tables, managed tables, external tables. This might work
> already, but we need to add tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work
[ https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299304#comment-16299304 ] Yongzhi Chen commented on HIVE-11609: - +1 for me. [~aihuaxu], could you review the new patch > Capability to add a filter to hbase scan via composite key doesn't work > --- > > Key: HIVE-11609 > URL: https://issues.apache.org/jira/browse/HIVE-11609 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Swarnim Kulkarni >Assignee: Barna Zsombor Klara > Attachments: HIVE-11609.08.patch, HIVE-11609.09.patch, > HIVE-11609.1.patch.txt, HIVE-11609.10.patch, HIVE-11609.2.patch.txt, > HIVE-11609.3.patch.txt, HIVE-11609.4.patch.txt, HIVE-11609.5.patch, > HIVE-11609.6.patch.txt, HIVE-11609.7.patch.txt > > > It seems like the capability to add filter to an hbase scan which was added > as part of HIVE-6411 doesn't work. This is primarily because in the > HiveHBaseInputFormat, the filter is added in the getsplits instead of > getrecordreader. This works fine for start and stop keys but not for filter > because a filter is respected only when an actual scan is performed. This is > also related to the initial refactoring that was done as part of HIVE-3420. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work
[ https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen reassigned HIVE-11609: --- Assignee: Barna Zsombor Klara (was: Yongzhi Chen) > Capability to add a filter to hbase scan via composite key doesn't work > --- > > Key: HIVE-11609 > URL: https://issues.apache.org/jira/browse/HIVE-11609 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Swarnim Kulkarni >Assignee: Barna Zsombor Klara > Attachments: HIVE-11609.08.patch, HIVE-11609.09.patch, > HIVE-11609.1.patch.txt, HIVE-11609.10.patch, HIVE-11609.2.patch.txt, > HIVE-11609.3.patch.txt, HIVE-11609.4.patch.txt, HIVE-11609.5.patch, > HIVE-11609.6.patch.txt, HIVE-11609.7.patch.txt > > > It seems like the capability to add filter to an hbase scan which was added > as part of HIVE-6411 doesn't work. This is primarily because in the > HiveHBaseInputFormat, the filter is added in the getsplits instead of > getrecordreader. This works fine for start and stop keys but not for filter > because a filter is respected only when an actual scan is performed. This is > also related to the initial refactoring that was done as part of HIVE-3420. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work
[ https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-11609: Attachment: HIVE-11609.10.patch Rebase the patch and remove the part for existing filter > Capability to add a filter to hbase scan via composite key doesn't work > --- > > Key: HIVE-11609 > URL: https://issues.apache.org/jira/browse/HIVE-11609 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Swarnim Kulkarni >Assignee: Yongzhi Chen > Attachments: HIVE-11609.08.patch, HIVE-11609.09.patch, > HIVE-11609.1.patch.txt, HIVE-11609.10.patch, HIVE-11609.2.patch.txt, > HIVE-11609.3.patch.txt, HIVE-11609.4.patch.txt, HIVE-11609.5.patch, > HIVE-11609.6.patch.txt, HIVE-11609.7.patch.txt > > > It seems like the capability to add filter to an hbase scan which was added > as part of HIVE-6411 doesn't work. This is primarily because in the > HiveHBaseInputFormat, the filter is added in the getsplits instead of > getrecordreader. This works fine for start and stop keys but not for filter > because a filter is respected only when an actual scan is performed. This is > also related to the initial refactoring that was done as part of HIVE-3420. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work
[ https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen reassigned HIVE-11609: --- Assignee: Yongzhi Chen (was: Barna Zsombor Klara) > Capability to add a filter to hbase scan via composite key doesn't work > --- > > Key: HIVE-11609 > URL: https://issues.apache.org/jira/browse/HIVE-11609 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Swarnim Kulkarni >Assignee: Yongzhi Chen > Attachments: HIVE-11609.08.patch, HIVE-11609.09.patch, > HIVE-11609.1.patch.txt, HIVE-11609.2.patch.txt, HIVE-11609.3.patch.txt, > HIVE-11609.4.patch.txt, HIVE-11609.5.patch, HIVE-11609.6.patch.txt, > HIVE-11609.7.patch.txt > > > It seems like the capability to add filter to an hbase scan which was added > as part of HIVE-6411 doesn't work. This is primarily because in the > HiveHBaseInputFormat, the filter is added in the getsplits instead of > getrecordreader. This works fine for start and stop keys but not for filter > because a filter is respected only when an actual scan is performed. This is > also related to the initial refactoring that was done as part of HIVE-3420. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18053) Support different table types for MVs
[
https://issues.apache.org/jira/browse/HIVE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299261#comment-16299261
]
Hive QA commented on HIVE-18053:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 10 new + 1197 unchanged - 6
fixed = 1207 total (was 1203) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0ecf649 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8342/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8342/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support different table types for MVs
> -
>
> Key: HIVE-18053
> URL: https://issues.apache.org/jira/browse/HIVE-18053
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-18053.patch
>
>
> MVs backed by MM tables, managed tables, external tables. This might work
> already, but we need to add tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[ https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-16821: --- Attachment: (was: HIVE-16821.8.patch) > Vectorization: support Explain Analyze in vectorized mode > - > > Key: HIVE-16821 > URL: https://issues.apache.org/jira/browse/HIVE-16821 > Project: Hive > Issue Type: Bug > Components: Diagnosability, Vectorization >Affects Versions: 2.1.1, 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-16821.1.patch, HIVE-16821.2.patch, > HIVE-16821.2.patch, HIVE-16821.3.patch, HIVE-16821.7.patch, HIVE-16821.8.patch > > > Currently, to avoid a branch in the operator inner loop - the runtime stats > are only available in non-vector mode. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[ https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-16821: --- Attachment: HIVE-16821.8.patch > Vectorization: support Explain Analyze in vectorized mode > - > > Key: HIVE-16821 > URL: https://issues.apache.org/jira/browse/HIVE-16821 > Project: Hive > Issue Type: Bug > Components: Diagnosability, Vectorization >Affects Versions: 2.1.1, 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-16821.1.patch, HIVE-16821.2.patch, > HIVE-16821.2.patch, HIVE-16821.3.patch, HIVE-16821.7.patch, HIVE-16821.8.patch > > > Currently, to avoid a branch in the operator inner loop - the runtime stats > are only available in non-vector mode. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[ https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-16821: --- Attachment: HIVE-16821.8.patch > Vectorization: support Explain Analyze in vectorized mode > - > > Key: HIVE-16821 > URL: https://issues.apache.org/jira/browse/HIVE-16821 > Project: Hive > Issue Type: Bug > Components: Diagnosability, Vectorization >Affects Versions: 2.1.1, 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-16821.1.patch, HIVE-16821.2.patch, > HIVE-16821.2.patch, HIVE-16821.3.patch, HIVE-16821.7.patch, HIVE-16821.8.patch > > > Currently, to avoid a branch in the operator inner loop - the runtime stats > are only available in non-vector mode. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299234#comment-16299234
]
Aihua Xu commented on HIVE-17684:
-
This FileNotFoundException issue should be related to hadoop change (see
HADOOP-15100). Hadoop common doesn't replace the system variables.
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18316) HiveEndPoint should only work with full acid tables
[
https://issues.apache.org/jira/browse/HIVE-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299233#comment-16299233
]
Hive QA commented on HIVE-18316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903096/HIVE-18316.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11535 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanQpChanges
(batchId=284)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenamePartitionWithCM
(batchId=225)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8341/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8341/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8341/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903096 - PreCommit-HIVE-Build
> HiveEndPoint should only work with full acid tables
> ---
>
> Key: HIVE-18316
> URL: https://issues.apache.org/jira/browse/HIVE-18316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18316.01.patch, HIVE-18316.02.patch
>
>
> now that we have full acid and 1/4 acid the check needs to be updated to
> check for full acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18317) Improve error messages in TransactionalValidationListerner
[ https://issues.apache.org/jira/browse/HIVE-18317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18317: -- Attachment: HIVE-18317.03.patch > Improve error messages in TransactionalValidationListerner > -- > > Key: HIVE-18317 > URL: https://issues.apache.org/jira/browse/HIVE-18317 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18317.01.patch, HIVE-18317.02.patch, > HIVE-18317.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18316) HiveEndPoint should only work with full acid tables
[
https://issues.apache.org/jira/browse/HIVE-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299187#comment-16299187
]
Hive QA commented on HIVE-18316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} ql: The patch generated 2 new + 117 unchanged - 0
fixed = 119 total (was 117) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 124
unchanged - 3 fixed = 125 total (was 127) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0ecf649 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8341/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8341/yetus/diff-checkstyle-hcatalog_streaming.txt
|
| modules | C: ql hcatalog/streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8341/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveEndPoint should only work with full acid tables
> ---
>
> Key: HIVE-18316
> URL: https://issues.apache.org/jira/browse/HIVE-18316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18316.01.patch, HIVE-18316.02.patch
>
>
> now that we have full acid and 1/4 acid the check needs to be updated to
> check for full acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18320) Support ACID Tables Replication
[ https://issues.apache.org/jira/browse/HIVE-18320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18320: --- Target Version/s: 3.0.0 Fix Version/s: (was: 3.0.0) > Support ACID Tables Replication > --- > > Key: HIVE-18320 > URL: https://issues.apache.org/jira/browse/HIVE-18320 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, Metastore, repl, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: anishek > Labels: ACID, DR, replication > > Currently, Full ACID and MM (Micro-Managed) tables are not supported for > replication. > Need to support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18317) Improve error messages in TransactionalValidationListerner
[ https://issues.apache.org/jira/browse/HIVE-18317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18317: -- Summary: Improve error messages in TransactionalValidationListerner (was: Improve error messages in TransactionValidationListerner) > Improve error messages in TransactionalValidationListerner > -- > > Key: HIVE-18317 > URL: https://issues.apache.org/jira/browse/HIVE-18317 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18317.01.patch, HIVE-18317.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15393) Update Guava version
[ https://issues.apache.org/jira/browse/HIVE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-15393: -- Attachment: HIVE-15393.9.patch Reverting to 19 since Spark client tests still failing. Will move to 21 in a follow up PR > Update Guava version > > > Key: HIVE-15393 > URL: https://issues.apache.org/jira/browse/HIVE-15393 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-15393.2.patch, HIVE-15393.3.patch, > HIVE-15393.5.patch, HIVE-15393.6.patch, HIVE-15393.7.patch, > HIVE-15393.8.patch, HIVE-15393.9.patch, HIVE-15393.patch > > > Druid base code is using newer version of guava 16.0.1 that is not compatible > with the current version used by Hive. > FYI Hadoop project is moving to Guava 18 not sure if it is better to move to > guava 18 or even 19. > https://issues.apache.org/jira/browse/HADOOP-10101 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18031) Support replication for Alter Database operation.
[
https://issues.apache.org/jira/browse/HIVE-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299164#comment-16299164
]
Hive QA commented on HIVE-18031:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903093/HIVE-18031.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11536 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8340/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8340/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8340/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903093 - PreCommit-HIVE-Build
> Support replication for Alter Database operation.
> -
>
> Key: HIVE-18031
> URL: https://issues.apache.org/jira/browse/HIVE-18031
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18031.01.patch, HIVE-18031.02.patch
>
>
> Currently alter database operations to alter the database properties or
> description are not generating any events due to which it is not getting
> replicated.
> Need to add an event for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-18293:
-
Assignee: Johannes Alberti (was: Eugene Koifman)
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Johannes Alberti
>Priority: Critical
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread$1.run(CompactorThread.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.findUserToRunAs(CompactorThread.java:169)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:151)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=hive,
[jira] [Updated] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18293:
--
Status: Patch Available (was: Open)
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread$1.run(CompactorThread.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.findUserToRunAs(CompactorThread.java:169)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:151)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=hive, access=EXECUTE,
>
[jira] [Updated] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18293:
--
Attachment: HIVE-18293.01.patch
reattaching the same patch - build bot was restarted and lost the original one
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread$1.run(CompactorThread.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.findUserToRunAs(CompactorThread.java:169)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:151)
> Caused by:
>
[jira] [Assigned] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-18293:
-
Assignee: Eugene Koifman (was: Johannes Alberti)
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread$1.run(CompactorThread.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.findUserToRunAs(CompactorThread.java:169)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:151)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=hive,
[jira] [Updated] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18293:
--
Status: Open (was: Patch Available)
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Johannes Alberti
>Priority: Critical
> Attachments: HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread$1.run(CompactorThread.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.findUserToRunAs(CompactorThread.java:169)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:151)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=hive, access=EXECUTE,
>
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17994: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17994: Fix Version/s: 3.0.0 > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299154#comment-16299154 ] Matt McCline commented on HIVE-17994: - Committed to master. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[ https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18190: -- Attachment: HIVE-18190.04.patch > Consider looking at ORC file schema rather than using _metadata_acid file > - > > Key: HIVE-18190 > URL: https://issues.apache.org/jira/browse/HIVE-18190 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch, > HIVE-18190.04.patch > > > See if it's possible to just look at the schema of the file in base_ or > delta_ to see if it has Acid metadata columns. If not, it's an 'original' > file and needs ROW_IDs generated. > see more discussion at https://reviews.apache.org/r/64131/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299153#comment-16299153 ] Matt McCline commented on HIVE-17994: - Thank you [~teddy.choi] for your review and [~gopalv] for your help. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18031) Support replication for Alter Database operation.
[
https://issues.apache.org/jira/browse/HIVE-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299113#comment-16299113
]
Hive QA commented on HIVE-18031:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore: The patch generated 11 new + 795
unchanged - 9 fixed = 806 total (was 804) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} ql: The patch generated 0 new + 540 unchanged - 1
fixed = 540 total (was 541) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch server-extensions passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 60b8504 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8340/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore ql hcatalog/server-extensions
itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8340/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support replication for Alter Database operation.
> -
>
> Key: HIVE-18031
> URL: https://issues.apache.org/jira/browse/HIVE-18031
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18031.01.patch, HIVE-18031.02.patch
>
>
> Currently alter database operations to alter the database properties or
> description are not generating any events due to which it is not getting
> replicated.
> Need to add an event for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17853) RetryingMetaStoreClient loses UGI impersonation-context when reconnecting after timeout
[
https://issues.apache.org/jira/browse/HIVE-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299079#comment-16299079
]
Thejas M Nair commented on HIVE-17853:
--
Created HIVE-18322.
[~cdrome]
Do you think you will be able to take a stab at it ?
> RetryingMetaStoreClient loses UGI impersonation-context when reconnecting
> after timeout
> ---
>
> Key: HIVE-17853
> URL: https://issues.apache.org/jira/browse/HIVE-17853
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 2.4.0, 2.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>Priority: Critical
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-17853.01-branch-2.patch, HIVE-17853.01.patch
>
>
> The {{RetryingMetaStoreClient}} is used to automatically reconnect to the
> Hive metastore, after client timeout, transparently to the user.
> In case of user impersonation (e.g. Oozie super-user {{oozie}} impersonating
> a Hadoop user {{mithun}}, to run a workflow), in case of timeout, we find
> that the reconnect causes the {{UGI.doAs()}} context to be lost. Any further
> metastore operations will be attempted as the login-user ({{oozie}}), as
> opposed to the effective user ({{mithun}}).
> We should have a fix for this shortly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17853) RetryingMetaStoreClient loses UGI impersonation-context when reconnecting after timeout
[
https://issues.apache.org/jira/browse/HIVE-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299065#comment-16299065
]
Thejas M Nair edited comment on HIVE-17853 at 12/20/17 8:48 PM:
We should also check to see if current user is same as the original UGI user,
and not do the ugi.doAs() if it is the same. Otherwise, this can potentially
cause problems where the users are not privileged users (ie, there is no intent
to do a "doAs").
Without such a check, you would get errors like " userX is not allowed to
impersonate userX".
was (Author: thejas):
We should also check to see if current user is same as the original UGI user,
and not do the ugi.doAs() if it is the same. Otherwise, this can potentially
cause problems where the users are not privileged users (ie, there is no intent
to do a "doAs").
You would get errors like " userX is not allowed to impersonate userX".
> RetryingMetaStoreClient loses UGI impersonation-context when reconnecting
> after timeout
> ---
>
> Key: HIVE-17853
> URL: https://issues.apache.org/jira/browse/HIVE-17853
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 2.4.0, 2.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>Priority: Critical
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-17853.01-branch-2.patch, HIVE-17853.01.patch
>
>
> The {{RetryingMetaStoreClient}} is used to automatically reconnect to the
> Hive metastore, after client timeout, transparently to the user.
> In case of user impersonation (e.g. Oozie super-user {{oozie}} impersonating
> a Hadoop user {{mithun}}, to run a workflow), in case of timeout, we find
> that the reconnect causes the {{UGI.doAs()}} context to be lost. Any further
> metastore operations will be attempted as the login-user ({{oozie}}), as
> opposed to the effective user ({{mithun}}).
> We should have a fix for this shortly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17853) RetryingMetaStoreClient loses UGI impersonation-context when reconnecting after timeout
[
https://issues.apache.org/jira/browse/HIVE-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299065#comment-16299065
]
Thejas M Nair commented on HIVE-17853:
--
We should also check to see if current user is same as the original UGI user,
and not do the ugi.doAs() if it is the same. Otherwise, this can potentially
cause problems where the users are not privileged users (ie, there is no intent
to do a "doAs").
You would get errors like " userX is not allowed to impersonate userX".
> RetryingMetaStoreClient loses UGI impersonation-context when reconnecting
> after timeout
> ---
>
> Key: HIVE-17853
> URL: https://issues.apache.org/jira/browse/HIVE-17853
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 2.4.0, 2.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>Priority: Critical
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-17853.01-branch-2.patch, HIVE-17853.01.patch
>
>
> The {{RetryingMetaStoreClient}} is used to automatically reconnect to the
> Hive metastore, after client timeout, transparently to the user.
> In case of user impersonation (e.g. Oozie super-user {{oozie}} impersonating
> a Hadoop user {{mithun}}, to run a workflow), in case of timeout, we find
> that the reconnect causes the {{UGI.doAs()}} context to be lost. Any further
> metastore operations will be attempted as the login-user ({{oozie}}), as
> opposed to the effective user ({{mithun}}).
> We should have a fix for this shortly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[
https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299064#comment-16299064
]
Hive QA commented on HIVE-18190:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903086/HIVE-18190.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 11536 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_buckets] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_exchangepartition]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=223)
org.apache.hadoop.hive.ql.TestTxnCommands2.testMmTableCompaction (batchId=278)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMmTableCompaction
(batchId=288)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForMmTable
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithUnionAll
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testMmTableCompaction
(batchId=257)
org.apache.hadoop.hive.ql.io.TestAcidUtils.testParsing (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8339/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8339/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8339/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903086 - PreCommit-HIVE-Build
> Consider looking at ORC file schema rather than using _metadata_acid file
> -
>
> Key: HIVE-18190
> URL: https://issues.apache.org/jira/browse/HIVE-18190
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch
>
>
> See if it's possible to just look at the schema of the file in base_ or
> delta_ to see if it has Acid metadata columns. If not, it's an 'original'
> file and needs ROW_IDs generated.
> see more discussion at https://reviews.apache.org/r/64131/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299027#comment-16299027 ] Eugene Koifman edited comment on HIVE-18192 at 12/20/17 8:41 PM: - [~anishek] will share the doc that includes explanation of write ids. Snapshot Isolation will of course be preserved. was (Author: ekoifman): I'll try to share the doc by end of the week. Snapshot Isolation will of course be preserved. > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Labels: ACID, DR > Fix For: 3.0.0 > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > a persistable map of global txn id -> to table -> write id for that table has > to be maintained to now allow Snapshot isolation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18315:
--
Attachment: HIVE-18315.03.patch
> update tests use non-acid tables
>
>
> Key: HIVE-18315
> URL: https://issues.apache.org/jira/browse/HIVE-18315
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18315.01.patch, HIVE-18315.02.patch,
> HIVE-18315.03.patch
>
>
> some tests like TestTxnLoadData need to create create non-acid table so that
> non-acid to acid conversion can be tested so they need explicit
> tblproperties('transactional'='false').
> HCat doesn't support acid so the tests need to use non-acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18315:
--
Status: Patch Available (was: Open)
> update tests use non-acid tables
>
>
> Key: HIVE-18315
> URL: https://issues.apache.org/jira/browse/HIVE-18315
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18315.01.patch, HIVE-18315.02.patch,
> HIVE-18315.03.patch
>
>
> some tests like TestTxnLoadData need to create create non-acid table so that
> non-acid to acid conversion can be tested so they need explicit
> tblproperties('transactional'='false').
> HCat doesn't support acid so the tests need to use non-acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18315:
--
Status: Open (was: Patch Available)
> update tests use non-acid tables
>
>
> Key: HIVE-18315
> URL: https://issues.apache.org/jira/browse/HIVE-18315
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18315.01.patch, HIVE-18315.02.patch
>
>
> some tests like TestTxnLoadData need to create create non-acid table so that
> non-acid to acid conversion can be tested so they need explicit
> tblproperties('transactional'='false').
> HCat doesn't support acid so the tests need to use non-acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18321) Support REBUILD for MVs backed by custom storage handlers
[ https://issues.apache.org/jira/browse/HIVE-18321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-18321: -- > Support REBUILD for MVs backed by custom storage handlers > - > > Key: HIVE-18321 > URL: https://issues.apache.org/jira/browse/HIVE-18321 > Project: Hive > Issue Type: Bug > Components: Druid integration, Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > For instance, currently REBUILD will fail for Druid backed MVs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18315:
--
Attachment: HIVE-18315.02.patch
> update tests use non-acid tables
>
>
> Key: HIVE-18315
> URL: https://issues.apache.org/jira/browse/HIVE-18315
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18315.01.patch, HIVE-18315.02.patch
>
>
> some tests like TestTxnLoadData need to create create non-acid table so that
> non-acid to acid conversion can be tested so they need explicit
> tblproperties('transactional'='false').
> HCat doesn't support acid so the tests need to use non-acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18315) update tests use non-acid tables
[
https://issues.apache.org/jira/browse/HIVE-18315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299046#comment-16299046
]
Eugene Koifman commented on HIVE-18315:
---
Fail intermittently:
ppd_join5, bucketsortoptimize_insert_7, auto_sortmerge_join_10,
bucket_map_join_tez1
> update tests use non-acid tables
>
>
> Key: HIVE-18315
> URL: https://issues.apache.org/jira/browse/HIVE-18315
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18315.01.patch, HIVE-18315.02.patch
>
>
> some tests like TestTxnLoadData need to create create non-acid table so that
> non-acid to acid conversion can be tested so they need explicit
> tblproperties('transactional'='false').
> HCat doesn't support acid so the tests need to use non-acid tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18053) Support different table types for MVs
[ https://issues.apache.org/jira/browse/HIVE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18053: --- Attachment: HIVE-18053.patch > Support different table types for MVs > - > > Key: HIVE-18053 > URL: https://issues.apache.org/jira/browse/HIVE-18053 > Project: Hive > Issue Type: Sub-task > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-18053.patch > > > MVs backed by MM tables, managed tables, external tables. This might work > already, but we need to add tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18053) Support different table types for MVs
[ https://issues.apache.org/jira/browse/HIVE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18053: --- Status: Patch Available (was: In Progress) > Support different table types for MVs > - > > Key: HIVE-18053 > URL: https://issues.apache.org/jira/browse/HIVE-18053 > Project: Hive > Issue Type: Sub-task > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-18053.patch > > > MVs backed by MM tables, managed tables, external tables. This might work > already, but we need to add tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299027#comment-16299027 ] Eugene Koifman commented on HIVE-18192: --- I'll try to share the doc by end of the week. Snapshot Isolation will of course be preserved. > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Labels: ACID, DR > Fix For: 3.0.0 > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > a persistable map of global txn id -> to table -> write id for that table has > to be maintained to now allow Snapshot isolation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18316) HiveEndPoint should only work with full acid tables
[ https://issues.apache.org/jira/browse/HIVE-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18316: -- Attachment: HIVE-18316.02.patch > HiveEndPoint should only work with full acid tables > --- > > Key: HIVE-18316 > URL: https://issues.apache.org/jira/browse/HIVE-18316 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18316.01.patch, HIVE-18316.02.patch > > > now that we have full acid and 1/4 acid the check needs to be updated to > check for full acid tables -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18031) Support replication for Alter Database operation.
[ https://issues.apache.org/jira/browse/HIVE-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299004#comment-16299004 ] Sankar Hariappan commented on HIVE-18031: - Request [~thejas]/[~anishek]/[~daijy] to please review the patch. > Support replication for Alter Database operation. > - > > Key: HIVE-18031 > URL: https://issues.apache.org/jira/browse/HIVE-18031 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-18031.01.patch, HIVE-18031.02.patch > > > Currently alter database operations to alter the database properties or > description are not generating any events due to which it is not getting > replicated. > Need to add an event for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18031) Support replication for Alter Database operation.
[ https://issues.apache.org/jira/browse/HIVE-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18031: Attachment: HIVE-18031.02.patch Attached 02.patch with fixes for check style issues. > Support replication for Alter Database operation. > - > > Key: HIVE-18031 > URL: https://issues.apache.org/jira/browse/HIVE-18031 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-18031.01.patch, HIVE-18031.02.patch > > > Currently alter database operations to alter the database properties or > description are not generating any events due to which it is not getting > replicated. > Need to add an event for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18031) Support replication for Alter Database operation.
[ https://issues.apache.org/jira/browse/HIVE-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18031: Status: Patch Available (was: Open) > Support replication for Alter Database operation. > - > > Key: HIVE-18031 > URL: https://issues.apache.org/jira/browse/HIVE-18031 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-18031.01.patch, HIVE-18031.02.patch > > > Currently alter database operations to alter the database properties or > description are not generating any events due to which it is not getting > replicated. > Need to add an event for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18031) Support replication for Alter Database operation.
[ https://issues.apache.org/jira/browse/HIVE-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18031: Status: Open (was: Patch Available) > Support replication for Alter Database operation. > - > > Key: HIVE-18031 > URL: https://issues.apache.org/jira/browse/HIVE-18031 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-18031.01.patch > > > Currently alter database operations to alter the database properties or > description are not generating any events due to which it is not getting > replicated. > Need to add an event for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18320) Support ACID Tables Replication
[ https://issues.apache.org/jira/browse/HIVE-18320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-18320: --- Assignee: anishek (was: Sankar Hariappan) > Support ACID Tables Replication > --- > > Key: HIVE-18320 > URL: https://issues.apache.org/jira/browse/HIVE-18320 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, Metastore, repl, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: anishek > Labels: ACID, DR, replication > Fix For: 3.0.0 > > > Currently, Full ACID and MM (Micro-Managed) tables are not supported for > replication. > Need to support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298997#comment-16298997 ] Andrew Sherman commented on HIVE-18192: --- Can someone share this design doc please? > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Labels: ACID, DR > Fix For: 3.0.0 > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > a persistable map of global txn id -> to table -> write id for that table has > to be maintained to now allow Snapshot isolation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[
https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298995#comment-16298995
]
Hive QA commented on HIVE-18190:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 462 unchanged - 0
fixed = 463 total (was 462) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 1c36983 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8339/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8339/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Consider looking at ORC file schema rather than using _metadata_acid file
> -
>
> Key: HIVE-18190
> URL: https://issues.apache.org/jira/browse/HIVE-18190
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch
>
>
> See if it's possible to just look at the schema of the file in base_ or
> delta_ to see if it has Acid metadata columns. If not, it's an 'original'
> file and needs ROW_IDs generated.
> see more discussion at https://reviews.apache.org/r/64131/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18192) Introduce WriteId per table rather than using global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Labels: ACID DR (was: ACID) > Introduce WriteId per table rather than using global transaction id > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Labels: ACID, DR > Fix For: 3.0.0 > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > a persistable map of global txn id -> to table -> write id for that table has > to be maintained to now allow Snapshot isolation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Summary: Introduce WriteID per table rather than using global transaction ID (was: Introduce WriteId per table rather than using global transaction id) > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Labels: ACID, DR > Fix For: 3.0.0 > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > a persistable map of global txn id -> to table -> write id for that table has > to be maintained to now allow Snapshot isolation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18192) Introduce WriteId per table rather than using global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-18192: --- Assignee: Sankar Hariappan (was: anishek) > Introduce WriteId per table rather than using global transaction id > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Labels: ACID > Fix For: 3.0.0 > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > a persistable map of global txn id -> to table -> write id for that table has > to be maintained to now allow Snapshot isolation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18192) Introduce WriteId per table rather than using global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Labels: ACID (was: ) > Introduce WriteId per table rather than using global transaction id > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan > Labels: ACID > Fix For: 3.0.0 > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > a persistable map of global txn id -> to table -> write id for that table has > to be maintained to now allow Snapshot isolation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18306) Fix spark smb tests
[
https://issues.apache.org/jira/browse/HIVE-18306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-18306:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master
> Fix spark smb tests
> ---
>
> Key: HIVE-18306
> URL: https://issues.apache.org/jira/browse/HIVE-18306
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Deepak Jaiswal
> Fix For: 3.0.0
>
> Attachments: HIVE-18306.1.patch, HIVE-18306.2.patch
>
>
> seems to me that
> {{TestSparkCliDriver#testCliDriver\[auto_sortmerge_join_10\]}} and
> {{TestSparkCliDriver#testCliDriver\[bucketsortoptimize_insert_7\]}} is
> failing since HIVE-18208 is in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18320) Support ACID Tables Replication
[ https://issues.apache.org/jira/browse/HIVE-18320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18320: Description: Currently, Full ACID and MM (Micro-Managed) tables are not supported for replication. Need to support it. was: Currently, Full ACID and MM (Micro-Managed) tables are not supported by Replv2. Need to support it. > Support ACID Tables Replication > --- > > Key: HIVE-18320 > URL: https://issues.apache.org/jira/browse/HIVE-18320 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, Metastore, repl, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: ACID, DR, replication > Fix For: 3.0.0 > > > Currently, Full ACID and MM (Micro-Managed) tables are not supported for > replication. > Need to support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
