[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Status: Patch Available (was: Open) > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Attachment: (was: HIVE-14014.04.patch) > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Status: Open (was: Patch Available) > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Attachment: HIVE-14014.04.patch > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Status: Open (was: Patch Available) > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Status: Patch Available (was: Open) > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[
https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339019#comment-15339019
]
Hive QA commented on HIVE-14054:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811723/HIVE-14054.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/185/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/185/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-185/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811723 - PreCommit-HIVE-MASTER-Build
> TestHiveMetaStoreChecker fails on master
> -
>
> Key: HIVE-14054
> URL: https://issues.apache.org/jira/browse/HIVE-14054
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Status: Open (was: Patch Available) > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Status: Patch Available (was: Open) > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[ https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14014: --- Attachment: HIVE-14014.04.patch > zero length file is being created for empty bucket in tez mode (II) > --- > > Key: HIVE-14014 > URL: https://issues.apache.org/jira/browse/HIVE-14014 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch, > HIVE-14014.03.patch, HIVE-14014.04.patch > > > The same problem happens when source table is not empty, e.g,, when "limit 0" > is not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Status: Patch Available (was: In Progress) > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch, > HIVE-14000.04.patch, HIVE-14000.05.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Attachment: HIVE-14000.05.patch > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch, > HIVE-14000.04.patch, HIVE-14000.05.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Status: In Progress (was: Patch Available) > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch, > HIVE-14000.04.patch, HIVE-14000.05.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338984#comment-15338984 ] Matt McCline commented on HIVE-14000: - Seem like test infrastructure problems: Unexpected exception in static initialization: Cannot remove data directory: /home/hiveptest/.. Trying again. > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch, > HIVE-14000.04.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14017) Compaction failed when run on ACID table with extended schema
[
https://issues.apache.org/jira/browse/HIVE-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338963#comment-15338963
]
Hong Dai Thanh edited comment on HIVE-14017 at 6/20/16 4:10 AM:
Added hive-site.xml configuration file
was (Author: nhahtdh):
hive-site.xml
> Compaction failed when run on ACID table with extended schema
> -
>
> Key: HIVE-14017
> URL: https://issues.apache.org/jira/browse/HIVE-14017
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
> Environment: HDP 2.4.0/Hive 1.2.1 on RHEL 6
>Reporter: Hong Dai Thanh
> Attachments: hive-site.xml
>
>
> Create an ACID table, insert some data into the table. Then we extend the
> schema of the table by adding a column at the end, and add data to the table
> with the extended schema.
> {code:borderStyle=solid}
> drop table if exists test purge;
> create table test (
> a int,
> b int
> )
> clustered by (a) into 10 buckets
> stored as orc
> tblproperties ('transactional' = 'true');
> insert into test values (1, 1), (2, 2), (3, 3);
> insert into test values (4, 4), (5, 5), (6, 6);
> alter table test add columns (c int);
> insert into test values (10, 10, 10), (11, 11, 11), (12, 12, 12);
> {code}
> We then run compaction on the table:
> {code}alter table test compact 'major';{code}
> However, the compaction job fails with the following exception:
> {code}
> 2016-06-15 09:54:52,517 INFO [IPC Server handler 5 on 25906]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1465960802609_0030_m_08_0 is : 0.0
> 2016-06-15 09:54:52,525 FATAL [IPC Server handler 4 on 25906]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1465960802609_0030_m_08_0 - exited : java.io.IOException: subtype
> 9 exceeds the included array size 9 fileTypes [kind: STRUCT
> subtypes: 1
> subtypes: 2
> subtypes: 3
> subtypes: 4
> subtypes: 5
> subtypes: 6
> fieldNames: "operation"
> fieldNames: "originalTransaction"
> fieldNames: "bucket"
> fieldNames: "rowId"
> fieldNames: "currentTransaction"
> fieldNames: "row"
> , kind: INT
> , kind: LONG
> , kind: INT
> , kind: LONG
> , kind: LONG
> , kind: STRUCT
> subtypes: 7
> subtypes: 8
> subtypes: 9
> fieldNames: "_col0"
> fieldNames: "_col1"
> fieldNames: "_col2"
> , kind: INT
> , kind: INT
> , kind: INT
> ] schemaTypes [kind: STRUCT
> subtypes: 1
> subtypes: 2
> subtypes: 3
> subtypes: 4
> subtypes: 5
> subtypes: 6
> fieldNames: "operation"
> fieldNames: "originalTransaction"
> fieldNames: "bucket"
> fieldNames: "rowId"
> fieldNames: "currentTransaction"
> fieldNames: "row"
> , kind: INT
> , kind: LONG
> , kind: INT
> , kind: LONG
> , kind: LONG
> , kind: STRUCT
> subtypes: 7
> subtypes: 8
> subtypes: 9
> fieldNames: "_col0"
> fieldNames: "_col1"
> fieldNames: "_col2"
> , kind: INT
> , kind: INT
> , kind: INT
> ] innerStructSubtype -1
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2066)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2492)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2072)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2492)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:219)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:179)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:476)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1463)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:573)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:552)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14017) Compaction failed when run on ACID table with extended schema
[
https://issues.apache.org/jira/browse/HIVE-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Dai Thanh updated HIVE-14017:
--
Attachment: hive-site.xml
hive-site.xml
> Compaction failed when run on ACID table with extended schema
> -
>
> Key: HIVE-14017
> URL: https://issues.apache.org/jira/browse/HIVE-14017
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
> Environment: HDP 2.4.0/Hive 1.2.1 on RHEL 6
>Reporter: Hong Dai Thanh
> Attachments: hive-site.xml
>
>
> Create an ACID table, insert some data into the table. Then we extend the
> schema of the table by adding a column at the end, and add data to the table
> with the extended schema.
> {code:borderStyle=solid}
> drop table if exists test purge;
> create table test (
> a int,
> b int
> )
> clustered by (a) into 10 buckets
> stored as orc
> tblproperties ('transactional' = 'true');
> insert into test values (1, 1), (2, 2), (3, 3);
> insert into test values (4, 4), (5, 5), (6, 6);
> alter table test add columns (c int);
> insert into test values (10, 10, 10), (11, 11, 11), (12, 12, 12);
> {code}
> We then run compaction on the table:
> {code}alter table test compact 'major';{code}
> However, the compaction job fails with the following exception:
> {code}
> 2016-06-15 09:54:52,517 INFO [IPC Server handler 5 on 25906]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1465960802609_0030_m_08_0 is : 0.0
> 2016-06-15 09:54:52,525 FATAL [IPC Server handler 4 on 25906]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1465960802609_0030_m_08_0 - exited : java.io.IOException: subtype
> 9 exceeds the included array size 9 fileTypes [kind: STRUCT
> subtypes: 1
> subtypes: 2
> subtypes: 3
> subtypes: 4
> subtypes: 5
> subtypes: 6
> fieldNames: "operation"
> fieldNames: "originalTransaction"
> fieldNames: "bucket"
> fieldNames: "rowId"
> fieldNames: "currentTransaction"
> fieldNames: "row"
> , kind: INT
> , kind: LONG
> , kind: INT
> , kind: LONG
> , kind: LONG
> , kind: STRUCT
> subtypes: 7
> subtypes: 8
> subtypes: 9
> fieldNames: "_col0"
> fieldNames: "_col1"
> fieldNames: "_col2"
> , kind: INT
> , kind: INT
> , kind: INT
> ] schemaTypes [kind: STRUCT
> subtypes: 1
> subtypes: 2
> subtypes: 3
> subtypes: 4
> subtypes: 5
> subtypes: 6
> fieldNames: "operation"
> fieldNames: "originalTransaction"
> fieldNames: "bucket"
> fieldNames: "rowId"
> fieldNames: "currentTransaction"
> fieldNames: "row"
> , kind: INT
> , kind: LONG
> , kind: INT
> , kind: LONG
> , kind: LONG
> , kind: STRUCT
> subtypes: 7
> subtypes: 8
> subtypes: 9
> fieldNames: "_col0"
> fieldNames: "_col1"
> fieldNames: "_col2"
> , kind: INT
> , kind: INT
> , kind: INT
> ] innerStructSubtype -1
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2066)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2492)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2072)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2492)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:219)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:179)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:476)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1463)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:573)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:552)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13872) Vectorization: Fix cross-product reduce sink serialization
[
https://issues.apache.org/jira/browse/HIVE-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338944#comment-15338944
]
Hive QA commented on HIVE-13872:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811717/HIVE-13872.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 10238 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mr_diff_schema_alias
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/184/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/184/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-184/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811717 - PreCommit-HIVE-MASTER-Build
> Vectorization: Fix cross-product reduce sink serialization
> --
>
> Key: HIVE-13872
> URL: https://issues.apache.org/jira/browse/HIVE-13872
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13872.01.patch, HIVE-13872.WIP.patch,
> customer_demographics.txt, vector_include_no_sel.q,
> vector_include_no_sel.q.out
>
>
> TPC-DS Q13 produces a cross-product without CBO simplifying the query
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> projection column num 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:349)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRow(VectorExtractRow.java:343)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:103)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> {code}
> Simplified query
> {code}
> set hive.cbo.enable=false;
> -- explain
> select count(1)
> from store_sales
> ,customer_demographics
> where (
> (
> customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'M'
> )or
> (
>
[jira] [Commented] (HIVE-9854) OutofMemory while read ORCFile table
[ https://issues.apache.org/jira/browse/HIVE-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338941#comment-15338941 ] bin wang commented on HIVE-9854: how to fix this? > OutofMemory while read ORCFile table > > > Key: HIVE-9854 > URL: https://issues.apache.org/jira/browse/HIVE-9854 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.13.1 >Reporter: Liao, Xiaoge > > Log: > Diagnostic Messages for this Task: > Error: java.io.IOException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:294) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:241) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:365) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:166) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:407) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:160) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:155) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:280) > ... 11 more > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.hive.ql.io.orc.DynamicByteArray.grow(DynamicByteArray.java:64) > at > org.apache.hadoop.hive.ql.io.orc.DynamicByteArray.readAll(DynamicByteArray.java:142) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.startStripe(RecordReaderImpl.java:1547) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.startStripe(RecordReaderImpl.java:1337) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.startStripe(RecordReaderImpl.java:1825) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2537) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2950) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2992) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:284) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:480) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:214) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.(OrcInputFormat.java:146) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:997) > at > org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:65) > ... 16 more > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Stage-Stage-1: Map: 105 Cumulative CPU: 656.39 sec HDFS Read: 4040094761 > HDFS Write: 139 FAIL > Total MapReduce CPU Time Spent: 10 minutes 56 seconds 390 msec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338896#comment-15338896 ] Takanobu Asanuma commented on HIVE-11527: - [~thejas] Thank you for your review! I will consider handling backward compatibility. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338890#comment-15338890 ] Ferdinand Xu commented on HIVE-14029: - OK, let us wait for GA release. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-14015:
Resolution: Fixed
Fix Version/s: 2.2.0
2.1.0
Status: Resolved (was: Patch Available)
The failures are not related.
Committed to master and branch-2.1
Thank [~ctang.ma] for reviewing the code.
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.1.0, 2.2.0
>
> Attachments: HIVE-14015.1.patch, HIVE-14015.2.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7454)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:662)
> at
>
[jira] [Commented] (HIVE-13873) Column pruning for nested fields
[ https://issues.apache.org/jira/browse/HIVE-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338884#comment-15338884 ] Ferdinand Xu commented on HIVE-13873: - Hi [~spena], could you help review this? Thank you! > Column pruning for nested fields > > > Key: HIVE-13873 > URL: https://issues.apache.org/jira/browse/HIVE-13873 > Project: Hive > Issue Type: New Feature > Components: Logical Optimizer >Reporter: Xuefu Zhang >Assignee: Ferdinand Xu > > Some columnar file formats such as Parquet store fields in struct type also > column by column using encoding described in Google Dramel pager. It's very > common in big data where data are stored in structs while queries only needs > a subset of the the fields in the structs. However, presently Hive still > needs to read the whole struct regardless whether all fields are selected. > Therefore, pruning unwanted sub-fields in struct or nested fields at file > reading time would be a big performance boost for such scenarios. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338878#comment-15338878
]
Hive QA commented on HIVE-14015:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811702/HIVE-14015.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/183/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/183/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-183/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811702 - PreCommit-HIVE-MASTER-Build
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch, HIVE-14015.2.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
>
[jira] [Commented] (HIVE-14038) miscellaneous acid improvements
[
https://issues.apache.org/jira/browse/HIVE-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338846#comment-15338846
]
Hive QA commented on HIVE-14038:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811667/HIVE-14038.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10236 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks2
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/182/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/182/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-182/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811667 - PreCommit-HIVE-MASTER-Build
> miscellaneous acid improvements
> ---
>
> Key: HIVE-14038
> URL: https://issues.apache.org/jira/browse/HIVE-14038
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14038.2.patch, HIVE-14038.patch
>
>
> 1. fix thread name inHouseKeeperServiceBase (currently they are all
> "org.apache.hadoop.hive.ql.txn.compactor.HouseKeeperServiceBase$1-0")
> 2. dump metastore configs from HiveConf on start up to help record values of
> properties
> 3. add some tests
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14056) Golden file updates for few tests
[
https://issues.apache.org/jira/browse/HIVE-14056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338803#comment-15338803
]
Hive QA commented on HIVE-14056:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811537/HIVE-14056.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/181/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/181/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-181/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811537 - PreCommit-HIVE-MASTER-Build
> Golden file updates for few tests
> -
>
> Key: HIVE-14056
> URL: https://issues.apache.org/jira/browse/HIVE-14056
> Project: Hive
> Issue Type: Task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14056.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14057) Add an option in llapstatus to generate output to a file
[ https://issues.apache.org/jira/browse/HIVE-14057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14057: -- Status: Patch Available (was: Open) > Add an option in llapstatus to generate output to a file > > > Key: HIVE-14057 > URL: https://issues.apache.org/jira/browse/HIVE-14057 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14057.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[
https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338768#comment-15338768
]
Hive QA commented on HIVE-14000:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811507/HIVE-14000.04.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 10236 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_aggregate_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_coalesce_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_count_distinct
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_precision
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_round
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_multi_insert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_simple
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/180/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/180/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-180/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811507 - PreCommit-HIVE-MASTER-Build
> (ORC) Changing a numeric type column of a partitioned table to lower type set
> values to something other than 'NULL'
> ---
>
> Key: HIVE-14000
> URL: https://issues.apache.org/jira/browse/HIVE-14000
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch,
> HIVE-14000.04.patch
>
>
> When an integer column is changed to a type that is smaller (e.g. bigint to
> int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the
> data is clipped instead of being NULL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14003) queries running against llap hang at times - preemption issues
[ https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14003: -- Resolution: Fixed Fix Version/s: 2.1.1 Status: Resolved (was: Patch Available) > queries running against llap hang at times - preemption issues > -- > > Key: HIVE-14003 > URL: https://issues.apache.org/jira/browse/HIVE-14003 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Siddharth Seth > Fix For: 2.1.1 > > Attachments: HIVE-14003.01.patch, HIVE-14003.02.patch > > > The preemption logic in the Hive processor needs some more work. There are > definitely windows where the abort flag is completely dropped within the Hive > processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14003) queries running against llap hang at times - preemption issues
[ https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338735#comment-15338735 ] Siddharth Seth commented on HIVE-14003: --- Thanks for the reviews [~prasanth_j], [~sershe]. Test failures are unrelated. Committing. Wonder why the test failures are up to 10 now. It was down to 4-5 a while ago. > queries running against llap hang at times - preemption issues > -- > > Key: HIVE-14003 > URL: https://issues.apache.org/jira/browse/HIVE-14003 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Siddharth Seth > Attachments: HIVE-14003.01.patch, HIVE-14003.02.patch > > > The preemption logic in the Hive processor needs some more work. There are > definitely windows where the abort flag is completely dropped within the Hive > processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14041) llap scripts add hadoop and other libraries from the machine local install to the daemon classpath
[ https://issues.apache.org/jira/browse/HIVE-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338729#comment-15338729 ] Siddharth Seth commented on HIVE-14041: --- [~sershe], [~gopalv] - could you review this please. > llap scripts add hadoop and other libraries from the machine local install to > the daemon classpath > -- > > Key: HIVE-14041 > URL: https://issues.apache.org/jira/browse/HIVE-14041 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14041.01.patch > > > `hadoop classpath` ends up getting added to the classpath of llap daemons. > This essentially means picking up the classpath from the local deploy. > This isn't required since the slider package includes relevant libraries > (shipped from the client) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14055) directSql - getting the number of partitions is broken
[ https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338672#comment-15338672 ] Sergio Peña commented on HIVE-14055: [~sershe] what if instead of return null we return a -1? I like to avoid null values in case a future developer uses the method without validating the null causing NPE exceptions. > directSql - getting the number of partitions is broken > -- > > Key: HIVE-14055 > URL: https://issues.apache.org/jira/browse/HIVE-14055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14055.patch > > > Noticed while looking at something else. If the filter cannot be pushed down > it just returns 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14056) Golden file updates for few tests
[ https://issues.apache.org/jira/browse/HIVE-14056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338671#comment-15338671 ] Sergio Peña commented on HIVE-14056: Any idea why do those files were failing? Possible due to Java8 changes? > Golden file updates for few tests > - > > Key: HIVE-14056 > URL: https://issues.apache.org/jira/browse/HIVE-14056 > Project: Hive > Issue Type: Task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14056.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Open (was: Patch Available) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Patch Available (was: Open) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Patch Available (was: Open) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Open (was: Patch Available) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Attachment: (was: HIVE-14054.2.patch) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Attachment: HIVE-14054.2.patch > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Open (was: Patch Available) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Patch Available (was: Open) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Patch Available (was: Open) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Open (was: Patch Available) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[
https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338661#comment-15338661
]
Hive QA commented on HIVE-13985:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811504/HIVE-13985.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10246 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/179/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/179/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-179/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811504 - PreCommit-HIVE-MASTER-Build
> ORC improvements for reducing the file system calls in task side
>
>
> Key: HIVE-13985
> URL: https://issues.apache.org/jira/browse/HIVE-13985
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch,
> HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch,
> HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch,
> HIVE-13985.3.patch, HIVE-13985.4.patch, HIVE-13985.5.patch
>
>
> HIVE-13840 fixed some issues with addition file system invocations during
> split generation. Similarly, this jira will fix issues with additional file
> system invocations on the task side. To avoid reading footers on the task
> side, users can set hive.orc.splits.include.file.footer to true which will
> serialize the orc footers on the splits. But this has issues with serializing
> unwanted information like column statistics and other metadata which are not
> really required for reading orc split on the task side. We can reduce the
> payload on the orc splits by serializing only the minimum required
> information (stripe information, types, compression details). This will
> decrease the payload on the orc splits and can potentially avoid OOMs in
> application master (AM) during split generation. This jira also address other
> issues concerning the AM cache. The local cache used by AM is soft reference
> cache. This can introduce unpredictability across multiple runs of the same
> query. We can cache the serialized footer in the local cache and also use
> strong reference cache which should avoid memory pressure and will have
> better predictability.
> One other improvement that we can do is when
> hive.orc.splits.include.file.footer is set to false, on the task side we make
> one additional file system call to know the size of the file. If we can
> serialize the file length in the orc split this can be avoided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Open (was: Patch Available) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Attachment: HIVE-14054.2.patch > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14054: --- Status: Patch Available (was: Open) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch, HIVE-14054.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-14054: -- Assignee: Pengcheng Xiong (was: Rajesh Balamohan) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Pengcheng Xiong > Attachments: HIVE-14054.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13872) Vectorization: Fix cross-product reduce sink serialization
[
https://issues.apache.org/jira/browse/HIVE-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13872:
Status: Patch Available (was: Open)
> Vectorization: Fix cross-product reduce sink serialization
> --
>
> Key: HIVE-13872
> URL: https://issues.apache.org/jira/browse/HIVE-13872
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13872.01.patch, HIVE-13872.WIP.patch,
> customer_demographics.txt, vector_include_no_sel.q,
> vector_include_no_sel.q.out
>
>
> TPC-DS Q13 produces a cross-product without CBO simplifying the query
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> projection column num 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:349)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRow(VectorExtractRow.java:343)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:103)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> {code}
> Simplified query
> {code}
> set hive.cbo.enable=false;
> -- explain
> select count(1)
> from store_sales
> ,customer_demographics
> where (
> (
> customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'M'
> )or
> (
>customer_demographics.cd_demo_sk = ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'U'
> ))
> ;
> {code}
> {code}
> Map 3
> Map Operator Tree:
> TableScan
> alias: customer_demographics
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> value expressions: cd_demo_sk (type: int),
> cd_marital_status (type: string)
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13872) Vectorization: Fix cross-product reduce sink serialization
[
https://issues.apache.org/jira/browse/HIVE-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13872:
Attachment: HIVE-13872.01.patch
> Vectorization: Fix cross-product reduce sink serialization
> --
>
> Key: HIVE-13872
> URL: https://issues.apache.org/jira/browse/HIVE-13872
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13872.01.patch, HIVE-13872.WIP.patch,
> customer_demographics.txt, vector_include_no_sel.q,
> vector_include_no_sel.q.out
>
>
> TPC-DS Q13 produces a cross-product without CBO simplifying the query
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> projection column num 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:349)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRow(VectorExtractRow.java:343)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:103)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> {code}
> Simplified query
> {code}
> set hive.cbo.enable=false;
> -- explain
> select count(1)
> from store_sales
> ,customer_demographics
> where (
> (
> customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'M'
> )or
> (
>customer_demographics.cd_demo_sk = ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'U'
> ))
> ;
> {code}
> {code}
> Map 3
> Map Operator Tree:
> TableScan
> alias: customer_demographics
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> value expressions: cd_demo_sk (type: int),
> cd_marital_status (type: string)
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14055) directSql - getting the number of partitions is broken
[
https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338586#comment-15338586
]
Hive QA commented on HIVE-14055:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811497/HIVE-14055.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/178/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/178/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-178/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811497 - PreCommit-HIVE-MASTER-Build
> directSql - getting the number of partitions is broken
> --
>
> Key: HIVE-14055
> URL: https://issues.apache.org/jira/browse/HIVE-14055
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14055.patch
>
>
> Noticed while looking at something else. If the filter cannot be pushed down
> it just returns 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-14054: Attachment: HIVE-14054.1.patch > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Rajesh Balamohan > Attachments: HIVE-14054.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-14054: Status: Patch Available (was: Open) > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Rajesh Balamohan > Attachments: HIVE-14054.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned HIVE-14054: --- Assignee: Rajesh Balamohan > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Rajesh Balamohan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13965) Empty resultset run into Exception when using Thrift Binary Serde
[
https://issues.apache.org/jira/browse/HIVE-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338525#comment-15338525
]
Hive QA commented on HIVE-13965:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810163/HIVE-13965.1.patch.txt
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/177/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/177/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-177/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12810163 - PreCommit-HIVE-MASTER-Build
> Empty resultset run into Exception when using Thrift Binary Serde
> -
>
> Key: HIVE-13965
> URL: https://issues.apache.org/jira/browse/HIVE-13965
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
> Attachments: HIVE-13965.1.patch.txt
>
>
> This error can be reproduced by enabling thrift binary serde, using beeline
> connect to hiveserver2 and executing the following commands:
> >create table test3(num1 int);
> >create table test4(num1 int);
> >insert into test3 values(1);
> >insert into test4 values(2);
> >select * from test3 join test4 on test3.num1=test4.num1;
> The result should be empty, but it gives an exception:
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:206)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1029)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:641)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:195)
> ... 8 more
> This error is caused in FileSinkOperator.java.
> If the resultset is empty, function process() will not be called, so variable
> "fpaths" will not be set. When run into CloseOp(),
> if (conf.isHiveServerQuery() && HiveConf.getBoolVar(hconf,
> HiveConf.ConfVars.HIVE_SERVER2_THRIFT_RESULTSET_SERIALIZE_IN_TASKS)
> &&
>
> serializer.getClass().getName().equalsIgnoreCase(ThriftJDBCBinarySerDe.class.getName()))
> {
> try {
>recordValue = serializer.serialize(null, inputObjInspectors[0]);
>rowOutWriters = fpaths.outWriters;
>rowOutWriters[0].write(recordValue);
> } catch (SerDeException | IOException e) {
>throw new HiveException(e);
> }
[jira] [Commented] (HIVE-14052) Cleanup of structures required when LLAP access from external clients completes
[
https://issues.apache.org/jira/browse/HIVE-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338499#comment-15338499
]
Hive QA commented on HIVE-14052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811479/HIVE-14052.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/176/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/176/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-176/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811479 - PreCommit-HIVE-MASTER-Build
> Cleanup of structures required when LLAP access from external clients
> completes
> ---
>
> Key: HIVE-14052
> URL: https://issues.apache.org/jira/browse/HIVE-14052
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-14052.1.patch
>
>
> Per [~sseth]: There's no cleanup at the moment, and structures used in LLAP
> to track a query will keep building up slowly over time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338465#comment-15338465
]
Yongzhi Chen commented on HIVE-14015:
-
The failures seem not related. Attach patch 2 to run the test again.
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch, HIVE-14015.2.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7454)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:662)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:966)
> at
>
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-14015:
Attachment: HIVE-14015.2.patch
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch, HIVE-14015.2.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7454)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:662)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:966)
> at
>
[jira] [Commented] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[
https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338461#comment-15338461
]
Hive QA commented on HIVE-13934:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811468/HIVE-13934.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 10236 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_binary_join_groupby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_char_mapjoin1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join30
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_nulls
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_mapjoin1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_nested_mapjoin
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/175/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/175/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-175/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811468 - PreCommit-HIVE-MASTER-Build
> Configure Tez to make nocondiional task size memory available for the
> Processor
> ---
>
> Key: HIVE-13934
> URL: https://issues.apache.org/jira/browse/HIVE-13934
> Project: Hive
> Issue Type: Bug
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch
>
>
> Currently, noconditionaltasksize is not validated against the container size,
> the reservations made in the container by Tez for Inputs / Outputs etc.
> Check this at compile time to see if enough memory is available, or set up
> the vertex to reserve additional memory for the Processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14051) Custom authenticaton in Hive JDBC
[
https://issues.apache.org/jira/browse/HIVE-14051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338431#comment-15338431
]
Hive QA commented on HIVE-14051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811459/HIVE-14051.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/174/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/174/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-174/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811459 - PreCommit-HIVE-MASTER-Build
> Custom authenticaton in Hive JDBC
> -
>
> Key: HIVE-14051
> URL: https://issues.apache.org/jira/browse/HIVE-14051
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: Vinoth Sathappan
>Assignee: Vinoth Sathappan
> Attachments: HIVE-14051.1.patch
>
>
> Enable the JDBC driver for Hive to use a pluggable module to connect to HS2
> behind gateways using OAuth, OpenID Connect, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338399#comment-15338399
]
Hive QA commented on HIVE-14015:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811461/HIVE-14015.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10221 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_env_var1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_with_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_null_projection
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_13
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_shufflejoin
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/173/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/173/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-173/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811461 - PreCommit-HIVE-MASTER-Build
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as
