[jira] [Comment Edited] (HIVE-17841) implement applying the resource plan
[ https://issues.apache.org/jira/browse/HIVE-17841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210612#comment-16210612 ] Sergey Shelukhin edited comment on HIVE-17841 at 10/19/17 5:40 AM: --- This preliminary patch is based on master although it will be committed after the pools patch. I'm pretty sure tests will fail... will look tomorrow. This rewrites WM to use a single thread to process almost everything that can happen to a session that is in use (so, excluding out of bounds expiration). This is because there are too many things coming from multiple sources and otherwise synchronization and reasoning about all the things happening in parallel becomes a huge pain. Also makes some things like getting session from the pool, waiting for AM registry, etc. async to facilitate that. There are some followup jiras that would be needed - handling internal AM restart (AM registry info update), and LLAP cluster information changes (the logic in WM is simple, the information propagation needs to be added). cc [~prasanth_j] was (Author: sershe): This preliminary patch is based on master although it will be committed after the pools patch. I'm pretty sure tests will fail... will look tomorrow. This rewrites WM to use a single thread to process almost everything that can happen to a session that is in use (so, excluding out of bounds expiration). There are some followup jiras that would be needed - handling internal AM restart (AM registry info update), and LLAP cluster information changes (the logic in WM is simple, the information propagation needs to be added). > implement applying the resource plan > > > Key: HIVE-17841 > URL: https://issues.apache.org/jira/browse/HIVE-17841 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17841.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17841) implement applying the resource plan
[ https://issues.apache.org/jira/browse/HIVE-17841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17841: --- > implement applying the resource plan > > > Key: HIVE-17841 > URL: https://issues.apache.org/jira/browse/HIVE-17841 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17696) Vectorized reader does not seem to be pushing down projection columns in certain code paths
[
https://issues.apache.org/jira/browse/HIVE-17696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-17696:
Status: Patch Available (was: Open)
> Vectorized reader does not seem to be pushing down projection columns in
> certain code paths
> ---
>
> Key: HIVE-17696
> URL: https://issues.apache.org/jira/browse/HIVE-17696
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17696.patch
>
>
> This is the code snippet from {{VectorizedParquetRecordReader.java}}
> {noformat}
> MessageType tableSchema;
> if (indexAccess) {
> List indexSequence = new ArrayList<>();
> // Generates a sequence list of indexes
> for(int i = 0; i < columnNamesList.size(); i++) {
> indexSequence.add(i);
> }
> tableSchema = DataWritableReadSupport.getSchemaByIndex(fileSchema,
> columnNamesList,
> indexSequence);
> } else {
> tableSchema = DataWritableReadSupport.getSchemaByName(fileSchema,
> columnNamesList,
> columnTypesList);
> }
> indexColumnsWanted =
> ColumnProjectionUtils.getReadColumnIDs(configuration);
> if (!ColumnProjectionUtils.isReadAllColumns(configuration) &&
> !indexColumnsWanted.isEmpty()) {
> requestedSchema =
> DataWritableReadSupport.getSchemaByIndex(tableSchema,
> columnNamesList, indexColumnsWanted);
> } else {
> requestedSchema = fileSchema;
> }
> this.reader = new ParquetFileReader(
> configuration, footer.getFileMetaData(), file, blocks,
> requestedSchema.getColumns());
> {noformat}
> Couple of things to notice here:
> Most of this code is duplicated from {{DataWritableReadSupport.init()}}
> method.
> the else condition passes in fileSchema instead of using tableSchema like we
> do in DataWritableReadSupport.init() method. Does this cause projection
> columns to be missed when we read parquet files? We should probably just
> reuse ReadContext returned from {{DataWritableReadSupport.init()}} method
> here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17696) Vectorized reader does not seem to be pushing down projection columns in certain code paths
[
https://issues.apache.org/jira/browse/HIVE-17696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-17696:
Attachment: HIVE-17696.patch
> Vectorized reader does not seem to be pushing down projection columns in
> certain code paths
> ---
>
> Key: HIVE-17696
> URL: https://issues.apache.org/jira/browse/HIVE-17696
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17696.patch
>
>
> This is the code snippet from {{VectorizedParquetRecordReader.java}}
> {noformat}
> MessageType tableSchema;
> if (indexAccess) {
> List indexSequence = new ArrayList<>();
> // Generates a sequence list of indexes
> for(int i = 0; i < columnNamesList.size(); i++) {
> indexSequence.add(i);
> }
> tableSchema = DataWritableReadSupport.getSchemaByIndex(fileSchema,
> columnNamesList,
> indexSequence);
> } else {
> tableSchema = DataWritableReadSupport.getSchemaByName(fileSchema,
> columnNamesList,
> columnTypesList);
> }
> indexColumnsWanted =
> ColumnProjectionUtils.getReadColumnIDs(configuration);
> if (!ColumnProjectionUtils.isReadAllColumns(configuration) &&
> !indexColumnsWanted.isEmpty()) {
> requestedSchema =
> DataWritableReadSupport.getSchemaByIndex(tableSchema,
> columnNamesList, indexColumnsWanted);
> } else {
> requestedSchema = fileSchema;
> }
> this.reader = new ParquetFileReader(
> configuration, footer.getFileMetaData(), file, blocks,
> requestedSchema.getColumns());
> {noformat}
> Couple of things to notice here:
> Most of this code is duplicated from {{DataWritableReadSupport.init()}}
> method.
> the else condition passes in fileSchema instead of using tableSchema like we
> do in DataWritableReadSupport.init() method. Does this cause projection
> columns to be missed when we read parquet files? We should probably just
> reuse ReadContext returned from {{DataWritableReadSupport.init()}} method
> here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17696) Vectorized reader does not seem to be pushing down projection columns in certain code paths
[
https://issues.apache.org/jira/browse/HIVE-17696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu reassigned HIVE-17696:
---
Assignee: Ferdinand Xu
> Vectorized reader does not seem to be pushing down projection columns in
> certain code paths
> ---
>
> Key: HIVE-17696
> URL: https://issues.apache.org/jira/browse/HIVE-17696
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
>
> This is the code snippet from {{VectorizedParquetRecordReader.java}}
> {noformat}
> MessageType tableSchema;
> if (indexAccess) {
> List indexSequence = new ArrayList<>();
> // Generates a sequence list of indexes
> for(int i = 0; i < columnNamesList.size(); i++) {
> indexSequence.add(i);
> }
> tableSchema = DataWritableReadSupport.getSchemaByIndex(fileSchema,
> columnNamesList,
> indexSequence);
> } else {
> tableSchema = DataWritableReadSupport.getSchemaByName(fileSchema,
> columnNamesList,
> columnTypesList);
> }
> indexColumnsWanted =
> ColumnProjectionUtils.getReadColumnIDs(configuration);
> if (!ColumnProjectionUtils.isReadAllColumns(configuration) &&
> !indexColumnsWanted.isEmpty()) {
> requestedSchema =
> DataWritableReadSupport.getSchemaByIndex(tableSchema,
> columnNamesList, indexColumnsWanted);
> } else {
> requestedSchema = fileSchema;
> }
> this.reader = new ParquetFileReader(
> configuration, footer.getFileMetaData(), file, blocks,
> requestedSchema.getColumns());
> {noformat}
> Couple of things to notice here:
> Most of this code is duplicated from {{DataWritableReadSupport.init()}}
> method.
> the else condition passes in fileSchema instead of using tableSchema like we
> do in DataWritableReadSupport.init() method. Does this cause projection
> columns to be missed when we read parquet files? We should probably just
> reuse ReadContext returned from {{DataWritableReadSupport.init()}} method
> here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17473) implement workload management pools
[
https://issues.apache.org/jira/browse/HIVE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210568#comment-16210568
]
Hive QA commented on HIVE-17473:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892888/HIVE-17473.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11284 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_non_partitioned]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16]
(batchId=243)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94]
(batchId=243)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94]
(batchId=241)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hadoop.hive.common.metrics.metrics2.TestCodahaleMetrics.testFileReporting
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7370/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7370/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7370/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892888 - PreCommit-HIVE-Build
> implement workload management pools
> ---
>
> Key: HIVE-17473
> URL: https://issues.apache.org/jira/browse/HIVE-17473
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17473.01.patch, HIVE-17473.03.patch,
> HIVE-17473.04.patch, HIVE-17473.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17164) Vectorization: Support PTF (Part 2: Unbounded Support-- Turn ON by default)
[ https://issues.apache.org/jira/browse/HIVE-17164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210555#comment-16210555 ] Matt McCline commented on HIVE-17164: - Committed to master. > Vectorization: Support PTF (Part 2: Unbounded Support-- Turn ON by default) > --- > > Key: HIVE-17164 > URL: https://issues.apache.org/jira/browse/HIVE-17164 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17164.01.patch, HIVE-17164.02.patch, > HIVE-17164.03.patch, HIVE-17164.04.patch > > > Add disk storage backing. Turn hive.vectorized.execution.ptf.enabled on by > default. > Add hive.vectorized.ptf.max.memory.buffering.batch.count to specify the > maximum number of vectorized row batch to buffer in memory before spilling to > disk. > Add hive.vectorized.testing.reducer.batch.size parameter to have the Tez > Reducer make small batches for making a lot of key group batches that cause > memory buffering and disk storage backing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17164) Vectorization: Support PTF (Part 2: Unbounded Support-- Turn ON by default)
[ https://issues.apache.org/jira/browse/HIVE-17164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17164: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorization: Support PTF (Part 2: Unbounded Support-- Turn ON by default) > --- > > Key: HIVE-17164 > URL: https://issues.apache.org/jira/browse/HIVE-17164 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17164.01.patch, HIVE-17164.02.patch, > HIVE-17164.03.patch, HIVE-17164.04.patch > > > Add disk storage backing. Turn hive.vectorized.execution.ptf.enabled on by > default. > Add hive.vectorized.ptf.max.memory.buffering.batch.count to specify the > maximum number of vectorized row batch to buffer in memory before spilling to > disk. > Add hive.vectorized.testing.reducer.batch.size parameter to have the Tez > Reducer make small batches for making a lot of key group batches that cause > memory buffering and disk storage backing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-17840: - > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17764) alter view fails when hive.metastore.disallow.incompatible.col.type.changes set to true
[
https://issues.apache.org/jira/browse/HIVE-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210534#comment-16210534
]
Hive QA commented on HIVE-17764:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892882/HIVE17764.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11281 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16]
(batchId=243)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94]
(batchId=243)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94]
(batchId=241)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7369/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7369/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7369/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892882 - PreCommit-HIVE-Build
> alter view fails when hive.metastore.disallow.incompatible.col.type.changes
> set to true
> ---
>
> Key: HIVE-17764
> URL: https://issues.apache.org/jira/browse/HIVE-17764
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE17764.1.patch, HIVE17764.2.patch
>
>
> A view is a virtual structure that derives the type information from the
> table(s) the view is based on.If the view definition is altered, the
> corresponding column types should be updated. The relevance of the change
> depending on the previous structure of the view is irrelevant.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17823) Fix subquery Qtest of Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-17823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17823: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, Thanks [~dapengsun] > Fix subquery Qtest of Hive on Spark > --- > > Key: HIVE-17823 > URL: https://issues.apache.org/jira/browse/HIVE-17823 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Attachments: HIVE-17823.001.patch > > > The JIRA is targeted to fix the Qtest files failures of HoS due to HIVE-17726 > introduced subquery fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17802) Remove unnecessary calls to FileSystem.setOwner() from FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210499#comment-16210499
]
Hive QA commented on HIVE-17802:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892929/HIVE-17802.2-branch-2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10650 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=218)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
org.apache.hive.hcatalog.pig.TestExtendedAcls.testPartitionedTableDynamic
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7368/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7368/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7368/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892929 - PreCommit-HIVE-Build
> Remove unnecessary calls to FileSystem.setOwner() from
> FileOutputCommitterContainer
> ---
>
> Key: HIVE-17802
> URL: https://issues.apache.org/jira/browse/HIVE-17802
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17802.1.patch, HIVE-17802.2-branch-2.patch,
> HIVE-17802.2.patch
>
>
> For large Pig/HCat queries that produce a large number of
> partitions/directories/files, we have seen cases where the HDFS NameNode
> groaned under the weight of {{FileSystem.setOwner()}} calls, originating from
> the commit-step. This was the result of the following code in
> FileOutputCommitterContainer:
> {code:java}
> private void applyGroupAndPerms(FileSystem fs, Path dir, FsPermission
> permission,
> List acls, String group, boolean recursive)
> throws IOException {
> ...
> if (recursive) {
> for (FileStatus fileStatus : fs.listStatus(dir)) {
> if (fileStatus.isDir()) {
> applyGroupAndPerms(fs, fileStatus.getPath(), permission, acls,
> group, true);
> } else {
> fs.setPermission(fileStatus.getPath(), permission);
> chown(fs, fileStatus.getPath(), group);
> }
> }
> }
> }
> private void chown(FileSystem fs, Path file, String group) throws
> IOException {
> try {
> fs.setOwner(file, null, group);
> } catch (AccessControlException ignore) {
> // Some users have wrong table group, ignore it.
> LOG.warn("Failed to change group of partition directories/files: " +
> file, ignore);
> }
> }
> {code}
> One call per file/directory is far too many. We have a patch that reduces the
> namenode pressure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17806) Create directory for metrics file if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17806: -- Status: Open (was: Patch Available) > Create directory for metrics file if it doesn't exist > - > > Key: HIVE-17806 > URL: https://issues.apache.org/jira/browse/HIVE-17806 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17806.04.patch > > > HIVE-17563 changed metrics code to use local file system operations instead > of Hadoop local file system operations. There is an unintended side effect - > hadoop file systems create the directory if it doesn't exist and java nio > interfaces don't. The purpose of this fix is to revert the behavior to the > original one to avoid surprises. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17806) Create directory for metrics file if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17806: -- Status: Patch Available (was: Open) > Create directory for metrics file if it doesn't exist > - > > Key: HIVE-17806 > URL: https://issues.apache.org/jira/browse/HIVE-17806 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17806.04.patch > > > HIVE-17563 changed metrics code to use local file system operations instead > of Hadoop local file system operations. There is an unintended side effect - > hadoop file systems create the directory if it doesn't exist and java nio > interfaces don't. The purpose of this fix is to revert the behavior to the > original one to avoid surprises. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17806) Create directory for metrics file if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17806: -- Attachment: HIVE-17806.04.patch > Create directory for metrics file if it doesn't exist > - > > Key: HIVE-17806 > URL: https://issues.apache.org/jira/browse/HIVE-17806 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17806.04.patch > > > HIVE-17563 changed metrics code to use local file system operations instead > of Hadoop local file system operations. There is an unintended side effect - > hadoop file systems create the directory if it doesn't exist and java nio > interfaces don't. The purpose of this fix is to revert the behavior to the > original one to avoid surprises. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17806) Create directory for metrics file if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17806: -- Attachment: (was: HIVE-17806.03.patch) > Create directory for metrics file if it doesn't exist > - > > Key: HIVE-17806 > URL: https://issues.apache.org/jira/browse/HIVE-17806 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17806.04.patch > > > HIVE-17563 changed metrics code to use local file system operations instead > of Hadoop local file system operations. There is an unintended side effect - > hadoop file systems create the directory if it doesn't exist and java nio > interfaces don't. The purpose of this fix is to revert the behavior to the > original one to avoid surprises. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17473) implement workload management pools
[ https://issues.apache.org/jira/browse/HIVE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210462#comment-16210462 ] Prasanth Jayachandran commented on HIVE-17473: -- The logic looks good to me. Considering most review comments will be gone in refactor, +1 on the latest patch. Pending tests. > implement workload management pools > --- > > Key: HIVE-17473 > URL: https://issues.apache.org/jira/browse/HIVE-17473 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17473.01.patch, HIVE-17473.03.patch, > HIVE-17473.04.patch, HIVE-17473.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17778) Add support for custom counters in trigger expression
[ https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17778: - Attachment: HIVE-17778.4.patch Fixes NPE in test. > Add support for custom counters in trigger expression > - > > Key: HIVE-17778 > URL: https://issues.apache.org/jira/browse/HIVE-17778 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17778.1.patch, HIVE-17778.2.patch, > HIVE-17778.3.patch, HIVE-17778.4.patch > > > HIVE-17508 only supports limited counters. This ticket is to extend it to > support custom counters (counters that are not supported by execution engine > will be dropped). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17836) Persisting nulls in bit vector field fails for postgres backed metastore
[ https://issues.apache.org/jira/browse/HIVE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17836: Status: Patch Available (was: Open) > Persisting nulls in bit vector field fails for postgres backed metastore > > > Key: HIVE-17836 > URL: https://issues.apache.org/jira/browse/HIVE-17836 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-17836.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17836) Persisting nulls in bit vector field fails for postgres backed metastore
[ https://issues.apache.org/jira/browse/HIVE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17836: Attachment: (was: HIVE-17836.patch) > Persisting nulls in bit vector field fails for postgres backed metastore > > > Key: HIVE-17836 > URL: https://issues.apache.org/jira/browse/HIVE-17836 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-17836.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17836) Persisting nulls in bit vector field fails for postgres backed metastore
[ https://issues.apache.org/jira/browse/HIVE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17836: Attachment: HIVE-17836.1.patch > Persisting nulls in bit vector field fails for postgres backed metastore > > > Key: HIVE-17836 > URL: https://issues.apache.org/jira/browse/HIVE-17836 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-17836.1.patch, HIVE-17836.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17836) Persisting nulls in bit vector field fails for postgres backed metastore
[ https://issues.apache.org/jira/browse/HIVE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17836: Status: Open (was: Patch Available) > Persisting nulls in bit vector field fails for postgres backed metastore > > > Key: HIVE-17836 > URL: https://issues.apache.org/jira/browse/HIVE-17836 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-17836.1.patch, HIVE-17836.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17640) Comparison of date return null if time part is provided in string.
[
https://issues.apache.org/jira/browse/HIVE-17640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210445#comment-16210445
]
Hive QA commented on HIVE-17640:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892889/HIVE-17640.01-branch-2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10651 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7367/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892889 - PreCommit-HIVE-Build
> Comparison of date return null if time part is provided in string.
> --
>
> Key: HIVE-17640
> URL: https://issues.apache.org/jira/browse/HIVE-17640
> Project: Hive
> Issue Type: Bug
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.4.0
>
> Attachments: HIVE-17640.01-branch-2.patch
>
>
> Reproduce:
> select '2017-01-01 00:00:00' < current_date;
> INFO : OK
> ...
> 1 row selected (18.324 seconds)
> ...
> NULL
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210423#comment-16210423 ] Prasanth Jayachandran edited comment on HIVE-17833 at 10/19/17 1:14 AM: Addressed review comments. Removed pom.xml changes. Will update it once tez makes a release until then this patch can't be committed anyway. 2 counters one for #files that will be read by input and other for #directories (did not call it partitions as we are not checking with partition spec). #directories comes from jobConf via mapred.input.dir. So only #files has to make sure the path in split is a file. There can be OrcSplit with delta directory, which is not accounted for in any of these counters (expensive to do list status on each delta directories here). was (Author: prasanth_j): Addressed review comments. Removed pom.xml changes. Will update it once tez makes a release until then this patch can't be committed anyway. 2 counters one for #files that will be read by input and other for #directories (did not call it partitions as we are not checking with partition spec). #directories comes from jobConf via mapred.input.dir. So only #files has to make sure the path in split is a file. There can be OrcSplit with delta directory is not account for in any of these counters (expensive to do list status on each delta directories here). > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210423#comment-16210423 ] Prasanth Jayachandran edited comment on HIVE-17833 at 10/19/17 1:13 AM: Addressed review comments. Removed pom.xml changes. Will update it once tez makes a release until then this patch can't be committed anyway. 2 counters one for #files that will be read by input and other for #directories (did not call it partitions as we are not checking with partition spec). #directories comes from jobConf via mapred.input.dir. So only #files has to make sure the path in split is a file. There can be OrcSplit with delta directory is not account for in any of these counters (expensive to do list status on each delta directories here). was (Author: prasanth_j): Addressed review comments. Removed pom.xml changes. Will update it once tez makes a release until then this patch can't be committed anyway. 2 counters one for #files that will be read by input and other for #directories (did not call it partitions as we are not checking with partition spec). #directories comes from jobConf via mapred.input.dir. So only #files has to make sure the path in split is a file. There can be OrcSplit with delta directory which don't account for in any of these counters (expensive to do list status on each delta directories here). > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17823) Fix subquery Qtest of Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-17823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210422#comment-16210422 ] Dapeng Sun commented on HIVE-17823: --- Thank [~vgarg] for your review. > Fix subquery Qtest of Hive on Spark > --- > > Key: HIVE-17823 > URL: https://issues.apache.org/jira/browse/HIVE-17823 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Attachments: HIVE-17823.001.patch > > > The JIRA is targeted to fix the Qtest files failures of HoS due to HIVE-17726 > introduced subquery fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17833: - Attachment: HIVE-17833.2.patch Addressed review comments. Removed pom.xml changes. Will update it once tez makes a release until then this patch can't be committed anyway. 2 counters one for #files that will be read by input and other for #directories (did not call it partitions as we are not checking with partition spec). #directories comes from jobConf via mapred.input.dir. So only #files has to make sure the path in split is a file. There can be OrcSplit with delta directory which don't account for in any of these counters (expensive to do list status on each delta directories here). > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17838) Make org.apache.hive.spark.client.rpc logging HoS specific
[
https://issues.apache.org/jira/browse/HIVE-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-17838:
---
> Make org.apache.hive.spark.client.rpc logging HoS specific
> --
>
> Key: HIVE-17838
> URL: https://issues.apache.org/jira/browse/HIVE-17838
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> A lot of the logging in {{org.apache.hive.spark.client.rpc}} emits
> information that a generic RPC library would, but this package is only used
> by HoS. We should make the logging more HoS specific so users don't get
> confused as to what this RPC library is doing. Ideally, they aren't even
> aware that a RPC library is in place, it should be more transparent to the
> user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17837) Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17837:
Attachment: HIVE-17837.1.patch
> Explicitly check if the HoS Remote Driver has been lost in the
> RemoteSparkJobMonitor
> -
>
> Key: HIVE-17837
> URL: https://issues.apache.org/jira/browse/HIVE-17837
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17837.1.patch
>
>
> Right now the {{RemoteSparkJobMonitor}} implicitly checks if the connection
> to the Spark remote driver is active. It does this everytime it triggers an
> invocation of the {{Rpc#call}} method (so any call to {{SparkClient#run}}).
> There are scenarios where we have seen the {{RemoteSparkJobMonitor}} when the
> connection to the driver dies, because the implicit call fails to be invoked
> (see HIVE-15860).
> It would be ideal if we made this call explicit, so we fail as soon as we
> know that the connection to the driver has died.
> The fix has the added benefit that it allows us to fail faster in the case
> where the {{RemoteSparkJobMonitor}} is in the QUEUED / SENT state. If its
> stuck in that state, it won't fail until it hits the monitor timeout (by
> default 1 minute), even though we already know the connection has died. The
> error message that is thrown is also a little imprecise, it says there could
> be queue contention, even though we know the real reason is that the
> connection was lost.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17837) Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17837:
Status: Patch Available (was: Open)
> Explicitly check if the HoS Remote Driver has been lost in the
> RemoteSparkJobMonitor
> -
>
> Key: HIVE-17837
> URL: https://issues.apache.org/jira/browse/HIVE-17837
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17837.1.patch
>
>
> Right now the {{RemoteSparkJobMonitor}} implicitly checks if the connection
> to the Spark remote driver is active. It does this everytime it triggers an
> invocation of the {{Rpc#call}} method (so any call to {{SparkClient#run}}).
> There are scenarios where we have seen the {{RemoteSparkJobMonitor}} when the
> connection to the driver dies, because the implicit call fails to be invoked
> (see HIVE-15860).
> It would be ideal if we made this call explicit, so we fail as soon as we
> know that the connection to the driver has died.
> The fix has the added benefit that it allows us to fail faster in the case
> where the {{RemoteSparkJobMonitor}} is in the QUEUED / SENT state. If its
> stuck in that state, it won't fail until it hits the monitor timeout (by
> default 1 minute), even though we already know the connection has died. The
> error message that is thrown is also a little imprecise, it says there could
> be queue contention, even though we know the real reason is that the
> connection was lost.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17835:
Status: Patch Available (was: Open)
> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17835
> URL: https://issues.apache.org/jira/browse/HIVE-17835
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17835.1.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[jira] [Assigned] (HIVE-17837) Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-17837:
---
> Explicitly check if the HoS Remote Driver has been lost in the
> RemoteSparkJobMonitor
> -
>
> Key: HIVE-17837
> URL: https://issues.apache.org/jira/browse/HIVE-17837
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Right now the {{RemoteSparkJobMonitor}} implicitly checks if the connection
> to the Spark remote driver is active. It does this everytime it triggers an
> invocation of the {{Rpc#call}} method (so any call to {{SparkClient#run}}).
> There are scenarios where we have seen the {{RemoteSparkJobMonitor}} when the
> connection to the driver dies, because the implicit call fails to be invoked
> (see HIVE-15860).
> It would be ideal if we made this call explicit, so we fail as soon as we
> know that the connection to the driver has died.
> The fix has the added benefit that it allows us to fail faster in the case
> where the {{RemoteSparkJobMonitor}} is in the QUEUED / SENT state. If its
> stuck in that state, it won't fail until it hits the monitor timeout (by
> default 1 minute), even though we already know the connection has died. The
> error message that is thrown is also a little imprecise, it says there could
> be queue contention, even though we know the real reason is that the
> connection was lost.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17805) SchemaTool validate locations should not return exit 1
[
https://issues.apache.org/jira/browse/HIVE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17805:
---
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
Status: Resolved (was: Patch Available)
Patch committed to master and branch-2
> SchemaTool validate locations should not return exit 1
> --
>
> Key: HIVE-17805
> URL: https://issues.apache.org/jira/browse/HIVE-17805
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17805.01.patch
>
>
> {{HiveSchemaTool}} can be used by other applications to validate the
> metastore schema. One of the validation check looks at the location URLs of
> tables/DBs and returns {{false}} which causes HiveSchemaTool to exit 1 to the
> calling script. Invalid locations although are a problem in some instances,
> cannot be termed as catastrophic errors in the schema which should cause Hive
> service failure/unusable. Ideally we should introduce warning levels and
> error levels in schemaTool validations so the caller can take appropriate
> action.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17834) Fix flaky triggers test
[ https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17834: - Status: Patch Available (was: Open) > Fix flaky triggers test > --- > > Key: HIVE-17834 > URL: https://issues.apache.org/jira/browse/HIVE-17834 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17834.1.patch > > > https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17834) Fix flaky triggers test
[ https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17834: - Attachment: HIVE-17834.1.patch Likely because test was timing out even before shuffle data. Giving it more time and introducing order by to add additional shuffle stage. > Fix flaky triggers test > --- > > Key: HIVE-17834 > URL: https://issues.apache.org/jira/browse/HIVE-17834 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17834.1.patch > > > https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17836) Persisting nulls in bit vector field fails for postgres backed metastore
[ https://issues.apache.org/jira/browse/HIVE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17836: Attachment: HIVE-17836.patch > Persisting nulls in bit vector field fails for postgres backed metastore > > > Key: HIVE-17836 > URL: https://issues.apache.org/jira/browse/HIVE-17836 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-17836.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17836) Persisting nulls in bit vector field fails for postgres backed metastore
[ https://issues.apache.org/jira/browse/HIVE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17836: Status: Patch Available (was: Open) > Persisting nulls in bit vector field fails for postgres backed metastore > > > Key: HIVE-17836 > URL: https://issues.apache.org/jira/browse/HIVE-17836 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-17836.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17836) Persisting nulls in bit vector field fails for postgres backed metastore
[ https://issues.apache.org/jira/browse/HIVE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-17836: --- > Persisting nulls in bit vector field fails for postgres backed metastore > > > Key: HIVE-17836 > URL: https://issues.apache.org/jira/browse/HIVE-17836 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17054) Expose SQL database constraints to Calcite
[ https://issues.apache.org/jira/browse/HIVE-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210411#comment-16210411 ] Jesus Camacho Rodriguez commented on HIVE-17054: HIVE-17432 already exposed the nullability (based on NOT NULL and PK), and referential constraints relationship (FK - PK/UK). This patch exposes unique columns (PK / UK). > Expose SQL database constraints to Calcite > -- > > Key: HIVE-17054 > URL: https://issues.apache.org/jira/browse/HIVE-17054 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 3.0.0 > > Attachments: HIVE-17054.patch > > > Hive already has support to declare multiple SQL constraints (PRIMARY KEY, > FOREIGN KEY, UNIQUE, and NOT NULL). Although these constraints cannot be > currently enforced on the data, they can be made available to the optimizer > by using the 'RELY' keyword. > Currently, even when they are declared with the RELY keyword, they are not > exposed to Calcite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17054) Expose SQL database constraints to Calcite
[ https://issues.apache.org/jira/browse/HIVE-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17054: --- Attachment: HIVE-17054.patch > Expose SQL database constraints to Calcite > -- > > Key: HIVE-17054 > URL: https://issues.apache.org/jira/browse/HIVE-17054 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 3.0.0 > > Attachments: HIVE-17054.patch > > > Hive already has support to declare multiple SQL constraints (PRIMARY KEY, > FOREIGN KEY, UNIQUE, and NOT NULL). Although these constraints cannot be > currently enforced on the data, they can be made available to the optimizer > by using the 'RELY' keyword. > Currently, even when they are declared with the RELY keyword, they are not > exposed to Calcite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17826) Error writing to RandomAccessFile after operation log is closed
[
https://issues.apache.org/jira/browse/HIVE-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210405#comment-16210405
]
Hive QA commented on HIVE-17826:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892871/HIVE-17826.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11280 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16]
(batchId=243)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94]
(batchId=243)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94]
(batchId=241)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7366/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7366/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7366/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892871 - PreCommit-HIVE-Build
> Error writing to RandomAccessFile after operation log is closed
> ---
>
> Key: HIVE-17826
> URL: https://issues.apache.org/jira/browse/HIVE-17826
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-17826.1.patch
>
>
> We are seeing the error from HS2 process stdout.
> {noformat}
> 2017-09-07 10:17:23,933 AsyncLogger-1 ERROR Attempted to append to
> non-started appender query-file-appender
> 2017-09-07 10:17:23,934 AsyncLogger-1 ERROR Attempted to append to
> non-started appender query-file-appender
> 2017-09-07 10:17:23,935 AsyncLogger-1 ERROR Unable to write to stream
> /var/log/hive/operation_logs/dd38df5b-3c09-48c9-ad64-a2eee093bea6/hive_20170907101723_1a6ad4b9-f662-4e7a-a495-06e3341308f9
> for appender query-file-appender
> 2017-09-07 10:17:23,935 AsyncLogger-1 ERROR An exception occurred processing
> Appender query-file-appender
> org.apache.logging.log4j.core.appender.AppenderLoggingException: Error
> writing to RandomAccessFile
> /var/log/hive/operation_logs/dd38df5b-3c09-48c9-ad64-a2eee093bea6/hive_20170907101723_1a6ad4b9-f662-4e7a-a495-06e3341308f9
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.flush(RandomAccessFileManager.java:114)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.write(RandomAccessFileManager.java:103)
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:136)
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:105)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileAppender.append(RandomAccessFileAppender.java:89)
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:152)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:125)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:116)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:112)
> at
>
[jira] [Work started] (HIVE-17054) Expose SQL database constraints to Calcite
[ https://issues.apache.org/jira/browse/HIVE-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17054 started by Jesus Camacho Rodriguez. -- > Expose SQL database constraints to Calcite > -- > > Key: HIVE-17054 > URL: https://issues.apache.org/jira/browse/HIVE-17054 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 3.0.0 > > > Hive already has support to declare multiple SQL constraints (PRIMARY KEY, > FOREIGN KEY, UNIQUE, and NOT NULL). Although these constraints cannot be > currently enforced on the data, they can be made available to the optimizer > by using the 'RELY' keyword. > Currently, even when they are declared with the RELY keyword, they are not > exposed to Calcite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17054) Expose SQL database constraints to Calcite
[ https://issues.apache.org/jira/browse/HIVE-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17054: --- Status: Patch Available (was: In Progress) > Expose SQL database constraints to Calcite > -- > > Key: HIVE-17054 > URL: https://issues.apache.org/jira/browse/HIVE-17054 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 3.0.0 > > > Hive already has support to declare multiple SQL constraints (PRIMARY KEY, > FOREIGN KEY, UNIQUE, and NOT NULL). Although these constraints cannot be > currently enforced on the data, they can be made available to the optimizer > by using the 'RELY' keyword. > Currently, even when they are declared with the RELY keyword, they are not > exposed to Calcite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210403#comment-16210403 ] Sergey Shelukhin commented on HIVE-17833: - Another nit: probably cheaper to check hashset first, then check the FS, even if it means adding more paths to the hashset Also, what's the logic behind isFile - what if it's not a file? > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17835:
Attachment: HIVE-17835.1.patch
> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17835
> URL: https://issues.apache.org/jira/browse/HIVE-17835
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17835.1.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
[jira] [Commented] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210394#comment-16210394 ] Sergey Shelukhin commented on HIVE-17833: - Nit: it appears that getVertexCounterName will be called a lot, perhaps the config setting should be saved somewhere in the caller and passed as an arg, w/o config? > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[ https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17718: Status: Open (was: Patch Available) > spark-client and job monitor logging improvements > - > > Key: HIVE-17718 > URL: https://issues.apache.org/jira/browse/HIVE-17718 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > > There a few improvements we can make to the logging in the HoS client to make > things easier to debug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Description: There a few improvements we can make to the logging in the HoS
client to make things easier to debug. (was: Example:
{code}
2017-10-05 17:47:11,881 ERROR
org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
[HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
exception 'java.lang.InterruptedException(sleep interrupted)'
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
at
org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
at
org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at
org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
[HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
2017-10-05 17:47:11,882 ERROR
org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
[HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
exception 'java.lang.InterruptedException(sleep interrupted)'
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
at
org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
at
org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at
org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code})
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL:
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Attachment: (was: HIVE-17718.3.patch)
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Attachment: (was: HIVE-17718.2.patch)
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Attachment: (was: HIVE-17718.1.patch)
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Commented] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210388#comment-16210388
]
Sahil Takiar commented on HIVE-17718:
-
As I started working on this, I noticed a lot of places things can be improved.
So breaking this down into multiple patches so they can be reviewed more easily.
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
>
[jira] [Commented] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210389#comment-16210389 ] Prasanth Jayachandran commented on HIVE-17833: -- The patch is small if we ignore the q test out. > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Attachment: (was: HIVE-17718.4.patch)
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Updated] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17833: - Attachment: HIVE-17833.1.patch cc/ [~aplusplus] [~sershe] > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Attachment: (was: HIVE-17718.5.patch)
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Updated] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17835:
Description:
Example:
{code}
2017-10-05 17:47:11,881 ERROR
org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
[HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
exception 'java.lang.InterruptedException(sleep interrupted)'
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
at
org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
at
org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at
org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
[HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
2017-10-05 17:47:11,882 ERROR
org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
[HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
exception 'java.lang.InterruptedException(sleep interrupted)'
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
at
org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
at
org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at
org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17835
> URL: https://issues.apache.org/jira/browse/HIVE-17835
> Project: Hive
> Issue Type:
[jira] [Assigned] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[ https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-17835: --- > HS2 Logs print unnecessary stack trace when HoS query is cancelled > -- > > Key: HIVE-17835 > URL: https://issues.apache.org/jira/browse/HIVE-17835 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210385#comment-16210385
]
Sahil Takiar commented on HIVE-16601:
-
[~pvary], [~zsombor.klara] attached updated patch.
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, Spark UI Applications List.png, Spark UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Attachment: HIVE-16601.6.patch
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, Spark UI Applications List.png, Spark UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-12631) LLAP IO: support ORC ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210384#comment-16210384 ] Prasanth Jayachandran commented on HIVE-12631: -- Filed HIVE-17834 to fix flaky triggers test > LLAP IO: support ORC ACID tables > > > Key: HIVE-12631 > URL: https://issues.apache.org/jira/browse/HIVE-12631 > Project: Hive > Issue Type: Bug > Components: llap, Transactions >Reporter: Sergey Shelukhin >Assignee: Teddy Choi > Fix For: 3.0.0 > > Attachments: HIVE-12631.1.patch, HIVE-12631.10.patch, > HIVE-12631.10.patch, HIVE-12631.11.patch, HIVE-12631.11.patch, > HIVE-12631.12.patch, HIVE-12631.13.patch, HIVE-12631.15.patch, > HIVE-12631.16.patch, HIVE-12631.17.patch, HIVE-12631.18.patch, > HIVE-12631.19.patch, HIVE-12631.2.patch, HIVE-12631.20.patch, > HIVE-12631.21.patch, HIVE-12631.22.patch, HIVE-12631.23.patch, > HIVE-12631.24.patch, HIVE-12631.25.patch, HIVE-12631.26.patch, > HIVE-12631.27.patch, HIVE-12631.28.patch, HIVE-12631.29.patch, > HIVE-12631.3.patch, HIVE-12631.30.patch, HIVE-12631.31.patch, > HIVE-12631.4.patch, HIVE-12631.5.patch, HIVE-12631.6.patch, > HIVE-12631.7.patch, HIVE-12631.8.patch, HIVE-12631.8.patch, HIVE-12631.9.patch > > > LLAP uses a completely separate read path in ORC to allow for caching and > parallelization of reads and processing. This path does not support ACID. As > far as I remember ACID logic is embedded inside ORC format; we need to > refactor it to be on top of some interface, if practical; or just port it to > LLAP read path. > Another consideration is how the logic will work with cache. The cache is > currently low-level (CB-level in ORC), so we could just use it to read bases > and deltas (deltas should be cached with higher priority) and merge as usual. > We could also cache merged representation in future. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17834) Fix flaky triggers test
[ https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-17834: > Fix flaky triggers test > --- > > Key: HIVE-17834 > URL: https://issues.apache.org/jira/browse/HIVE-17834 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[
https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210377#comment-16210377
]
Thejas M Nair commented on HIVE-17830:
--
+1
> dbnotification fails to work with rdbms other than postgres
> ---
>
> Key: HIVE-17830
> URL: https://issues.apache.org/jira/browse/HIVE-17830
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: anishek
>Assignee: Daniel Dai
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch
>
>
> as part of HIVE-17721 we had changed the direct sql to acquire the lock for
> postgres as
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> however this breaks other databases and we have to use different sql
> statements for different databases
> for postgres use
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> for SQLServer
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock);
> {code}
> for other databases
> {code}
> select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17458) VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
[ https://issues.apache.org/jira/browse/HIVE-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17458: -- Attachment: HIVE-17458.02.patch > VectorizedOrcAcidRowBatchReader doesn't handle 'original' files > --- > > Key: HIVE-17458 > URL: https://issues.apache.org/jira/browse/HIVE-17458 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17458.01.patch, HIVE-17458.02.patch > > > VectorizedOrcAcidRowBatchReader will not be used for original files. This > will likely look like a perf regression when converting a table from non-acid > to acid until it runs through a major compaction. > With Load Data support, if large files are added via Load Data, the read ops > will not vectorize until major compaction. > There is no reason why this should be the case. Just like > OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other > files in the logical tranche/bucket and calculate the offset for the RowBatch > of the split. (Presumably getRecordReader().getRowNumber() works the same in > vector mode). > In this case we don't even need OrcSplit.isOriginal() - the reader can infer > it from file path... which in particular simplifies > OrcInputFormat.determineSplitStrategies() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17458) VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
[ https://issues.apache.org/jira/browse/HIVE-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17458: -- Status: Open (was: Patch Available) > VectorizedOrcAcidRowBatchReader doesn't handle 'original' files > --- > > Key: HIVE-17458 > URL: https://issues.apache.org/jira/browse/HIVE-17458 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17458.01.patch, HIVE-17458.02.patch > > > VectorizedOrcAcidRowBatchReader will not be used for original files. This > will likely look like a perf regression when converting a table from non-acid > to acid until it runs through a major compaction. > With Load Data support, if large files are added via Load Data, the read ops > will not vectorize until major compaction. > There is no reason why this should be the case. Just like > OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other > files in the logical tranche/bucket and calculate the offset for the RowBatch > of the split. (Presumably getRecordReader().getRowNumber() works the same in > vector mode). > In this case we don't even need OrcSplit.isOriginal() - the reader can infer > it from file path... which in particular simplifies > OrcInputFormat.determineSplitStrategies() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17831) HiveSemanticAnalyzerHookContext does not update the HiveOperation after sem.analyze() is called
[ https://issues.apache.org/jira/browse/HIVE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210349#comment-16210349 ] Aihua Xu commented on HIVE-17831: - [~spena] Can you take a look the simple fix? We will update the operation type after the analyze(). > HiveSemanticAnalyzerHookContext does not update the HiveOperation after > sem.analyze() is called > --- > > Key: HIVE-17831 > URL: https://issues.apache.org/jira/browse/HIVE-17831 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.4.0, 2.2.1, 2.3.1 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-17831.1.patch > > > The SemanticAnalyzer.analyze() called on the Driver.compile() method updates > the HiveOperation based on the analysis this does. However, the patch done on > HIVE-17048 does not update such operation and is send an invalid operation to > the postAnalyze() call. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17831) HiveSemanticAnalyzerHookContext does not update the HiveOperation after sem.analyze() is called
[ https://issues.apache.org/jira/browse/HIVE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-17831: Status: Patch Available (was: Open) > HiveSemanticAnalyzerHookContext does not update the HiveOperation after > sem.analyze() is called > --- > > Key: HIVE-17831 > URL: https://issues.apache.org/jira/browse/HIVE-17831 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.4.0, 2.2.1, 2.3.1 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-17831.1.patch > > > The SemanticAnalyzer.analyze() called on the Driver.compile() method updates > the HiveOperation based on the analysis this does. However, the patch done on > HIVE-17048 does not update such operation and is send an invalid operation to > the postAnalyze() call. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17831) HiveSemanticAnalyzerHookContext does not update the HiveOperation after sem.analyze() is called
[ https://issues.apache.org/jira/browse/HIVE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-17831: Attachment: HIVE-17831.1.patch > HiveSemanticAnalyzerHookContext does not update the HiveOperation after > sem.analyze() is called > --- > > Key: HIVE-17831 > URL: https://issues.apache.org/jira/browse/HIVE-17831 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.4.0, 2.2.1, 2.3.1 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-17831.1.patch > > > The SemanticAnalyzer.analyze() called on the Driver.compile() method updates > the HiveOperation based on the analysis this does. However, the patch done on > HIVE-17048 does not update such operation and is send an invalid operation to > the postAnalyze() call. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[
https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-17830:
--
Attachment: HIVE-17830.1.patch
[~thejas] found not quoting table name should be enough to solve this issue.
Attach patch.
> dbnotification fails to work with rdbms other than postgres
> ---
>
> Key: HIVE-17830
> URL: https://issues.apache.org/jira/browse/HIVE-17830
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: anishek
>Assignee: Daniel Dai
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch
>
>
> as part of HIVE-17721 we had changed the direct sql to acquire the lock for
> postgres as
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> however this breaks other databases and we have to use different sql
> statements for different databases
> for postgres use
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> for SQLServer
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock);
> {code}
> for other databases
> {code}
> select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[
https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reassigned HIVE-17830:
-
Assignee: Daniel Dai (was: anishek)
> dbnotification fails to work with rdbms other than postgres
> ---
>
> Key: HIVE-17830
> URL: https://issues.apache.org/jira/browse/HIVE-17830
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: anishek
>Assignee: Daniel Dai
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-17830.0.patch
>
>
> as part of HIVE-17721 we had changed the direct sql to acquire the lock for
> postgres as
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> however this breaks other databases and we have to use different sql
> statements for different databases
> for postgres use
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> for SQLServer
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock);
> {code}
> for other databases
> {code}
> select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17771) Implement commands to manage resource plan.
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210334#comment-16210334 ] Sergey Shelukhin commented on HIVE-17771: - Left the comments on RB. Overall makes sense. > Implement commands to manage resource plan. > --- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17806) Create directory for metrics file if it doesn't exist
[
https://issues.apache.org/jira/browse/HIVE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210319#comment-16210319
]
Hive QA commented on HIVE-17806:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892868/HIVE-17806.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11280 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16]
(batchId=243)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94]
(batchId=243)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94]
(batchId=241)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7365/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7365/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7365/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892868 - PreCommit-HIVE-Build
> Create directory for metrics file if it doesn't exist
> -
>
> Key: HIVE-17806
> URL: https://issues.apache.org/jira/browse/HIVE-17806
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17806.03.patch
>
>
> HIVE-17563 changed metrics code to use local file system operations instead
> of Hadoop local file system operations. There is an unintended side effect -
> hadoop file systems create the directory if it doesn't exist and java nio
> interfaces don't. The purpose of this fix is to revert the behavior to the
> original one to avoid surprises.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-17833: > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17802) Remove unnecessary calls to FileSystem.setOwner() from FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210240#comment-16210240
]
Mithun Radhakrishnan edited comment on HIVE-17802 at 10/18/17 11:04 PM:
I've uploaded the {{branch-2}} patch that's likely to break
{{itests/TestExtendedAcls}}. We might have to sort out the call to
{{FileSystem.setOwner()}} to modify the group, in
{{FileOutputCommitterContainer::applyGroupAndPerms()}}.
was (Author: mithun):
I've uploaded the {{branch-2.2}} patch that's likely to break
{{itests/TestExtendedAcls}}. We might have to sort out the call to
{{FileSystem.setOwner()}} to modify the group, in
{{FileOutputCommitterContainer::applyGroupAndPerms()}}.
> Remove unnecessary calls to FileSystem.setOwner() from
> FileOutputCommitterContainer
> ---
>
> Key: HIVE-17802
> URL: https://issues.apache.org/jira/browse/HIVE-17802
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17802.1.patch, HIVE-17802.2-branch-2.patch,
> HIVE-17802.2.patch
>
>
> For large Pig/HCat queries that produce a large number of
> partitions/directories/files, we have seen cases where the HDFS NameNode
> groaned under the weight of {{FileSystem.setOwner()}} calls, originating from
> the commit-step. This was the result of the following code in
> FileOutputCommitterContainer:
> {code:java}
> private void applyGroupAndPerms(FileSystem fs, Path dir, FsPermission
> permission,
> List acls, String group, boolean recursive)
> throws IOException {
> ...
> if (recursive) {
> for (FileStatus fileStatus : fs.listStatus(dir)) {
> if (fileStatus.isDir()) {
> applyGroupAndPerms(fs, fileStatus.getPath(), permission, acls,
> group, true);
> } else {
> fs.setPermission(fileStatus.getPath(), permission);
> chown(fs, fileStatus.getPath(), group);
> }
> }
> }
> }
> private void chown(FileSystem fs, Path file, String group) throws
> IOException {
> try {
> fs.setOwner(file, null, group);
> } catch (AccessControlException ignore) {
> // Some users have wrong table group, ignore it.
> LOG.warn("Failed to change group of partition directories/files: " +
> file, ignore);
> }
> }
> {code}
> One call per file/directory is far too many. We have a patch that reduces the
> namenode pressure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17802) Remove unnecessary calls to FileSystem.setOwner() from FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210240#comment-16210240
]
Mithun Radhakrishnan commented on HIVE-17802:
-
I've uploaded the {{branch-2.2}} patch that's likely to break
{{itests/TestExtendedAcls}}. We might have to sort out the call to
{{FileSystem.setOwner()}} to modify the group, in
{{FileOutputCommitterContainer::applyGroupAndPerms()}}.
> Remove unnecessary calls to FileSystem.setOwner() from
> FileOutputCommitterContainer
> ---
>
> Key: HIVE-17802
> URL: https://issues.apache.org/jira/browse/HIVE-17802
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17802.1.patch, HIVE-17802.2-branch-2.patch,
> HIVE-17802.2.patch
>
>
> For large Pig/HCat queries that produce a large number of
> partitions/directories/files, we have seen cases where the HDFS NameNode
> groaned under the weight of {{FileSystem.setOwner()}} calls, originating from
> the commit-step. This was the result of the following code in
> FileOutputCommitterContainer:
> {code:java}
> private void applyGroupAndPerms(FileSystem fs, Path dir, FsPermission
> permission,
> List acls, String group, boolean recursive)
> throws IOException {
> ...
> if (recursive) {
> for (FileStatus fileStatus : fs.listStatus(dir)) {
> if (fileStatus.isDir()) {
> applyGroupAndPerms(fs, fileStatus.getPath(), permission, acls,
> group, true);
> } else {
> fs.setPermission(fileStatus.getPath(), permission);
> chown(fs, fileStatus.getPath(), group);
> }
> }
> }
> }
> private void chown(FileSystem fs, Path file, String group) throws
> IOException {
> try {
> fs.setOwner(file, null, group);
> } catch (AccessControlException ignore) {
> // Some users have wrong table group, ignore it.
> LOG.warn("Failed to change group of partition directories/files: " +
> file, ignore);
> }
> }
> {code}
> One call per file/directory is far too many. We have a patch that reduces the
> namenode pressure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17802) Remove unnecessary calls to FileSystem.setOwner() from FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17802:
Attachment: HIVE-17802.2-branch-2.patch
> Remove unnecessary calls to FileSystem.setOwner() from
> FileOutputCommitterContainer
> ---
>
> Key: HIVE-17802
> URL: https://issues.apache.org/jira/browse/HIVE-17802
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17802.1.patch, HIVE-17802.2-branch-2.patch,
> HIVE-17802.2.patch
>
>
> For large Pig/HCat queries that produce a large number of
> partitions/directories/files, we have seen cases where the HDFS NameNode
> groaned under the weight of {{FileSystem.setOwner()}} calls, originating from
> the commit-step. This was the result of the following code in
> FileOutputCommitterContainer:
> {code:java}
> private void applyGroupAndPerms(FileSystem fs, Path dir, FsPermission
> permission,
> List acls, String group, boolean recursive)
> throws IOException {
> ...
> if (recursive) {
> for (FileStatus fileStatus : fs.listStatus(dir)) {
> if (fileStatus.isDir()) {
> applyGroupAndPerms(fs, fileStatus.getPath(), permission, acls,
> group, true);
> } else {
> fs.setPermission(fileStatus.getPath(), permission);
> chown(fs, fileStatus.getPath(), group);
> }
> }
> }
> }
> private void chown(FileSystem fs, Path file, String group) throws
> IOException {
> try {
> fs.setOwner(file, null, group);
> } catch (AccessControlException ignore) {
> // Some users have wrong table group, ignore it.
> LOG.warn("Failed to change group of partition directories/files: " +
> file, ignore);
> }
> }
> {code}
> One call per file/directory is far too many. We have a patch that reduces the
> namenode pressure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17832) Allow hive.metastore.disallow.incompatible.col.type.changes to be changed within a session
[ https://issues.apache.org/jira/browse/HIVE-17832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-17832: --- Attachment: HIVE17832.1.patch > Allow hive.metastore.disallow.incompatible.col.type.changes to be changed > within a session > -- > > Key: HIVE-17832 > URL: https://issues.apache.org/jira/browse/HIVE-17832 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE17832.1.patch > > > hive.metastore.disallow.incompatible.col.type.changes when set to true, will > disallow incompatible column type changes through alter table. But, this > parameter is set system wide, and changing it requires restart of HMS. The > default value of this parameter is true. User can set the parameter to false > and change the column type through alter if this can be modified within a > session. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17832) Allow hive.metastore.disallow.incompatible.col.type.changes to be changed within a session
[ https://issues.apache.org/jira/browse/HIVE-17832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-17832: --- Fix Version/s: 3.0.0 Affects Version/s: 2.1.1 Status: Patch Available (was: Open) > Allow hive.metastore.disallow.incompatible.col.type.changes to be changed > within a session > -- > > Key: HIVE-17832 > URL: https://issues.apache.org/jira/browse/HIVE-17832 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE17832.1.patch > > > hive.metastore.disallow.incompatible.col.type.changes when set to true, will > disallow incompatible column type changes through alter table. But, this > parameter is set system wide, and changing it requires restart of HMS. The > default value of this parameter is true. User can set the parameter to false > and change the column type through alter if this can be modified within a > session. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-12631) LLAP IO: support ORC ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12631: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks a lot for all the work on the patch! > LLAP IO: support ORC ACID tables > > > Key: HIVE-12631 > URL: https://issues.apache.org/jira/browse/HIVE-12631 > Project: Hive > Issue Type: Bug > Components: llap, Transactions >Reporter: Sergey Shelukhin >Assignee: Teddy Choi > Fix For: 3.0.0 > > Attachments: HIVE-12631.1.patch, HIVE-12631.10.patch, > HIVE-12631.10.patch, HIVE-12631.11.patch, HIVE-12631.11.patch, > HIVE-12631.12.patch, HIVE-12631.13.patch, HIVE-12631.15.patch, > HIVE-12631.16.patch, HIVE-12631.17.patch, HIVE-12631.18.patch, > HIVE-12631.19.patch, HIVE-12631.2.patch, HIVE-12631.20.patch, > HIVE-12631.21.patch, HIVE-12631.22.patch, HIVE-12631.23.patch, > HIVE-12631.24.patch, HIVE-12631.25.patch, HIVE-12631.26.patch, > HIVE-12631.27.patch, HIVE-12631.28.patch, HIVE-12631.29.patch, > HIVE-12631.3.patch, HIVE-12631.30.patch, HIVE-12631.31.patch, > HIVE-12631.4.patch, HIVE-12631.5.patch, HIVE-12631.6.patch, > HIVE-12631.7.patch, HIVE-12631.8.patch, HIVE-12631.8.patch, HIVE-12631.9.patch > > > LLAP uses a completely separate read path in ORC to allow for caching and > parallelization of reads and processing. This path does not support ACID. As > far as I remember ACID logic is embedded inside ORC format; we need to > refactor it to be on top of some interface, if practical; or just port it to > LLAP read path. > Another consideration is how the logic will work with cache. The cache is > currently low-level (CB-level in ORC), so we could just use it to read bases > and deltas (deltas should be cached with higher priority) and merge as usual. > We could also cache merged representation in future. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-12631) LLAP IO: support ORC ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12631: Summary: LLAP IO: support ORC ACID tables (was: LLAP: support ORC ACID tables) > LLAP IO: support ORC ACID tables > > > Key: HIVE-12631 > URL: https://issues.apache.org/jira/browse/HIVE-12631 > Project: Hive > Issue Type: Bug > Components: llap, Transactions >Reporter: Sergey Shelukhin >Assignee: Teddy Choi > Attachments: HIVE-12631.1.patch, HIVE-12631.10.patch, > HIVE-12631.10.patch, HIVE-12631.11.patch, HIVE-12631.11.patch, > HIVE-12631.12.patch, HIVE-12631.13.patch, HIVE-12631.15.patch, > HIVE-12631.16.patch, HIVE-12631.17.patch, HIVE-12631.18.patch, > HIVE-12631.19.patch, HIVE-12631.2.patch, HIVE-12631.20.patch, > HIVE-12631.21.patch, HIVE-12631.22.patch, HIVE-12631.23.patch, > HIVE-12631.24.patch, HIVE-12631.25.patch, HIVE-12631.26.patch, > HIVE-12631.27.patch, HIVE-12631.28.patch, HIVE-12631.29.patch, > HIVE-12631.3.patch, HIVE-12631.30.patch, HIVE-12631.31.patch, > HIVE-12631.4.patch, HIVE-12631.5.patch, HIVE-12631.6.patch, > HIVE-12631.7.patch, HIVE-12631.8.patch, HIVE-12631.8.patch, HIVE-12631.9.patch > > > LLAP uses a completely separate read path in ORC to allow for caching and > parallelization of reads and processing. This path does not support ACID. As > far as I remember ACID logic is embedded inside ORC format; we need to > refactor it to be on top of some interface, if practical; or just port it to > LLAP read path. > Another consideration is how the logic will work with cache. The cache is > currently low-level (CB-level in ORC), so we could just use it to read bases > and deltas (deltas should be cached with higher priority) and merge as usual. > We could also cache merged representation in future. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17765) expose Hive keywords
[ https://issues.apache.org/jira/browse/HIVE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17765: Attachment: HIVE-17765.03.patch Addressing the code review feedback > expose Hive keywords > - > > Key: HIVE-17765 > URL: https://issues.apache.org/jira/browse/HIVE-17765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Thejas M Nair > Attachments: HIVE-17765.01.patch, HIVE-17765.02.patch, > HIVE-17765.03.patch, HIVE-17765.nogen.patch, HIVE-17765.patch > > > This could be useful e.g. for BI tools (via ODBC/JDBC drivers) to decide on > SQL capabilities of Hive -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17832) Allow hive.metastore.disallow.incompatible.col.type.changes to be changed within a session
[ https://issues.apache.org/jira/browse/HIVE-17832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-17832: --- Description: hive.metastore.disallow.incompatible.col.type.changes when set to true, will disallow incompatible column type changes through alter table. But, this parameter is set system wide, and changing it requires restart of HMS. The default value of this parameter is true. User can set the parameter to false and change the column type through alter if this can be modified within a session. > Allow hive.metastore.disallow.incompatible.col.type.changes to be changed > within a session > -- > > Key: HIVE-17832 > URL: https://issues.apache.org/jira/browse/HIVE-17832 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > > hive.metastore.disallow.incompatible.col.type.changes when set to true, will > disallow incompatible column type changes through alter table. But, this > parameter is set system wide, and changing it requires restart of HMS. The > default value of this parameter is true. User can set the parameter to false > and change the column type through alter if this can be modified within a > session. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17832) Allow hive.metastore.disallow.incompatible.col.type.changes to be changed within a session
[ https://issues.apache.org/jira/browse/HIVE-17832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani reassigned HIVE-17832: -- Assignee: Janaki Lahorani > Allow hive.metastore.disallow.incompatible.col.type.changes to be changed > within a session > -- > > Key: HIVE-17832 > URL: https://issues.apache.org/jira/browse/HIVE-17832 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15522) REPL LOAD & DUMP support for incremental ALTER_TABLE/ALTER_PTN including renames
[
https://issues.apache.org/jira/browse/HIVE-15522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210115#comment-16210115
]
Mithun Radhakrishnan commented on HIVE-15522:
-
With this change, {{AddPartitionMessage}} includes the full-fat
{{api.Partition}} objects. At least in production at Yahoo, that number can run
into the thousands. This isn't as big a problem for {{CreateTableMessage}},
{{CreateDatabaseMessage}}, etc. Depending on how the messages are transmitted
(e.g. over JMS), the message-size can pose a challenge.
Note that the corresponding code in HCatalog leaves out the Thrift objects.
I'll raise a separate JIRA to see if we can find middle-ground, with regard to
the Thrift objects.
> REPL LOAD & DUMP support for incremental ALTER_TABLE/ALTER_PTN including
> renames
>
>
> Key: HIVE-15522
> URL: https://issues.apache.org/jira/browse/HIVE-15522
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Fix For: 2.2.0
>
> Attachments: HIVE-15522.2.patch, HIVE-15522.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16604) Use [NOT] ENFORCED for column constraint characteristics
[
https://issues.apache.org/jira/browse/HIVE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-16604:
---
Attachment: HIVE-16604.patch
> Use [NOT] ENFORCED for column constraint characteristics
>
>
> Key: HIVE-16604
> URL: https://issues.apache.org/jira/browse/HIVE-16604
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16604.patch
>
>
> From HIVE-16575:
> {quote}
> SQL:2011 spec, 10.8, uses [ NOT ] ENFORCED for column constraint
> characteristics.
> I think Oracle invented DISABLE and NORELY and maybe others have copied it
> over time.
> A quick check I see DB2 uses the [ NOT ] ENFORCED. Teradata uses something
> else entirely.
> Personally I think there are advantages to adopting the standard approach.
> There is no standard approach for hinting the optimizer so we're on our own
> there.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Work started] (HIVE-16604) Use [NOT] ENFORCED for column constraint characteristics
[
https://issues.apache.org/jira/browse/HIVE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-16604 started by Jesus Camacho Rodriguez.
--
> Use [NOT] ENFORCED for column constraint characteristics
>
>
> Key: HIVE-16604
> URL: https://issues.apache.org/jira/browse/HIVE-16604
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Jesus Camacho Rodriguez
>
> From HIVE-16575:
> {quote}
> SQL:2011 spec, 10.8, uses [ NOT ] ENFORCED for column constraint
> characteristics.
> I think Oracle invented DISABLE and NORELY and maybe others have copied it
> over time.
> A quick check I see DB2 uses the [ NOT ] ENFORCED. Teradata uses something
> else entirely.
> Personally I think there are advantages to adopting the standard approach.
> There is no standard approach for hinting the optimizer so we're on our own
> there.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17828) Metastore: mysql upgrade scripts to 3.0.0 is broken
[
https://issues.apache.org/jira/browse/HIVE-17828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210099#comment-16210099
]
Sergey Shelukhin commented on HIVE-17828:
-
cc [~harishjp]
> Metastore: mysql upgrade scripts to 3.0.0 is broken
> ---
>
> Key: HIVE-17828
> URL: https://issues.apache.org/jira/browse/HIVE-17828
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gopal V
>
> {code}
> +-+
> | |
> +-+
> | Finished upgrading MetaStore schema from 2.2.0 to 2.3.0 |
> +-+
> 1 row in set, 1 warning (0.00 sec)
> mysql> source upgrade-2.3.0-to-3.0.0.mysql.sql
> ++
> ||
> ++
> | Upgrading MetaStore schema from 2.3.0 to 3.0.0 |
> ++
> {code}
> {code}
> --
> CREATE TABLE WM_RESOURCEPLAN (
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `QUERY_PARALLELISM` int(11),
> `STATUS` varchar(20) NOT NULL,
> PRIMARY KEY (`RP_ID`),
> KEY `UNIQUE_WM_RESOURCEPLAN` (`NAME`),
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near ') ENGINE=InnoDB DEFAULT CHARSET=latin1' at line 8
> --
> CREATE TABLE WM_POOL
> (
> `POOL_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `PATH` varchar(1024) NOT NULL,
> `PARENT_POOL_ID` bigint(20),
> `ALLOC_FRACTION` DOUBLE,
> `QUERY_PARALLELISM` int(11),
> PRIMARY KEY (`POOL_ID`),
> KEY `UNIQUE_WM_POOL` (`RP_ID`, `PATH`),
> CONSTRAINT `WM_POOL_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`),
> CONSTRAINT `WM_POOL_FK2` FOREIGN KEY (`PARENT_POOL_ID`) REFERENCES
> `WM_POOL` (`POOL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
> --
> CREATE TABLE WM_TRIGGER
> (
> `TRIGGER_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `TRIGGER_EXPRESSION` varchar(1024),
> `ACTION_EXPRESSION` varchar(1024),
> PRIMARY KEY (`TRIGGER_ID`),
> KEY `UNIQUE_WM_TRIGGER` (`RP_ID`, `NAME`),
> CONSTRAINT `WM_TRIGGER_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1215 (HY000): Cannot add foreign key constraint
> --
> CREATE TABLE WM_POOL_TO_TRIGGER
> (
> `POOL_ID` bigint(20) NOT NULL,
> `TRIGGER_ID` bigint(20) NOT NULL,
> PRIMARY KEY (`POOL_ID`, `TRIGGER_ID`),
> CONSTRAINT `WM_POOL_TO_TRIGGER_FK1` FOREIGN KEY (`POOL_ID`) REFERENCES
> `WM_POOL` (`POOL_ID`),
> CONSTRAINT `WM_POOL_TO_TRIGGER_FK2` FOREIGN KEY (`TRIGGER_ID`) REFERENCES
> `WM_TRIGGER` (`TRIGGER_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1215 (HY000): Cannot add foreign key constraint
> --
> CREATE TABLE WM_MAPPING
> (
> `MAPPING_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `ENTITY_TYPE` varchar(10) NOT NULL,
> `ENTITY_NAME` varchar(128) NOT NULL,
> `POOL_ID` bigint(20) NOT NULL,
> `ORDERING int,
> PRIMARY KEY (`MAPPING_ID`),
> KEY `UNIQUE_WM_MAPPING` (`RP_ID`, `ENTITY_TYPE`, `ENTITY_NAME`),
> CONSTRAINT `WM_MAPPING_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`),
> CONSTRAINT `WM_MAPPING_FK2` FOREIGN KEY (`POOL_ID`) REFERENCES `WM_POOL`
> (`POOL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> --
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near 'MAPPING_ID`),
> KEY `UNIQUE_WM_MAPPING` (`RP_ID`, `ENTITY_TYPE`, `ENTITY_NAME`' at line 8
> --
> UPDATE VERSION SET SCHEMA_VERSION='3.0.0', VERSION_COMMENT='Hive release
> version 3.0.0' where VER_ID=1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17789) Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems
[ https://issues.apache.org/jira/browse/HIVE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210069#comment-16210069 ] Andrew Sherman commented on HIVE-17789: --- Thanks [~stakiar] > Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing > related problems > - > > Key: HIVE-17789 > URL: https://issues.apache.org/jira/browse/HIVE-17789 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.0.0 >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 3.0.0 > > Attachments: HIVE-17789.1.patch > > > The test is waiting for a worker thread to be timed out. The time after which > the timeout should happen in 3000 ms. The test waits for 3200 ms, and > sometimes this is not enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17789) Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems
[ https://issues.apache.org/jira/browse/HIVE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17789: Issue Type: Test (was: Bug) > Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing > related problems > - > > Key: HIVE-17789 > URL: https://issues.apache.org/jira/browse/HIVE-17789 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.0.0 >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 3.0.0 > > Attachments: HIVE-17789.1.patch > > > The test is waiting for a worker thread to be timed out. The time after which > the timeout should happen in 3000 ms. The test waits for 3200 ms, and > sometimes this is not enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17789) Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems
[ https://issues.apache.org/jira/browse/HIVE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17789: Component/s: Test > Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing > related problems > - > > Key: HIVE-17789 > URL: https://issues.apache.org/jira/browse/HIVE-17789 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.0.0 >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 3.0.0 > > Attachments: HIVE-17789.1.patch > > > The test is waiting for a worker thread to be timed out. The time after which > the timeout should happen in 3000 ms. The test waits for 3200 ms, and > sometimes this is not enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17789) Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems
[ https://issues.apache.org/jira/browse/HIVE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17789: Resolution: Fixed Fix Version/s: 3.0.0 Target Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks [~asherman] for the review. > Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing > related problems > - > > Key: HIVE-17789 > URL: https://issues.apache.org/jira/browse/HIVE-17789 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 3.0.0 > > Attachments: HIVE-17789.1.patch > > > The test is waiting for a worker thread to be timed out. The time after which > the timeout should happen in 3000 ms. The test waits for 3200 ms, and > sometimes this is not enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17789) Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems
[ https://issues.apache.org/jira/browse/HIVE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17789: Affects Version/s: 3.0.0 > Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing > related problems > - > > Key: HIVE-17789 > URL: https://issues.apache.org/jira/browse/HIVE-17789 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 3.0.0 > > Attachments: HIVE-17789.1.patch > > > The test is waiting for a worker thread to be timed out. The time after which > the timeout should happen in 3000 ms. The test waits for 3200 ms, and > sometimes this is not enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17831) HiveSemanticAnalyzerHookContext does not update the HiveOperation after sem.analyze() is called
[ https://issues.apache.org/jira/browse/HIVE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu reassigned HIVE-17831: --- Assignee: Aihua Xu > HiveSemanticAnalyzerHookContext does not update the HiveOperation after > sem.analyze() is called > --- > > Key: HIVE-17831 > URL: https://issues.apache.org/jira/browse/HIVE-17831 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.4.0, 2.2.1, 2.3.1 >Reporter: Sergio Peña >Assignee: Aihua Xu > > The SemanticAnalyzer.analyze() called on the Driver.compile() method updates > the HiveOperation based on the analysis this does. However, the patch done on > HIVE-17048 does not update such operation and is send an invalid operation to > the postAnalyze() call. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16603) Enforce foreign keys to refer to primary keys or unique keys
[ https://issues.apache.org/jira/browse/HIVE-16603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-16603: --- Attachment: HIVE-16603.patch > Enforce foreign keys to refer to primary keys or unique keys > > > Key: HIVE-16603 > URL: https://issues.apache.org/jira/browse/HIVE-16603 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-16603.patch > > > Follow-up on HIVE-16575. > Currently we do not enforce foreign keys to refer to primary keys or unique > keys (as opposed to PostgreSQL and others); we should do that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16603) Enforce foreign keys to refer to primary keys or unique keys
[ https://issues.apache.org/jira/browse/HIVE-16603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-16603: --- Status: Patch Available (was: In Progress) > Enforce foreign keys to refer to primary keys or unique keys > > > Key: HIVE-16603 > URL: https://issues.apache.org/jira/browse/HIVE-16603 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-16603.patch > > > Follow-up on HIVE-16575. > Currently we do not enforce foreign keys to refer to primary keys or unique > keys (as opposed to PostgreSQL and others); we should do that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-16603) Enforce foreign keys to refer to primary keys or unique keys
[ https://issues.apache.org/jira/browse/HIVE-16603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-16603 started by Jesus Camacho Rodriguez. -- > Enforce foreign keys to refer to primary keys or unique keys > > > Key: HIVE-16603 > URL: https://issues.apache.org/jira/browse/HIVE-16603 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Follow-up on HIVE-16575. > Currently we do not enforce foreign keys to refer to primary keys or unique > keys (as opposed to PostgreSQL and others); we should do that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17789) Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems
[ https://issues.apache.org/jira/browse/HIVE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210016#comment-16210016 ] Andrew Sherman commented on HIVE-17789: --- Thanks [~stakiar]and [~aihuaxu] for reviews so can one of you push when you get a chance > Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing > related problems > - > > Key: HIVE-17789 > URL: https://issues.apache.org/jira/browse/HIVE-17789 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-17789.1.patch > > > The test is waiting for a worker thread to be timed out. The time after which > the timeout should happen in 3000 ms. The test waits for 3200 ms, and > sometimes this is not enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17789) Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems
[ https://issues.apache.org/jira/browse/HIVE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210010#comment-16210010 ] Aihua Xu commented on HIVE-17789: - The change looks good. +1. > Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing > related problems > - > > Key: HIVE-17789 > URL: https://issues.apache.org/jira/browse/HIVE-17789 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-17789.1.patch > > > The test is waiting for a worker thread to be timed out. The time after which > the timeout should happen in 3000 ms. The test waits for 3200 ms, and > sometimes this is not enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17640) Comparison of date return null if time part is provided in string.
[ https://issues.apache.org/jira/browse/HIVE-17640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-17640: Attachment: HIVE-17640.01-branch-2.patch > Comparison of date return null if time part is provided in string. > -- > > Key: HIVE-17640 > URL: https://issues.apache.org/jira/browse/HIVE-17640 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.4.0 > > Attachments: HIVE-17640.01-branch-2.patch > > > Reproduce: > select '2017-01-01 00:00:00' < current_date; > INFO : OK > ... > 1 row selected (18.324 seconds) > ... > NULL -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17640) Comparison of date return null if time part is provided in string.
[ https://issues.apache.org/jira/browse/HIVE-17640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-17640: Status: Patch Available (was: Open) > Comparison of date return null if time part is provided in string. > -- > > Key: HIVE-17640 > URL: https://issues.apache.org/jira/browse/HIVE-17640 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.4.0 > > Attachments: HIVE-17640.01-branch-2.patch > > > Reproduce: > select '2017-01-01 00:00:00' < current_date; > INFO : OK > ... > 1 row selected (18.324 seconds) > ... > NULL -- This message was sent by Atlassian JIRA (v6.4.14#64029)
