[jira] [Created] (HIVE-20138) Sessions: Closing Tez sessions if opened outside of default sessions throws NPE
Gopal V created HIVE-20138:
--
Summary: Sessions: Closing Tez sessions if opened outside of
default sessions throws NPE
Key: HIVE-20138
URL: https://issues.apache.org/jira/browse/HIVE-20138
Project: Hive
Issue Type: Bug
Components: Tez
Reporter: Gopal V
On a busy cluster, if a session is currently in ACCEPTED state (i.e YARN hasn't
started it yet), running a query will NPE.
{code}
java.lang.NullPointerException: null
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.getSession(TezSessionState.java:711)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:646)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:353)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:467)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getUnmanagedSession(WorkloadManagerFederation.java:66)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getSession(WorkloadManagerFederation.java:38)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:184)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] hive pull request #398: HIVE-20131 : SQL Script changes for creating txn wri...
GitHub user maheshk114 opened a pull request: https://github.com/apache/hive/pull/398 HIVE-20131 : SQL Script changes for creating txn write notification in 3.2.0 files You can merge this pull request into a Git repository by running: $ git pull https://github.com/maheshk114/hive HIVE-20131 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/398.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #398 commit c3968158b7e5cefdf7dda7729b8d4ed6a8a89e13 Author: Mahesh Kumar Behera Date: 2018-07-11T02:46:19Z HIVE-20131 : SQL Script changes for creating txn write notification in 3.2.0 files ---
[jira] [Created] (HIVE-20137) Truncate for Transactional tables should use base_x
Eugene Koifman created HIVE-20137: - Summary: Truncate for Transactional tables should use base_x Key: HIVE-20137 URL: https://issues.apache.org/jira/browse/HIVE-20137 Project: Hive Issue Type: Improvement Components: Transactions Affects Versions: 3.0.0 Reporter: Eugene Koifman This is a follow up to HIVE-19387. Once we have a lock that blocks writers but not readers (HIVE-19369), it would make sense to make truncate create a new base_x, where is x is a writeId in current txn - the same as Insert Overwrite does. This would mean it can work w/o interfering with existing writers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 67731: HIVE-19902 Provide Metastore micro-benchmarks
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67731/ --- (Updated July 10, 2018, 8:28 p.m.) Review request for hive, Andrew Sherman, Alan Gates, Janaki Lahorani, Marta Kuczora, Peter Vary, Sahil Takiar, and Vihang Karajgaonkar. Changes --- Moved to picocli for option processing Summary (updated) - HIVE-19902 Provide Metastore micro-benchmarks Bugs: HIVE-19902 https://issues.apache.org/jira/browse/HIVE-19902 Repository: hive-git Description (updated) --- HIVE-19902 Provide Metastore micro-benchmarks Diffs (updated) - metastore-tools/data/conf/.keepme PRE-CREATION metastore-tools/metastore-benchmarks/README.md PRE-CREATION metastore-tools/metastore-benchmarks/pom.xml PRE-CREATION metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchData.java PRE-CREATION metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java PRE-CREATION metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java PRE-CREATION metastore-tools/metastore-benchmarks/src/main/resources/log4j.properties PRE-CREATION metastore-tools/metastore-benchmarks/src/main/resources/log4j2.xml PRE-CREATION metastore-tools/pom.xml PRE-CREATION metastore-tools/tools-common/pom.xml PRE-CREATION metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java PRE-CREATION metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Constants.java PRE-CREATION metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java PRE-CREATION metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/MicroBenchmark.java PRE-CREATION metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java PRE-CREATION metastore-tools/tools-common/src/test/java/org/apache/hadoop/hive/metastore/tools/HMSClientTest.java PRE-CREATION metastore-tools/tools-common/src/test/java/org/apache/hadoop/hive/metastore/tools/UtilTest.java PRE-CREATION pom.xml 28ad152545e26d9df0ef8cc341840299e9b2743c Diff: https://reviews.apache.org/r/67731/diff/3/ Changes: https://reviews.apache.org/r/67731/diff/2-3/ Testing --- Thanks, Alexander Kolbasov
Re: [VOTE] Should we release storage-api 2.7.0 rc0?
Thanks Owen for finding this out. I will work on the next RC once this blocker is resolved. Regards, Deepak On 7/10/18, 9:40 AM, "Owen O'Malley" wrote: Ok, Jesus and I tracked it down and I've filed https://issues.apache.org/jira/browse/HIVE-20135 that is a blocker on storage-api 2.7.0. The impact was that orc 1.5 and master failed with the RC. orc 1.4 and older were fine. .. Owen On Tue, Jul 10, 2018 at 8:17 AM, Owen O'Malley wrote: > I wanted to give an update on this. For now, I'm -1 because the ORC > (branch-1.5) tests fail with this RC. I'll dig into what is wrong, but it > looks like something in the timezone changes broke backwards compatibility. > > .. Owen > > On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal > wrote: > >> Thanks Alan. >> >> On 7/9/18, 10:17 AM, "Alan Gates" wrote: >> >> +1. Did a build with a clean maven repo, checked the signature and >> sha >> hash, ran RAT. >> >> Alan. >> >> On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal < >> djais...@hortonworks.com> >> wrote: >> >> > Hi, >> > >> > I would like to make a new release of the storage-api. It contains >> changes >> > required for Hive 3.1 release. >> > >> > Artifcats: >> > Tag : >> > https://github.com/apache/hive/releases/tag/storage-release- >> 2.7.0-rc0 >> > Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/ >> > >> > Regards, >> > Deepak >> > >> >> >> >
Re: [VOTE] Should we release storage-api 2.7.0 rc0?
Ok, Jesus and I tracked it down and I've filed https://issues.apache.org/jira/browse/HIVE-20135 that is a blocker on storage-api 2.7.0. The impact was that orc 1.5 and master failed with the RC. orc 1.4 and older were fine. .. Owen On Tue, Jul 10, 2018 at 8:17 AM, Owen O'Malley wrote: > I wanted to give an update on this. For now, I'm -1 because the ORC > (branch-1.5) tests fail with this RC. I'll dig into what is wrong, but it > looks like something in the timezone changes broke backwards compatibility. > > .. Owen > > On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal > wrote: > >> Thanks Alan. >> >> On 7/9/18, 10:17 AM, "Alan Gates" wrote: >> >> +1. Did a build with a clean maven repo, checked the signature and >> sha >> hash, ran RAT. >> >> Alan. >> >> On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal < >> djais...@hortonworks.com> >> wrote: >> >> > Hi, >> > >> > I would like to make a new release of the storage-api. It contains >> changes >> > required for Hive 3.1 release. >> > >> > Artifcats: >> > Tag : >> > https://github.com/apache/hive/releases/tag/storage-release- >> 2.7.0-rc0 >> > Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/ >> > >> > Regards, >> > Deepak >> > >> >> >> >
[jira] [Created] (HIVE-20136) Code Review of ArchiveUtils Class
BELUGA BEHR created HIVE-20136:
--
Summary: Code Review of ArchiveUtils Class
Key: HIVE-20136
URL: https://issues.apache.org/jira/browse/HIVE-20136
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Affects Versions: 3.0.0, 4.0.0
Reporter: BELUGA BEHR
General code review of {{ArchiveUtil}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20135) Fix incompatible change in TimestampColumnVector to default to UTC
Owen O'Malley created HIVE-20135: Summary: Fix incompatible change in TimestampColumnVector to default to UTC Key: HIVE-20135 URL: https://issues.apache.org/jira/browse/HIVE-20135 Project: Hive Issue Type: Improvement Reporter: Owen O'Malley Assignee: Jesus Camacho Rodriguez HIVE-20007 changed the default for TimestampColumnVector to be to use UTC, which breaks the API compatibility with storage-api 2.6. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Should we release storage-api 2.7.0 rc0?
I wanted to give an update on this. For now, I'm -1 because the ORC (branch-1.5) tests fail with this RC. I'll dig into what is wrong, but it looks like something in the timezone changes broke backwards compatibility. .. Owen On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal wrote: > Thanks Alan. > > On 7/9/18, 10:17 AM, "Alan Gates" wrote: > > +1. Did a build with a clean maven repo, checked the signature and sha > hash, ran RAT. > > Alan. > > On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal < > djais...@hortonworks.com> > wrote: > > > Hi, > > > > I would like to make a new release of the storage-api. It contains > changes > > required for Hive 3.1 release. > > > > Artifcats: > > Tag : > > https://github.com/apache/hive/releases/tag/storage- > release-2.7.0-rc0 > > Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/ > > > > Regards, > > Deepak > > > > >
[GitHub] hive pull request #397: HIVE-20044: Arrow Serde should pad char values and h...
GitHub user pudidic opened a pull request: https://github.com/apache/hive/pull/397 HIVE-20044: Arrow Serde should pad char values and handle empty strin⦠â¦gs correctly (Teddy Choi) Change-Id: Ia87503aabf38c9599b887795d2b62566d8965f9b You can merge this pull request into a Git repository by running: $ git pull https://github.com/pudidic/hive HIVE-20044 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/397.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #397 commit e7f463c1865d8518a5eb1756b9a548442fb776c1 Author: Teddy Choi Date: 2018-07-10T13:07:13Z HIVE-20044: Arrow Serde should pad char values and handle empty strings correctly (Teddy Choi) Change-Id: Ia87503aabf38c9599b887795d2b62566d8965f9b ---
[GitHub] hive pull request #396: HIVE-20044: Arrow Serde should pad char values and h...
GitHub user pudidic opened a pull request: https://github.com/apache/hive/pull/396 HIVE-20044: Arrow Serde should pad char values and handle empty strin⦠â¦gs correctly (Teddy Choi) Change-Id: Iaa4417056a6e2a88335ed0cd5436142e128009c6 You can merge this pull request into a Git repository by running: $ git pull https://github.com/pudidic/hive HIVE-20044-branch-3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/396.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #396 commit edc90d42a4260272ae5abfeaef8f7966357518a1 Author: Teddy Choi Date: 2018-07-10T12:50:49Z HIVE-20044: Arrow Serde should pad char values and handle empty strings correctly (Teddy Choi) Change-Id: Iaa4417056a6e2a88335ed0cd5436142e128009c6 ---
[jira] [Created] (HIVE-20134) Improve logging when HoS Driver is killed due to exceeding memory limits
Sahil Takiar created HIVE-20134:
---
Summary: Improve logging when HoS Driver is killed due to
exceeding memory limits
Key: HIVE-20134
URL: https://issues.apache.org/jira/browse/HIVE-20134
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Sahil Takiar
This was improved in HIVE-18093, but more can be done. If a HoS Driver gets
killed because it exceeds its memory limits, YARN will issue a SIGTERM on the
process. The SIGTERM will cause the shutdown hook in the HoS Driver to be
triggered. This causes the Driver to kill all running jobs, even if they are
running. The user ends up seeing an error like the one below. Which isn't very
informative. We should propagate the error from the Driver shutdown hook to the
user.
{code:java}
INFO : 2018-07-09 17:48:42,580 Stage-64_0: 526/526 Finished Stage-65_0:
1405/1405 Finished Stage-66_0: 0(+759)/1102 Stage-67_0: 0/1099 Stage-68_0:
0/1099 Stage-69_0: 0/1
INFO : 2018-07-09 17:48:44,589 Stage-64_0: 526/526 Finished Stage-65_0:
1405/1405 Finished Stage-66_0: 1(+759)/1102 Stage-67_0: 0/1099 Stage-68_0:
0/1099 Stage-69_0: 0/1
INFO : 2018-07-09 17:48:45,591 Stage-64_0: 526/526 Finished Stage-65_0:
1405/1405 Finished Stage-66_0: 2(+759)/1102 Stage-67_0: 0/1099 Stage-68_0:
0/1099 Stage-69_0: 0/1
INFO : 2018-07-09 17:48:48,596 Stage-64_0: 526/526 Finished Stage-65_0:
1405/1405 Finished Stage-66_0: 2(+759)/1102 Stage-67_0: 0/1099 Stage-68_0:
0/1099 Stage-69_0: 0/1
ERROR : Spark job[23] failed
java.lang.InterruptedException: null
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
~[?:1.8.0_141]
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
~[?:1.8.0_141]
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:202)
~[scala-library-2.11.8.jar:?]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
~[scala-library-2.11.8.jar:?]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153)
~[scala-library-2.11.8.jar:?]
at org.apache.spark.SimpleFutureAction.ready(FutureAction.scala:125)
~[spark-core_2.11-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.spark.SimpleFutureAction.ready(FutureAction.scala:114)
~[spark-core_2.11-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.spark.util.ThreadUtils$.awaitReady(ThreadUtils.scala:222)
~[spark-core_2.11-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.spark.JavaFutureActionWrapper.getImpl(FutureAction.scala:264)
~[spark-core_2.11-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.spark.JavaFutureActionWrapper.get(FutureAction.scala:277)
~[spark-core_2.11-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:391)
~[hive-exec-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:352)
~[hive-exec-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_141]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
ERROR : FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask. null
INFO : Completed executing
command(queryId=hive_20180709174140_0f64ee17-f793-441a-9a77-3ee0cd0a9c32); Time
taken: 249.727 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.spark.SparkTask. null
(state=08S01,code=1){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20133) Cannot set fs.s3a.access.key and fs.s3a.secret.key for HiveMetastoreClient
Sandhya Agarwal created HIVE-20133: -- Summary: Cannot set fs.s3a.access.key and fs.s3a.secret.key for HiveMetastoreClient Key: HIVE-20133 URL: https://issues.apache.org/jira/browse/HIVE-20133 Project: Hive Issue Type: Bug Components: Standalone Metastore Affects Versions: 3.0.0 Reporter: Sandhya Agarwal Hello, We want to leverage standalone metastore for our project, for which I want to enable access to multiple S3 buckets, each with its own access key and secret key. I am trying to access the metastore operations from a Java client using HiveMetastoreClient and using the thrift metastore URI to connect to the metastore service. For the database location and table location, I am using the S3A file system scheme. I am trying to set the access key and secret key for the S3 bucket location in the HiveMetastoreClient, but the client side settings are never honoured and I cannot figure out a way to do this. One way is to provide the per-bucket access key and secret key on the server side in metastore-site.xml. However, I want this to be dynamic as I want the metastore to be multi tenant enabled and based on the current tenant user, I wish to provide these keys through my client. I tried looking through the metastore source code, but did not find a way to do this. Can this be done ? Thank you, Sandhya -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] hive pull request #395: HIVE-20070 : ptest optimization - Replicate ACID/MM ...
GitHub user maheshk114 opened a pull request: https://github.com/apache/hive/pull/395 HIVE-20070 : ptest optimization - Replicate ACID/MM tables write operations ⦠You can merge this pull request into a Git repository by running: $ git pull https://github.com/maheshk114/hive HIVE-20121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/395.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #395 commit 916ba47cb33c71cfaba55b1e96335e3deadbc424 Author: Mahesh Kumar Behera Date: 2018-07-10T09:25:04Z HIVE-20070 : ptest optimization - Replicate ACID/MM tables write operations ---
[jira] [Created] (HIVE-20132) External Table: Alter Table Change column is not supported.
Dileep Kumar Chiguruvada created HIVE-20132:
---
Summary: External Table: Alter Table Change column is not
supported.
Key: HIVE-20132
URL: https://issues.apache.org/jira/browse/HIVE-20132
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 3.0.0
Reporter: Dileep Kumar Chiguruvada
Fix For: 3.0.0
External Table: Alter Table Change column is not supported..
It fails with "ALTER TABLE can only be used for [ADDPROPS, DROPPROPS, ADDCOLS]
to a non-native table"
{code}
0: jdbc:hive2://ctr-e138-1518143905142-404953> alter table calcs change column
key string int;
Error: Error while compiling statement: FAILED: SemanticException [Error
10134]: ALTER TABLE can only be used for [ADDPROPS, DROPPROPS, ADDCOLS] to a
non-native table calcs (state=42000,code=10134)
{code}
This is very much required for upgraded clusters where managed tables(in 2.6
clusters) automatically converted to external tables(in 3.0.0).
One such use case is Storagehandler Tables, where we might need to alter
columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
