Review Request 67712: HIVE-19820 add ACID stats support to background stats updater
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67712/ --- Review request for hive, Eugene Koifman and Seong (Steve) Yeom. Repository: hive-git Description --- see jira Diffs - itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 580bae9c3f1307325842a08275e085a8e31f9351 ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java ddca70497a3f51c3ec9ea532fac2a42aa36149b3 ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java dd0929f2b9748d83d55ccc271cec6aa07933bde1 ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUpdaterThread.java 14f86eabbcf4bfc38c92294cd5d71d4905eb5c30 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 4296084381df1e109248820b96739a4eb5ee0490 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 51e081b22fa27b013715bb6eddf7fbbcf6bbd061 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java 9266879ad0134dbf87598af6f9305b73cc8c40ba standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java 8cc9d2c586a411712d01d599ff2986f6ad5e0cfd standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java e4894fa12bfee78f51f3796e0ccaaf51c7ac4136 standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 001c3edcff5a4d0ea67b73e83075b1f867342654 standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java d6a882e8e98f92eefbdb7900bdf43e3274a21c5d standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java c9a6a471cb7fc28845efb6d774601dba0cef2a85 Diff: https://reviews.apache.org/r/67712/diff/1/ Testing --- Thanks, Sergey Shelukhin
[jira] [Created] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table.
Steve Yeom created HIVE-19975: - Summary: Checking writeIdList per table may not check the commit level of a partition on a partitioned table. Key: HIVE-19975 URL: https://issues.apache.org/jira/browse/HIVE-19975 Project: Hive Issue Type: Sub-task Components: Transactions Affects Versions: 4.0.0 Reporter: Steve Yeom Fix For: 4.0.0 writeIdList is per table entity but stats for a partitioned table are per partition. I.e., each record in PARTITIONS has an independent stats. So if we check the validity of a partition's stats, we need to check in the context of a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19974) Show tables statement includes views and materialized views
Jesus Camacho Rodriguez created HIVE-19974: -- Summary: Show tables statement includes views and materialized views Key: HIVE-19974 URL: https://issues.apache.org/jira/browse/HIVE-19974 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Probably it would be more logical to show only the tables, since there exist 'show views' and 'show materialized views' statements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19973) Enable materialized view rewriting by default
Jesus Camacho Rodriguez created HIVE-19973:
--
Summary: Enable materialized view rewriting by default
Key: HIVE-19973
URL: https://issues.apache.org/jira/browse/HIVE-19973
Project: Hive
Issue Type: Bug
Components: Materialized views
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Change property value for {{hive.materializedview.rewriting}} to {{true}}. For
tests, it is already {{true}} by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Review Request 67710: HIVE-19481
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67710/ --- Review request for hive, Jason Dere and Sergey Shelukhin. Bugs: HIVE-19481 https://issues.apache.org/jira/browse/HIVE-19481 Repository: hive-git Description --- sample10.q returns wrong results. Multiple issues were fixed 1. Instead of using old MR logic which assumes there is 1 file for each bucket, lookup buckets by name(non-managed tables) 2. Skip bucket pruning for managed tables. Diffs - ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 9dbd869d57 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SamplePruner.java 8200e6a237 ql/src/test/queries/clientpositive/sample10_mm.q PRE-CREATION ql/src/test/results/clientpositive/llap/sample10.q.out 1b95314980 ql/src/test/results/clientpositive/llap/sample10_mm.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/sample10.q.out ac28779591 Diff: https://reviews.apache.org/r/67710/diff/1/ Testing --- Thanks, Deepak Jaiswal
[jira] [Created] (HIVE-19972) Followup to HIVE-19928 : Fix the check for managed table
Deepak Jaiswal created HIVE-19972: - Summary: Followup to HIVE-19928 : Fix the check for managed table Key: HIVE-19972 URL: https://issues.apache.org/jira/browse/HIVE-19972 Project: Hive Issue Type: Bug Reporter: Deepak Jaiswal Assignee: Deepak Jaiswal The check for managed table should use ENUM comparison rather than string comparison. The check in the patch will always return false, thus maintaining existing behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19971) TestRuntimeStats.testCleanup() is flaky
Andrew Sherman created HIVE-19971: - Summary: TestRuntimeStats.testCleanup() is flaky Key: HIVE-19971 URL: https://issues.apache.org/jira/browse/HIVE-19971 Project: Hive Issue Type: Task Reporter: Andrew Sherman Assignee: Andrew Sherman This test is timing dependent and sometimes fails. [You can see that it sometimes fails in otherwise clean runs|https://issues.apache.org/jira/issues/?jql=text%20~%20%22TestRuntimeStats%22]. The test inserts a stat, sleeps for 2 seconds, inserts another stat, then deletes stats that are older than 1 second. The test asserts that exactly one stat is deleted. If the deletion is slow for some reason (perhaps a GC?) then 2 stats will be deleted and the test will fail. The trouble is that the 1 second window is too small to work consistently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 67636: HIVE-19176 : Add HoS support to progress bar on Beeline client.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67636/ --- (Updated June 22, 2018, 8:01 p.m.) Review request for hive, Peter Vary, Sahil Takiar, and Vihang Karajgaonkar. Repository: hive-git Description --- This logic is similar to the RenderStrategy used in Tez to print the progress bar. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7f5fc0c6a3d527671c354db8ef2c9772aab6dd0 jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java ad8d1a7f1cca3a763bb7c07335998ab7d39d7598 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/LocalSparkJobMonitor.java 2a6c33bfd4824c96e7004cd1ecce48c62c97d685 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java 004b50ba95934280cf302055a46a5d984b421e07 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RenderStrategy.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 3531ac25a9959aacd5766a9a42316890c68a1cd5 ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java 368fa9f1fa3ccc8b78cc4f9e98acf352cbc1c4c3 ql/src/test/org/apache/hadoop/hive/ql/exec/spark/status/TestSparkJobMonitor.java e66354f0869738bd3cf0eb831c13fa6af1eda256 service/src/java/org/apache/hive/service/ServiceUtils.java 226e43244df10c22143b91f92ef312e56739d036 service/src/java/org/apache/hive/service/cli/SparkProgressMonitorStatusMapper.java PRE-CREATION service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 68fe8d8aa143fafbfc611253ce3a12065016a537 Diff: https://reviews.apache.org/r/67636/diff/3/ Changes: https://reviews.apache.org/r/67636/diff/2-3/ Testing --- Thanks, Bharathkrishna Guruvayoor Murali
[jira] [Created] (HIVE-19970) Replication dump has a NPE when table is empty
mahesh kumar behera created HIVE-19970: -- Summary: Replication dump has a NPE when table is empty Key: HIVE-19970 URL: https://issues.apache.org/jira/browse/HIVE-19970 Project: Hive Issue Type: Task Components: repl Affects Versions: 3.1.0, 4.0.0 Reporter: mahesh kumar behera Assignee: mahesh kumar behera Fix For: 3.1.0, 4.0.0 if table directory or partition directory is missing ..dump is throwing NPE instead of file missing exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] hive pull request #378: HIVE-19316: StatsTask fails due to ClassCastExceptio...
GitHub user beltran opened a pull request: https://github.com/apache/hive/pull/378 HIVE-19316: StatsTask fails due to ClassCastException You can merge this pull request into a Git repository by running: $ git pull https://github.com/beltran/hive HIVE-19316 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/378.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #378 commit a9566b22761aa1da14585ce829e6d65f9d272f48 Author: Jaume Marhuenda Date: 2018-06-22T17:46:35Z HIVE-19316: StatsTask fails due to ClassCastException ---
Re: Review Request 67437: HIVE-19649: Clean up inputs in JDBC PreparedStatement. Add unit tests.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67437/ --- (Updated June 22, 2018, 5:09 p.m.) Review request for hive, Ashutosh Chauhan, Eugene Koifman, and Zoltan Haindrich. Repository: hive-git Description --- Initial commit. Diffs (updated) - jdbc/src/test/org/apache/hive/jdbc/TestHivePreparedStatement.java 2a68c91 Diff: https://reviews.apache.org/r/67437/diff/3/ Changes: https://reviews.apache.org/r/67437/diff/2-3/ Testing --- Thanks, Mykhailo Kysliuk
Re: Review Request 67437: HIVE-19649: Clean up inputs in JDBC PreparedStatement. Add unit tests.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67437/ --- (Updated June 22, 2018, 5:04 p.m.) Review request for hive, Ashutosh Chauhan, Eugene Koifman, and Zoltan Haindrich. Repository: hive-git Description --- Initial commit. Diffs (updated) - jdbc/src/test/org/apache/hive/jdbc/TestHivePreparedStatement.java 2a68c91 Diff: https://reviews.apache.org/r/67437/diff/2/ Changes: https://reviews.apache.org/r/67437/diff/1-2/ Testing --- Thanks, Mykhailo Kysliuk
Re: [ANNOUNCE] New committer: Adam Szita
Thank you all! On 21 June 2018 at 19:11, Andrew Sherman wrote: > Yay Adam! > > On Thu, Jun 21, 2018 at 7:27 AM Sahil Takiar > wrote: > > > Congrats Adam! > > > > On Thu, Jun 21, 2018 at 2:52 AM, Peter Vary > > wrote: > > > > > Well done Adam! > > > Congratulations! > > > > > > > On Jun 20, 2018, at 21:02, Ashutosh Chauhan > > > wrote: > > > > > > > > Apache Hive's Project Management Committee (PMC) has invited Adam > Szita > > > > to become a committer, and we are pleased to announce that he has > > > accepted. > > > > > > > > Adam, welcome, thank you for your contributions, and we look forward > > your > > > > further interactions with the community! > > > > > > > > Ashutosh Chauhan (on behalf of the Apache Hive PMC) > > > > > > > > > > > > -- > > Sahil Takiar > > Software Engineer > > takiar.sa...@gmail.com | (510) 673-0309 > > >
[jira] [Created] (HIVE-19969) Dependency order (dirlist) assessment fails in yetus run
Adam Szita created HIVE-19969:
-
Summary: Dependency order (dirlist) assessment fails in yetus run
Key: HIVE-19969
URL: https://issues.apache.org/jira/browse/HIVE-19969
Project: Hive
Issue Type: Bug
Reporter: Adam Szita
Assignee: Adam Szita
As seen here, the dirlist step of yetus fails to determine order of modules to
be built. It silently falls back to alphabetical order which may or may not
work depending on the patch.
{code:java}
Thu Jun 21 02:43:04 UTC 2018
cd /data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958
mvn -q exec:exec -Dexec.executable=pwd -Dexec.args=''
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/storage-api
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/upgrade-acid
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/classification
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/shims/common
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/shims/0.23
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/shims/scheduler
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/shims/aggregator
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/common
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/service-rpc
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11958/serde
Usage: java [-options] class [args...]
(to execute a class)
or java [-options] -jar jarfile [args...]
(to execute a jar file)
where options include:{code}
The problem is in standalone-metastore module: maven plugin 'exec' has a global
config set {{executable=java}} disregarding the dirlist task's
{{-Dexec.executable=pwd}} and causing the above error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-19968) UDF exception is not throw out
sandflee created HIVE-19968:
---
Summary: UDF exception is not throw out
Key: HIVE-19968
URL: https://issues.apache.org/jira/browse/HIVE-19968
Project: Hive
Issue Type: Bug
Reporter: sandflee
udf init failed, and throw a exception, but hive catch it and do nothing,
leading to app succ, but no data is generated.
GenericUDFReflect.java#evaluate()
try {
o = null;
o = ReflectionUtils.newInstance(c, null);
} catch (Exception e) {
// ignored
}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Set up kerberos for Hadoop error
Hello, everyone After setting up kerberos for Hadoop, but I keep getting the error when I run hadoop dfs -ls /, I searched a lot on the internet and still couldn’t resolve it, does anyone give some advice? 18/06/22 07:57:53 WARN ipc.Client: Couldn't setup connection for postg...@cw.com to hou-1/10.0.0.6:9000 javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758) at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618) at org.apache.hadoop.ipc.Client.call(Client.java:1449) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158) at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423) at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1435) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:265) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:297) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:350) Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 41 more Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73) at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126) at sun.security.krb5.Credentials.acquireServiceCreds(Crede
Review Request 67698: HIVE-19967
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67698/ --- Review request for hive, Gunther Hagleitner and Jason Dere. Bugs: HIVE-19967 https://issues.apache.org/jira/browse/HIVE-19967 Repository: hive-git Description --- SMB Join : Need Optraits for PTFOperator ala GBY Op Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/AnnotateWithOpTraits.java 3c8e61d47b ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java dbcbbfd1a6 ql/src/test/queries/clientpositive/llap_smb_ptf.q PRE-CREATION ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out PRE-CREATION Diff: https://reviews.apache.org/r/67698/diff/1/ Testing --- Thanks, Deepak Jaiswal
