[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257474#comment-13257474 ] Thomas Weise commented on HIVE-2646: The patch needs to be applied through arc: arc patch D2883 chmod u+x testutils/hadoop svn propset svn:executable yes testutils/hadoop Al least for me, arc patch will choke on ivy/common-configurations.xml - download the file from phabricator directly. I had the problem with Andrew's patch also, so I'm not sure whether this is just me or an arc issue. Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Assignee: Andrew Bayer Priority: Critical Fix For: 0.9.0 Attachments: HIVE-2646-fixtests.patch, HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, HIVE-2646.D2133.15.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, HIVE-2646.D2883.1.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257833#comment-13257833 ] Thomas Weise commented on HIVE-2646: All test pass in latest run: Tests FailuresErrors Success rateTime 16780 0 100.00% 19245.326 Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Assignee: Andrew Bayer Priority: Critical Fix For: 0.9.0 Attachments: HIVE-2646-fixtests.patch, HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, HIVE-2646.D2133.15.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, HIVE-2646.D2883.1.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2740) ant clean package -Dshims.include=0.23 results in compilation errors
[
https://issues.apache.org/jira/browse/HIVE-2740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192315#comment-13192315
]
Thomas Weise commented on HIVE-2740:
This should be enough:
{{ant clean package -Dshims.include=0.23 -Dhadoop.version=0.23.0}}
Since you are only compiling the .23 shim other Hadoop versions won't be
fetched and you need to specify hadoop.version to compile the common code.
ant clean package -Dshims.include=0.23 results in compilation errors
Key: HIVE-2740
URL: https://issues.apache.org/jira/browse/HIVE-2740
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.8.1
Reporter: Amareshwari Sriramadasu
ant clean package -Dshims.include=0.23 results in compilation errors for
component common.
For ex:
{noformat}
[echo] Project: common
[javac] Compiling 15 source files to
/home/amarsri/workspace/hive/build/common/classes
[javac]
/home/amarsri/workspace/hive/common/src/java/org/apache/hadoop/hive/common/FileUtils.java:26:
package org.apache.hadoop.conf does not exist
[javac] import org.apache.hadoop.conf.Configuration;
[javac] ^
{noformat}
Thomas, Can you look into it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2609) NPE when pruning partitions by thrift method get_partitions_by_filter
[
https://issues.apache.org/jira/browse/HIVE-2609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192699#comment-13192699
]
Thomas Weise commented on HIVE-2609:
We replace them after the build.
NPE when pruning partitions by thrift method get_partitions_by_filter
-
Key: HIVE-2609
URL: https://issues.apache.org/jira/browse/HIVE-2609
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.7.1
Reporter: Min Zhou
It's a datanucleus bug indeed.
try this code:
{code}
boolean open = false;
for (int i = 0; i 5 !open; ++i) {
try {
transport.open();
open = true;
} catch (TTransportException e) {
System.out.println(failed to connect to MetaStore, re-trying...);
try {
Thread.sleep(1000);
} catch (InterruptedException ignore) {}
}
}
try {
ListPartition parts =
client.get_partitions_by_filter(default, partitioned_nation,
pt '2', (short) -1);
for (Partition part : parts) {
System.out.println(part.getSd().getLocation());
}
} catch (Exception te) {
te.printStackTrace();
}
{code}
A NPEexception would be thrown on the thrift server side
{noformat}
11/11/25 13:11:55 ERROR api.ThriftHiveMetastore$Processor: Internal error
processing get_partitions_by_filter
java.lang.NullPointerException
at
org.datanucleus.store.mapped.mapping.MappingHelper.getMappingIndices(MappingHelper.java:35)
at
org.datanucleus.store.mapped.expression.StatementText.applyParametersToStatement(StatementText.java:194)
at
org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:233)
at
org.datanucleus.store.rdbms.query.legacy.SQLEvaluator.evaluate(SQLEvaluator.java:115)
at
org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.performExecute(JDOQLQuery.java:288)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1657)
at
org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
at org.datanucleus.store.query.Query.executeWithMap(Query.java:1526)
at org.datanucleus.jdo.JDOQuery.executeWithMap(JDOQuery.java:334)
at
org.apache.hadoop.hive.metastore.ObjectStore.listMPartitionsByFilter(ObjectStore.java:1329)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilter(ObjectStore.java:1241)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2369)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2366)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_filter(HiveMetaStore.java:2366)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_filter.process(ThriftHiveMetastore.j
ava:6099)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor.process(ThriftHiveMetastore.java:4789)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$TLoggingProcessor.process(HiveMetaStore.java:3167)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
A null JavaTypeMapping was passed into
org.datanucleus.store.mapped.mapping.MappingHelper.(int initialPosition,
JavaTypeMapping mapping), that caused NPE.
After digged into the datanucleus source, I found that the null value was
born in the constructor of
org.datanucleus.store.mapped.expression.SubstringExpression. see
{code}
/**
* Constructs the substring
* @param str the String Expression
* @param begin The start position
* @param end The end position expression
**/
public SubstringExpression(StringExpression str, NumericExpression begin,
NumericExpression end)
{
super(str.getQueryExpression());
st.append(SUBSTRING().append(str).append( FROM )
.append(begin.add(new IntegerLiteral(qs, mapping,
BigInteger.ONE)))
.append( FOR ).append(end.sub(begin)).append(')');
}
{code}
The field mapping hasn't been instanced at that moment.
How do you deal with such a external bug?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (HIVE-2719) Fix test failures caused by HIVE-2589
[ https://issues.apache.org/jira/browse/HIVE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187901#comment-13187901 ] Thomas Weise commented on HIVE-2719: I observed inconsistent results before r1230390. On the same machine I had a run where all tests passed and one where a few failed. The few that failed passed when running them separately. Fix test failures caused by HIVE-2589 - Key: HIVE-2719 URL: https://issues.apache.org/jira/browse/HIVE-2719 Project: Hive Issue Type: Task Reporter: Carl Steinbach Assignee: Ashutosh Chauhan Priority: Blocker 149 new test failures appeared on Hudson after HIVE-2589 and HIVE-2682 were committed: https://builds.apache.org/job/Hive-trunk-h0.21/1195/ @Ashutosh: Can you please take a look? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[
https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185168#comment-13185168
]
Thomas Weise commented on HIVE-2698:
As for the following added dependency:
dependency org=org.slf4j name=slf4j-api rev=${slf4j-api.version}/
This already comes with Hadoop? If so, we should reference it accordingly.
Please see HCATALOG-201
Enable Hadoop-1.0.0 in Hive
---
Key: HIVE-2698
URL: https://issues.apache.org/jira/browse/HIVE-2698
Project: Hive
Issue Type: New Feature
Components: Security, Shims
Affects Versions: 0.9.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Attachments: HIVE-2698_v1.patch, HIVE-2698_v2.patch
Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the
0.20S release.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185258#comment-13185258 ] Thomas Weise commented on HIVE-2698: That's odd, all other versions have it: $ find . -name slf4j*.* ./hadoop-0.20.1/lib/slf4j-api-1.4.3.jar ./hadoop-0.20.1/lib/slf4j-log4j12-1.4.3.jar ./hadoop-0.20.3-CDH3-SNAPSHOT/lib/slf4j-api-1.4.3.jar ./hadoop-0.20.3-CDH3-SNAPSHOT/lib/slf4j-log4j12-1.4.3.jar ./hadoop-0.23.0/lib/slf4j-api-1.6.1.jar ./hadoop-0.23.0/lib/slf4j-log4j12-1.6.1.jar ./hadoop-0.23.0/share/hadoop/common/lib/slf4j-api-1.5.11.jar ./hadoop-0.23.0/share/hadoop/common/lib/slf4j-log4j12-1.5.11.jar ./hadoop-0.23.0/share/hadoop/hdfs/lib/slf4j-api-1.5.11.jar ./hadoop-0.23.0/share/hadoop/hdfs/lib/slf4j-log4j12-1.5.11.jar Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2698_v1.patch, HIVE-2698_v2.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183359#comment-13183359 ] Thomas Weise commented on HIVE-2629: On head revision: ant clean package test -Dtestcase=TestCliDriver -Dqfile=combine2.q -Dtest.silent=false -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23 Does not even compile. I remember though that some tests on 0.23 failed, before and after the patch as noted earlier. Fixing all unit tests for 0.23 is not part of this patch. Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach Assignee: Thomas Weise Fix For: 0.8.1, 0.9.0 Attachments: HIVE-2629.3.patch, HIVE-2629.D711.1.patch, HIVE-2629.D711.2.patch, HIVE-2629.D711.3.patch, HIVE-2629.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183379#comment-13183379 ] Thomas Weise commented on HIVE-2629: Another run on one of our boxes completed overnight with no failures - all tests pass for 0.20. BUILD SUCCESSFUL Total time: 289 minutes 12 seconds [1]+ Doneant test test.log 21 This JIRA's purpose is not to fix all unit tests for 0.23, but I had checked that not more test fail than there were already. Per previous comment, currently the head revision does not compile for 0.23, but I went back to a prior revision and it shows that the test mentioned above was broken before: svn --revision 1220920 up ant clean package test -Dtestcase=TestCliDriver -Dqfile=combine2.q -Dtest.silent=false -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23 BUILD FAILED /Users/thw/java/hive/trunk2/build.xml:310: Keepgoing execution: 1 of 12 iterations failed. Total time: 2 minutes 47 seconds Can we please create a new JIRA for making broken tests on 0.23 work and resolve this one? Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach Assignee: Thomas Weise Fix For: 0.8.1, 0.9.0 Attachments: HIVE-2629.3.patch, HIVE-2629.D711.1.patch, HIVE-2629.D711.2.patch, HIVE-2629.D711.3.patch, HIVE-2629.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183483#comment-13183483 ] Thomas Weise commented on HIVE-2698: We should treat tarball structure and shims as separate issues. As you mentioned, a few tweaks may be required for the build classpath, just as for 0.23). The tarball stuff is a hack with known issues, at some point we need to use ivy/maven (see Pig project, for example). As long as the compiled binary works on both, 0.20.205 and 1.0.0, I see no compelling reason to build an extra shim. Was that not addressed by HIVE-2631? Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Attachments: HIVE-2698_v1.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183518#comment-13183518 ] Thomas Weise commented on HIVE-2698: +1 for changing the default version used to build 0.20S shim from 0.20.3-CDH3 to 1.0.0, let's see what others say. As stated above, the default version does not prevent you from using an override on the ant command line. Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Attachments: HIVE-2698_v1.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182972#comment-13182972 ] Thomas Weise commented on HIVE-2629: I had run the tests successfully when I first submitted the patch. I did so again today on two different machines. There was one failure in each run (in different test case) which each passed on the other machine as well as on re-run on the same machine. Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach Assignee: Thomas Weise Fix For: 0.8.1, 0.9.0 Attachments: HIVE-2629.3.patch, HIVE-2629.D711.1.patch, HIVE-2629.D711.2.patch, HIVE-2629.D711.3.patch, HIVE-2629.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183043#comment-13183043 ] Thomas Weise commented on HIVE-2698: Since 1.0.0 does not require a new shim, it should be sufficient to update the version used to compile the 20S shim. -hadoop-0.20S.version=0.20.3-CDH3-SNAPSHOT +hadoop-0.20S.version=1.0.0 Or, on the ant command line, add -Dhadoop-0.20S.version=1.0.0 The hive build as is should run on 1.0.0. Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Attachments: HIVE-2698_v1.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176810#comment-13176810 ] Thomas Weise commented on HIVE-2629: bump Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach Assignee: Thomas Weise Fix For: 0.9.0 Attachments: HIVE-2629.D711.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2665) Support for metastore specific HADOOP_OPTS environment setting
[ https://issues.apache.org/jira/browse/HIVE-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173454#comment-13173454 ] Thomas Weise commented on HIVE-2665: Would something like this be acceptable? Index: bin/ext/metastore.sh === --- bin/ext/metastore.sh(revision 1205535) +++ bin/ext/metastore.sh(working copy) @@ -26,6 +26,7 @@ # hadoop 20 or newer - skip the aux_jars option and hiveconf + export HADOOP_OPTS=$HIVE_METASTORE_HADOOP_OPTS $HADOOP_OPTS exec $HADOOP jar $JAR $CLASS $@ } Support for metastore specific HADOOP_OPTS environment setting -- Key: HIVE-2665 URL: https://issues.apache.org/jira/browse/HIVE-2665 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Thomas Weise Assignee: Thomas Weise Priority: Minor For development/testing it would be helpful to have a way to define HADOOP_OPTS that apply only to a specific launcher and don't affect everything else launched through bin/hadoop. In this specific case I'm looking for a way to set metastore JVM debug options w/o modifying the HADOOP_OPTS environment setting or the hive scripts (which are replaced on every build). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168793#comment-13168793 ] Thomas Weise commented on HIVE-2629: As indicated, unit tests pass (or status unchanged with respect to 0.23). Review/comments anyone? Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach Assignee: Thomas Weise Attachments: HIVE-2629.D711.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166642#comment-13166642 ] Thomas Weise commented on HIVE-2629: Unit test results: ant package test All tests pass for default hadoop versions. ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dshims.include=0.23 clean package test testreport 86% tests pass. Same result as current trunk, tests run with: ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23 package test testreport Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach Assignee: Thomas Weise Attachments: HIVE-2629.D711.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2486) Phabricator for code review
[ https://issues.apache.org/jira/browse/HIVE-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165726#comment-13165726 ] Thomas Weise commented on HIVE-2486: What would be a more appropriate place to report this? arc has a problem with deleted files in subdirectories: $ svn stat $ svn del shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java D shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java $ arc diff Linting... Exception: Filesystem entity `/Users/thw/java/hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java' does not exist. (Run with --trace for a full exception trace.) $ Phabricator for code review --- Key: HIVE-2486 URL: https://issues.apache.org/jira/browse/HIVE-2486 Project: Hive Issue Type: Bug Reporter: Marek Sapota Assignee: Marek Sapota Fix For: 0.8.0, 0.9.0 Attachments: D135.1.patch, D135.2.patch, D387.1.patch, D51.1.patch, D51.2.patch, D51.2.patch, D57.1.patch, D57.2.patch, D63.1.patch, D63.2.patch, D63.3.patch, D69.1.patch, D69.2.patch, D69.3.patch, D75.1.patch, D75.2.patch, D9.1.diff, D9.2.diff, HIVE-2486.D387.2.patch, SOAP, SOAP, TEST, x Replacing Review Board with Phabricator and Differential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164607#comment-13164607 ] Thomas Weise commented on HIVE-2629: Looking at this in the context of HCATALOG-179. As of now 0.23 shim is incomplete (does not support security, authentication code is not included). I'm looking into restructuring the secure shim code so that 20S and 23 share common code and only redefine what is unique to 20S vs. 23. Do we want all 3 shims to be compiled as part of the build? Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160178#comment-13160178 ] Thomas Weise commented on HIVE-2467: All unit tests pass with the patch. HA Support for Metastore Server Key: HIVE-2467 URL: https://issues.apache.org/jira/browse/HIVE-2467 Project: Hive Issue Type: Improvement Components: Metastore, Security, Server Infrastructure Affects Versions: 0.8.0, 0.9.0 Reporter: Thomas Weise Assignee: Thomas Weise Fix For: 0.9.0 Attachments: HIVE-2467.2.patch, HIVE-2467.patch We require HA deployment for metastore server for HCatalog: * Multiple server instances run behind VIP * Database provides HA Metastore server instances will need to be able to share any state required for VIP outside RDBMS. As of Hive 0.8 affected conversational state that needs to support VIP/HA setup is limited to current delegation tokens. Is this correct? We are planning to use ZooKeeper to share current delegation tokens and master keys between nodes of the VIP. ZK is already (optionally) used by Hive for concurrency control. Access to ZK would be limited on the network level or in the future, when ZooKeeper supports security, through Kerberos, similar to NN access. Currently Hive taps into Hadoop core security delegation token support through extension of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManagerTokenIdent A solution could amend the Hive specific extension to support: * Pluggable delegation token and master key store (ZooKeeper as alternative for in-memory AbstractDelegationTokenSecretManager) * Delegation token retrieval from token store when not found in memory (wrap/extend retrievePassword(...)) * Cancellation of token in token store * Purging of expired tokens from token store http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2468) Make Hive compile against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160209#comment-13160209 ] Thomas Weise commented on HIVE-2468: I assume (based on findings in linked Pig issue) that when Hive is built against .23 then it won't be byte code compatible with .20 and so even though the shim is present cannot run on .20? Make Hive compile against Hadoop 0.23 - Key: HIVE-2468 URL: https://issues.apache.org/jira/browse/HIVE-2468 Project: Hive Issue Type: Task Reporter: Konstantin Shvachko Assignee: Tom White Attachments: HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2468) Make Hive compile against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153009#comment-13153009 ] Thomas Weise commented on HIVE-2468: The proposed change would duplicate package/class names from 20S shim. That's problematic as all go into a single shim jar file. I would suggest to use a different package (or rename all classes) for the .23 shim. In those cases where classes from 20S are copied w/o modification, we should try to separate them out to a common location that is shared between 20S and 23? Make Hive compile against Hadoop 0.23 - Key: HIVE-2468 URL: https://issues.apache.org/jira/browse/HIVE-2468 Project: Hive Issue Type: Task Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: HIVE-2468.patch Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13151690#comment-13151690 ] Thomas Weise commented on HIVE-2467: @Ashutosh: I will file JIRA to incorporate relevant changes to Hadoop trunk so the next shim version (.23 and later) can use it from there (without the various tricks to make this work for .20). I would suggest to address the TokenStore first class interface as part of that patch? The logic that puts tokens into the cache and removes them immediately is necessary due to field/method visibility issues in the Hadoop code. All that won't be needed when we prepare a Hadoop patch. I added a few comments in the code. HA Support for Metastore Server Key: HIVE-2467 URL: https://issues.apache.org/jira/browse/HIVE-2467 Project: Hive Issue Type: Improvement Components: Metastore, Security, Server Infrastructure Affects Versions: 0.8.0, 0.9.0 Reporter: Thomas Weise Assignee: Thomas Weise Fix For: 0.9.0 Attachments: HIVE-2467.2.patch, HIVE-2467.patch We require HA deployment for metastore server for HCatalog: * Multiple server instances run behind VIP * Database provides HA Metastore server instances will need to be able to share any state required for VIP outside RDBMS. As of Hive 0.8 affected conversational state that needs to support VIP/HA setup is limited to current delegation tokens. Is this correct? We are planning to use ZooKeeper to share current delegation tokens and master keys between nodes of the VIP. ZK is already (optionally) used by Hive for concurrency control. Access to ZK would be limited on the network level or in the future, when ZooKeeper supports security, through Kerberos, similar to NN access. Currently Hive taps into Hadoop core security delegation token support through extension of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManagerTokenIdent A solution could amend the Hive specific extension to support: * Pluggable delegation token and master key store (ZooKeeper as alternative for in-memory AbstractDelegationTokenSecretManager) * Delegation token retrieval from token store when not found in memory (wrap/extend retrievePassword(...)) * Cancellation of token in token store * Purging of expired tokens from token store http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149742#comment-13149742 ] Thomas Weise commented on HIVE-2467: Yes, this was the intention. Also, the current ZK token store implementation could be translated into a SQL implementation with relatively small effort (it does not depend on ZK watches for invalidation). For our use, we plan to deal with the security limitation at L3 and use ZK security when it becomes available. Longer term, assuming there will be use cases outside HCatalog/Hive for long running processes that require token support in conjunction with HA but without HA RDBMS already available, a ZK based token storage might be a more likely choice. While plain HDFS is secure it does not support the access pattern needed here (fast random reads and writes). HA Support for Metastore Server Key: HIVE-2467 URL: https://issues.apache.org/jira/browse/HIVE-2467 Project: Hive Issue Type: Improvement Components: Metastore, Security, Server Infrastructure Affects Versions: 0.8.0, 0.9.0 Reporter: Thomas Weise Assignee: Thomas Weise Fix For: 0.9.0 Attachments: HIVE-2467.patch We require HA deployment for metastore server for HCatalog: * Multiple server instances run behind VIP * Database provides HA Metastore server instances will need to be able to share any state required for VIP outside RDBMS. As of Hive 0.8 affected conversational state that needs to support VIP/HA setup is limited to current delegation tokens. Is this correct? We are planning to use ZooKeeper to share current delegation tokens and master keys between nodes of the VIP. ZK is already (optionally) used by Hive for concurrency control. Access to ZK would be limited on the network level or in the future, when ZooKeeper supports security, through Kerberos, similar to NN access. Currently Hive taps into Hadoop core security delegation token support through extension of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManagerTokenIdent A solution could amend the Hive specific extension to support: * Pluggable delegation token and master key store (ZooKeeper as alternative for in-memory AbstractDelegationTokenSecretManager) * Delegation token retrieval from token store when not found in memory (wrap/extend retrievePassword(...)) * Cancellation of token in token store * Purging of expired tokens from token store http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13148752#comment-13148752 ] Thomas Weise commented on HIVE-2467: That is correct. The ZooKeeper installation that will be used for this purpose will have to be secured on layer 3 until we have security available in ZK (which is also coming). We are aware of and consider this a limitation that will be addressed by ZK security, not by the client using ZK. HA Support for Metastore Server Key: HIVE-2467 URL: https://issues.apache.org/jira/browse/HIVE-2467 Project: Hive Issue Type: Improvement Components: Metastore, Security, Server Infrastructure Affects Versions: 0.8.0, 0.9.0 Reporter: Thomas Weise Assignee: Thomas Weise Fix For: 0.9.0 Attachments: HIVE-2467.patch We require HA deployment for metastore server for HCatalog: * Multiple server instances run behind VIP * Database provides HA Metastore server instances will need to be able to share any state required for VIP outside RDBMS. As of Hive 0.8 affected conversational state that needs to support VIP/HA setup is limited to current delegation tokens. Is this correct? We are planning to use ZooKeeper to share current delegation tokens and master keys between nodes of the VIP. ZK is already (optionally) used by Hive for concurrency control. Access to ZK would be limited on the network level or in the future, when ZooKeeper supports security, through Kerberos, similar to NN access. Currently Hive taps into Hadoop core security delegation token support through extension of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManagerTokenIdent A solution could amend the Hive specific extension to support: * Pluggable delegation token and master key store (ZooKeeper as alternative for in-memory AbstractDelegationTokenSecretManager) * Delegation token retrieval from token store when not found in memory (wrap/extend retrievePassword(...)) * Cancellation of token in token store * Purging of expired tokens from token store http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
