[jira] [Created] (HADOOP-9478) The get operation of deprecatedKeyMap of org.apache.hadoop.conf.Configuration should be synchronized.
dong created HADOOP-9478:
Summary: The get operation of deprecatedKeyMap of
org.apache.hadoop.conf.Configuration should be synchronized.
Key: HADOOP-9478
URL: https://issues.apache.org/jira/browse/HADOOP-9478
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Environment: OS:
CentOS release 6.3 (Final)
JDK:
java version 1.6.0_27
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
Hadoop:
hadoop-2.0.0-cdh4.1.3/hadoop-2.0.0-cdh4.2.0
Security:
Kerberos
Reporter: dong
When we lanuch the client appliation which use kerberos security,the FileSystem
can't be create because the exception ' java.lang.NoClassDefFoundError: Could
not initialize class org.apache.hadoop.security.SecurityUtil'.
I check the exception stack trace,it maybe caused by the unsafe get operation
of the deprecatedKeyMap which used by the org.apache.hadoop.conf.Configuration.
So I write a simple test case:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.hdfs.HdfsConfiguration;
public class HTest {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.addResource(core-site.xml);
conf.addResource(hdfs-site.xml);
FileSystem fileSystem = FileSystem.get(conf);
System.out.println(fileSystem);
System.exit(0);
}
}
Then I launch this test case many times,the following exception is thrown:
Exception in thread TGT Renewer for XXX java.lang.ExceptionInInitializerError
at
org.apache.hadoop.security.UserGroupInformation.getTGT(UserGroupInformation.java:719)
at
org.apache.hadoop.security.UserGroupInformation.access$1100(UserGroupInformation.java:77)
at
org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:746)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 16
at java.util.HashMap.getEntry(HashMap.java:345)
at java.util.HashMap.containsKey(HashMap.java:335)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1989)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1867)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1785)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:712)
at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:731)
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1047)
at org.apache.hadoop.security.SecurityUtil.clinit(SecurityUtil.java:76)
... 4 more
Exception in thread main java.io.IOException: Couldn't create proxy provider
class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
at
org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:453)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:133)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:436)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:403)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:125)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2262)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2296)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
at HTest.main(HTest.java:11)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:442)
... 11 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.security.SecurityUtil
at
org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:231)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:159)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
at
org.apache.hadoop.hdfs.DFSUtil.getAddressesForNameserviceId(DFSUtil.java:452)
at org.apache.hadoop.hdfs.DFSUtil.getAddresses(DFSUtil.java:434)
at
Re: [VOTE] Release Apache Hadoop 0.23.7
+1 (binding) I downloaded the release and ran a few sanity tests on it. --Bobby On 4/11/13 2:55 PM, Thomas Graves tgra...@yahoo-inc.com wrote: I've created a release candidate (RC0) for hadoop-0.23.7 that I would like to release. This release is a sustaining release with several important bug fixes in it. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Tom Graves
[jira] [Reopened] (HADOOP-9473) typo in FileUtil copy() method
[
https://issues.apache.org/jira/browse/HADOOP-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao reopened HADOOP-9473:
---
We also needs to fix the same typo in testConf.xml accordingly. Otherwise
TestCLI will fail.
typo in FileUtil copy() method
--
Key: HADOOP-9473
URL: https://issues.apache.org/jira/browse/HADOOP-9473
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.0-alpha, 1.1.2
Reporter: Glen Mazza
Priority: Trivial
Fix For: 1.2.0, 2.0.5-beta
Attachments: HADOOP-9473.branch-1.patch, HADOOP-9473.patch
typo:
{code}
Index: src/core/org/apache/hadoop/fs/FileUtil.java
===
--- src/core/org/apache/hadoop/fs/FileUtil.java (revision 1467295)
+++ src/core/org/apache/hadoop/fs/FileUtil.java (working copy)
@@ -178,7 +178,7 @@
// Check if dest is directory
if (!dstFS.exists(dst)) {
throw new IOException(` + dst +': specified destination directory
+
-doest not exist);
+does not exist);
} else {
FileStatus sdst = dstFS.getFileStatus(dst);
if (!sdst.isDir())
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 0.23.7
+1 (binding) Verified sigs, ran sample jobs on a single-node cluster. Arun On Apr 12, 2013, at 1:25 AM, Thomas Graves wrote: I've created a release candidate (RC0) for hadoop-0.23.7 that I would like to release. This release is a sustaining release with several important bug fixes in it. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Tom Graves -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: [VOTE] Release Apache Hadoop 0.23.7
+1 (binding) - Verified signatures and checksums - Installed single-node cluster from binary tarball and ran sample jobs - Built from source, installed single-node cluster from resulting binaries, and ran sample jobs Jason On 04/11/2013 02:55 PM, Thomas Graves wrote: I've created a release candidate (RC0) for hadoop-0.23.7 that I would like to release. This release is a sustaining release with several important bug fixes in it. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Tom Graves
Re: [VOTE] Release Apache Hadoop 0.23.7
+1. Downloaded source, built and ran a couple of sample jobs on a single node cluster. -- Hitesh On Apr 11, 2013, at 12:55 PM, Thomas Graves wrote: I've created a release candidate (RC0) for hadoop-0.23.7 that I would like to release. This release is a sustaining release with several important bug fixes in it. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Tom Graves
[jira] [Created] (HADOOP-9479) Ability to plugin custom authentication mechanism based on Jaas and Sasl
Benoy Antony created HADOOP-9479: Summary: Ability to plugin custom authentication mechanism based on Jaas and Sasl Key: HADOOP-9479 URL: https://issues.apache.org/jira/browse/HADOOP-9479 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Benoy Antony Assignee: Benoy Antony Currently, it is not possible to hookup new/modified authentication mechanism to Hadoop. The task is to create an extension in hadoop to plugin new Authentication mechanism. The new authentication mechanism should have both Jaas and Sasl implementations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 2.0.4-alpha
+1 (non-binding) Built from source and ran sample jobs concurrently with the fair scheduler on a single node cluster. On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would like to release. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: [VOTE] Release Apache Hadoop 2.0.4-alpha
+1 Built from source successfully, verified signatures, stood up a 1-node cluster with CS, ran one Pi MR job, and the DistributedShell application. On Wed, Apr 17, 2013 at 6:10 AM, Sandy Ryza sandy.r...@cloudera.com wrote: +1 (non-binding) Built from source and ran sample jobs concurrently with the fair scheduler on a single node cluster. On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would like to release. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Harsh J
Re: [VOTE] Release Apache Hadoop 0.23.7
+1 Downloaded sources, built successfully, stood up a 1-node cluster and ran a Pi MR job. On Wed, Apr 17, 2013 at 2:27 AM, Hitesh Shah hit...@hortonworks.com wrote: +1. Downloaded source, built and ran a couple of sample jobs on a single node cluster. -- Hitesh On Apr 11, 2013, at 12:55 PM, Thomas Graves wrote: I've created a release candidate (RC0) for hadoop-0.23.7 that I would like to release. This release is a sustaining release with several important bug fixes in it. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Tom Graves -- Harsh J
Re: [VOTE] Release Apache Hadoop 2.0.4-alpha
+1 (non-binding). Downloaded from source, tested distributed shell and simple MR jobs on a single-node cluster. -- Hitesh On Apr 12, 2013, at 2:56 PM, Arun C Murthy wrote: Folks, I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would like to release. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: [VOTE] Release Apache Hadoop 0.23.7
+1 (non-binding) 1. Downloaded the source bundle and the binary bundle, and built the source code successfully. 2. Verified the checksum and the signature 3. Setup a single-node cluster, and ran a couple of examples successfully On Tue, Apr 16, 2013 at 8:28 PM, Harsh J ha...@cloudera.com wrote: +1 Downloaded sources, built successfully, stood up a 1-node cluster and ran a Pi MR job. On Wed, Apr 17, 2013 at 2:27 AM, Hitesh Shah hit...@hortonworks.com wrote: +1. Downloaded source, built and ran a couple of sample jobs on a single node cluster. -- Hitesh On Apr 11, 2013, at 12:55 PM, Thomas Graves wrote: I've created a release candidate (RC0) for hadoop-0.23.7 that I would like to release. This release is a sustaining release with several important bug fixes in it. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Tom Graves -- Harsh J -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/
[jira] [Created] (HADOOP-9480) The windows installer should pick the config from src\packages\win\template\conf
Ivan Mitic created HADOOP-9480: -- Summary: The windows installer should pick the config from src\packages\win\template\conf Key: HADOOP-9480 URL: https://issues.apache.org/jira/browse/HADOOP-9480 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic We should pick the config files from the src\packages\win\template\conf location rather than the conf\ location in the Windows installer. Reported by [~mostafae]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9475) Distcp issue
[ https://issues.apache.org/jira/browse/HADOOP-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved HADOOP-9475. Resolution: Invalid Resolving as invalid per Steve's comment. Distcp issue Key: HADOOP-9475 URL: https://issues.apache.org/jira/browse/HADOOP-9475 Project: Hadoop Common Issue Type: Bug Reporter: Sambit Sahoo 2013-04-13 05:11:43,327 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2013-04-13 05:11:43,439 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2013-04-13 05:11:43,750 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 2013-04-13 05:11:43,981 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded initialized native-zlib library 2013-04-13 05:31:08,282 INFO org.apache.hadoop.mapred.Task: Task:attempt_201302011155_224614_m_09_0 is done. And is in the process of commiting 2013-04-13 05:31:09,359 INFO org.apache.hadoop.mapred.Task: Task attempt_201302011155_224614_m_09_0 is allowed to commit now 2013-04-13 05:31:09,937 INFO org.apache.hadoop.mapred.FileOutputCommitter: Saved output of task 'attempt_201302011155_224614_m_09_0' to /tmp/_distcp_logs_spti36 2013-04-13 05:31:09,939 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201302011155_224614_m_09_0' done. 2013-04-13 05:31:09,942 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 I am facing some delay during disctcp from one cluster to another. Here i am copying snappy compressed data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
