The work-process file creation and block generation and InputSplit generation
Hello. I want to see every detail of 1) Block generation on hdfs when I put the file which has size over at lease 64MB. 2) Generating inputsplits given several blocks in hdfs. So, at first I tried to find related unit test source codes and I found TestFileCreationClient.java. However, there is any specific reference such as Block, BlockMetaInfo and I just found a FSDataOutputStream reference. In addition, In the TestFileCreation.java in mapred test package, it is so complicated and really hard to figure how this all processes were occurred. Please give me some tips and advices or relative documents for these processes. Thanks!
Hadoop-Mapreduce-trunk - Build # 1572 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1572/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 33100 lines...] at hudson.model.Run.execute(Run.java:1603) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:246) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:516) at hudson.remoting.Request.call(Request.java:129) at hudson.remoting.Channel.call(Channel.java:714) at hudson.FilePath.act(FilePath.java:898) ... 13 more Caused by: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Channel$CloseCommand.execute(Channel.java:919) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: Command close created at at hudson.remoting.Command.init(Command.java:56) at hudson.remoting.Channel$CloseCommand.init(Channel.java:913) at hudson.remoting.Channel$CloseCommand.init(Channel.java:911) at hudson.remoting.Channel.close(Channel.java:994) at hudson.remoting.Channel.close(Channel.java:977) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:918) ... 2 more FATAL: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) at hudson.remoting.Request.call(Request.java:174) at hudson.remoting.Channel.call(Channel.java:714) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:167) at com.sun.proxy.$Proxy42.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:925) at hudson.Launcher$ProcStarter.join(Launcher.java:360) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:567) at hudson.model.Run.execute(Run.java:1603) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:246) Caused by: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:774) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:919) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: hudson.remoting.Channel$OrderlyShutdown ... 3 more Caused by: Command close created at at hudson.remoting.Command.init(Command.java:56) at hudson.remoting.Channel$CloseCommand.init(Channel.java:913) at hudson.remoting.Channel$CloseCommand.init(Channel.java:911) at hudson.remoting.Channel.close(Channel.java:994) at hudson.remoting.Channel.close(Channel.java:977) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:918) ... 2 more ### ## FAILED TESTS (if any) ## No tests ran.
Re: [VOTE] Release Apache Hadoop 2.2.0
+1. Downloaded, verified signature/md5, CHANGES.txt, NOTICE, LICENSE, README, release notes, built the source tar ball, and ran a few small jobs on a pseudo-distributed cluster. Tom On 10/7/13 2:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-5574) History server returns 500 error on job conf page if user lacks permissions
Jason Lowe created MAPREDUCE-5574: - Summary: History server returns 500 error on job conf page if user lacks permissions Key: MAPREDUCE-5574 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5574 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.1.1-beta, 0.23.9 Reporter: Jason Lowe If a user does not have view ACL permissions for a job and tries to view the job configuration URL (i.e.: .../jobhistory/conf/jobid) then the history server returns a 500 error rather than a descriptive error message informing the user that they lack permissions. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 (non-binding) Built from source and ran a few jobs on a pseudo-distributed cluster with the Fair Scheduler. On Tue, Oct 8, 2013 at 6:48 AM, Thomas Graves tgra...@yahoo-inc.com wrote: +1. Downloaded, verified signature/md5, CHANGES.txt, NOTICE, LICENSE, README, release notes, built the source tar ball, and ran a few small jobs on a pseudo-distributed cluster. Tom On 10/7/13 2:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
