Re: [VOTE] Release Apache Hadoop 2.7.1 RC0
+1 (non binding) We have been testing rolling upgrades and downgrades from 2.6 to this release and have had successful runs. -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Jun 29, 2015, at 12:45 PM, Lei Xu l...@cloudera.com wrote: +1 binding Downloaded src and bin distribution, verified md5, sha1 and sha256 checksums of both tar files. Built src using mvn package. Ran a pseudo HDFS cluster Ran dfs -put some files, and checked files on NN's web interface. On Mon, Jun 29, 2015 at 11:54 AM, Wangda Tan wheele...@gmail.com wrote: +1 (non-binding) Compiled and deployed a single node cluster, tried to change node labels and run distributed_shell with node label specified. On Mon, Jun 29, 2015 at 10:30 AM, Ted Yu yuzhih...@gmail.com wrote: +1 (non-binding) Compiled hbase branch-1 with Java 1.8.0_45 Ran unit test suite which passed. On Mon, Jun 29, 2015 at 7:22 AM, Steve Loughran ste...@hortonworks.com wrote: +1 binding from me. Tests: Rebuild slider with Hadoop.version=2.7.1; ran all the tests including against a secure cluster. Repeated for windows running Java 8. All tests passed On 29 Jun 2015, at 09:45, Vinod Kumar Vavilapalli vino...@apache.org wrote: Hi all, I've created a release candidate RC0 for Apache Hadoop 2.7.1. As discussed before, this is the next stable release to follow up 2.6.0, and the first stable one in the 2.7.x line. The RC is available for validation at: *http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/ http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/* The RC tag in git is: release-2.7.1-RC0 The maven artifacts are available via repository.apache.org at * https://repository.apache.org/content/repositories/orgapachehadoop-1019/ https://repository.apache.org/content/repositories/orgapachehadoop-1019/ * Please try the release and vote; the vote will run for the usual 5 days. Thanks, Vinod PS: It took 2 months instead of the planned [1] 2 weeks in getting this release out: post-mortem in a separate thread. [1]: A 2.7.1 release to follow up 2.7.0 http://markmail.org/thread/zwzze6cqqgwq4rmw -- Lei (Eddy) Xu Software Engineer, Cloudera
Re: Updates on migration to git
some of this info is here https://git-wip-us.apache.org/ On Mon, Aug 25, 2014 at 3:45 PM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for bringing these points up, Zhijie. By the way, a revised How-to-commit wiki is at: https://wiki.apache.org/hadoop/HowToCommitWithGit . Please feel free to make changes and improve it. On Mon, Aug 25, 2014 at 11:00 AM, Zhijie Shen zs...@hortonworks.com wrote: Do we have any convention about user.name and user.email? For example, we'd like to use @apache.org for the email. May be, we can ask people to use project-specific configs here and use their real name and @apache.org address. Is there any downside to letting people use their global values for these configs? Moreover, do we want to use --author=Author Name em...@address.com when committing on behalf of a particular contributor? Fetching the email-address is complicated here. Should we use the contributor's email from JIRA? What if that is not their @apache address? On Mon, Aug 25, 2014 at 9:56 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for your input, Steve. Sorry for sending the email out that late, I sent it as soon as I could. On Mon, Aug 25, 2014 at 2:20 AM, Steve Loughran ste...@hortonworks.com wrote: just caught up with this after some offlininess...15:48 PST is too late for me. I'd be -1 to a change to master because of that risk that it does break existing code -especially people that have trunk off the git mirrors and automated builds/merges to go with it. Fair enough. It makes sense to leave it as trunk, unless someone is against it being trunk. master may be viewed as the official git way, but it doesn't have to be. For git-flow workflows (which we use in slider) master/ is for releases, develop/ for dev. On 24 August 2014 02:31, Karthik Kambatla ka...@cloudera.com wrote: Couple of things: 1. Since no one expressed any reservations against doing this on Sunday or renaming trunk to master, I ll go ahead and confirm that. I think that serves us better in the long run. 2. Arpit brought up the precommit builds - we should definitely fix them as soon as we can. I understand Giri maintains those builds, do we have anyone else who has access in case Giri is not reachable? Giri - please shout out if you can help us with this either on Sunday or Monday. Thanks Karthik On Fri, Aug 22, 2014 at 3:50 PM, Karthik Kambatla ka...@cloudera.com wrote: Also, does anyone know what we use for integration between JIRA and svn? I am assuming svn2jira. On Fri, Aug 22, 2014 at 3:48 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, For the SCM migration, feel free to follow https://issues.apache.org/jira/browse/INFRA-8195 Most of this is planned to be handled this Sunday. As a result, the subversion repository would be read-only. If this is a major issue for you, please shout out. Daniel Gruno, the one helping us with the migration, was asking if we are open to renaming trunk to master to better conform to git lingo. I am tempted to say yes, but wanted to check. Would greatly appreciate any help with checking the git repo has everything. Thanks Karthik -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 (non binding) Ran secure and non secure multi node clusters and tested HA and RM recovery tests. -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Oct 7, 2013, at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-5565) job clean up fails on secure cluster as the file system is not created in the context of the ugi running the job
Arpit Gupta created MAPREDUCE-5565:
--
Summary: job clean up fails on secure cluster as the file system
is not created in the context of the ugi running the job
Key: MAPREDUCE-5565
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5565
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.2.1
Reporter: Arpit Gupta
Assignee: Arun C Murthy
Priority: Critical
On secure clusters we see the following exceptions in the jt log
{code}
2013-10-04 04:52:31,753 ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:tt/host@REALM
cause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
{code}
And after the job finishes the staging dir is not cleaned up. While debugging
with [~acmurthy] we determined that file system object needs to be created in
the the context of the user who ran the job.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5488) Job fails after killing all the running containers for the app
Arpit Gupta created MAPREDUCE-5488:
--
Summary: Job fails after killing all the running containers for
the app
Key: MAPREDUCE-5488
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5488
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Jian He
Here is the client stack trace
{code}
RUNNING: /usr/lib/hadoop/bin/hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.1.0.2.0.5.0-66.jar
wordcount -Dmapreduce.reduce.input.limit=-1
/user/user/test_yarn_ha/medium_wordcount_input
/user/hrt_qa/test_yarn_ha/test_mapred_ha_single_job_applicationmaster-1-time
13/08/30 08:45:39 INFO client.RMProxy: Connecting to ResourceManager at
hostname/68.142.247.148:8032
13/08/30 08:45:40 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 19
for user on ha-hdfs:ha-2-secure
13/08/30 08:45:40 INFO security.TokenCache: Got dt for hdfs://ha-2-secure;
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ha-2-secure, Ident:
(HDFS_DELEGATION_TOKEN token 19 for user)
13/08/30 08:45:40 INFO input.FileInputFormat: Total input paths to process : 20
13/08/30 08:45:40 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/08/30 08:45:40 INFO lzo.LzoCodec: Successfully loaded initialized
native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
13/08/30 08:45:40 INFO mapreduce.JobSubmitter: number of splits:180
13/08/30 08:45:40 WARN conf.Configuration: user.name is deprecated. Instead,
use mapreduce.job.user.name
13/08/30 08:45:40 WARN conf.Configuration: mapred.jar is deprecated. Instead,
use mapreduce.job.jar
13/08/30 08:45:40 WARN conf.Configuration: mapred.output.value.class is
deprecated. Instead, use mapreduce.job.output.value.class
13/08/30 08:45:40 WARN conf.Configuration: mapreduce.combine.class is
deprecated. Instead, use mapreduce.job.combine.class
13/08/30 08:45:40 WARN conf.Configuration: mapreduce.map.class is deprecated.
Instead, use mapreduce.job.map.class
13/08/30 08:45:40 WARN conf.Configuration: mapred.job.name is deprecated.
Instead, use mapreduce.job.name
13/08/30 08:45:40 WARN conf.Configuration: mapreduce.reduce.class is
deprecated. Instead, use mapreduce.job.reduce.class
13/08/30 08:45:40 WARN conf.Configuration: mapred.input.dir is deprecated.
Instead, use mapreduce.input.fileinputformat.inputdir
13/08/30 08:45:40 WARN conf.Configuration: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
13/08/30 08:45:40 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
13/08/30 08:45:40 WARN conf.Configuration: mapred.output.key.class is
deprecated. Instead, use mapreduce.job.output.key.class
13/08/30 08:45:40 WARN conf.Configuration: mapred.working.dir is deprecated.
Instead, use mapreduce.job.working.dir
13/08/30 08:45:41 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1377851032086_0003
13/08/30 08:45:41 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN,
Service: ha-hdfs:ha-2-secure, Ident: (HDFS_DELEGATION_TOKEN token 19 for user)
13/08/30 08:45:42 INFO impl.YarnClientImpl: Submitted application
application_1377851032086_0003 to ResourceManager at
hostname/68.142.247.148:8032
13/08/30 08:45:42 INFO mapreduce.Job: The url to track the job:
http://hostname:8088/proxy/application_1377851032086_0003/
13/08/30 08:45:42 INFO mapreduce.Job: Running job: job_1377851032086_0003
13/08/30 08:45:48 INFO mapreduce.Job: Job job_1377851032086_0003 running in
uber mode : false
13/08/30 08:45:48 INFO mapreduce.Job: map 0% reduce 0%
stop applicationmaster
beaver.component.hadoop|INFO|Kill container
container_1377851032086_0003_01_01 on host hostname
RUNNING: ssh -o StrictHostKeyChecking=no hostname sudo su - -c \ps aux | grep
container_1377851032086_0003_01_01 | awk '{print \\\$2}' | xargs kill -9\
root
Warning: Permanently added 'hostname,68.142.247.155' (RSA) to the list of known
hosts.
kill 8978: No such process
waiting for down time 10 seconds for service applicationmaster
13/08/30 08:45:55 INFO ipc.Client: Retrying connect to server:
hostname/68.142.247.155:52713. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1 SECONDS)
13/08/30 08:45:56 INFO ipc.Client: Retrying connect to server:
hostname/68.142.247.155:52713. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1 SECONDS)
13/08/30 08:45:56 ERROR security.UserGroupInformation:
PriviledgedActionException as:user@REALM (auth:KERBEROS)
cause:java.io.IOException: java.net.ConnectException: Call From
hostname.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
java.io.IOException: java.net.ConnectException: Call From
hostname.ConnectException: Connection refused
[jira] [Created] (MAPREDUCE-5474) Add support for mapreduce.reduce.input.limit in MR2
Arpit Gupta created MAPREDUCE-5474: -- Summary: Add support for mapreduce.reduce.input.limit in MR2 Key: MAPREDUCE-5474 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5474 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Arpit Gupta In MR1 we could set mapreduce.reduce.input.limit to control the reduce input limit and it defaults to -1 meaning no limit. There is no such property in MR2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem
[
https://issues.apache.org/jira/browse/MAPREDUCE-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Gupta reopened MAPREDUCE-5351:
Reopening as with this fix we are seeing jobs fail with the following exception
{code}
13/07/02 16:06:57 DEBUG mapred.JobClient: Printing tokens for job:
job_201307020820_0012
13/07/02 16:06:57 DEBUG ipc.Client: IPC Client (47) connection to host/ip:50300
from hortonar sending #32
13/07/02 16:06:57 DEBUG ipc.Client: IPC Client (47) connection to host/ip:50300
from hortonar got value #32
13/07/02 16:06:57 DEBUG retry.RetryUtils: RETRY 0) policy=TryOnceThenFail,
exception=org.apache.hadoop.ipc.RemoteException: java.io.IOException:
Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:383)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1633)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1166)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:350)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3599)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438)
13/07/02 16:06:57 INFO mapred.JobClient: Cleaning up the staging area
hdfs://host:8020/user/hortonar/.staging/job_201307020820_0012
13/07/02 16:06:57 ERROR security.UserGroupInformation:
PriviledgedActionException as:hortonar
cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException: Filesystem
closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:383)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1633)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1166)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:350)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3599)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438)
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:383)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1633)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1166)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:350)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3599)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440
[jira] [Created] (MAPREDUCE-5354) some job submission fail when one of the configured local disks on jt is read only
Arpit Gupta created MAPREDUCE-5354:
--
Summary: some job submission fail when one of the configured local
disks on jt is read only
Key: MAPREDUCE-5354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arpit Gupta
Following exception is seen in the jt
{code}
2013-06-25 06:58:26,988 INFO org.apache.hadoop.mapred.JobTracker: jobToken
generated and stored with users keys in
/mapred/system/job_201306250645_0002/jobToken
2013-06-25 06:58:26,997 ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:user cause:java.io.IOException:
java.io.FileNotFoundException:
/grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file
system)
2013-06-25 06:58:26,997 INFO org.apache.hadoop.ipc.Server: IPC Server handler
33 on 50300, call submitJob(job_201306250645_0002,
hdfs://hostname:8020/user/hrt_qa/.staging/job_201306250645_0002,
org.apache.hadoop.security.Credentials@34e77781) from 68.142.244.30:56631:
error: java.io.IOException: java.io.FileNotFoundException:
/grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file
system)
java.io.IOException: java.io.FileNotFoundException:
/grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file
system)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438)
Caused by: java.io.FileNotFoundException:
/grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file
system)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:194)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.init(RawLocalFileSystem.java:193)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.init(RawLocalFileSystem.java:189)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:260)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.init(ChecksumFileSystem.java:335)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:229)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1248)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1229)
at org.apache.hadoop.mapred.JobInProgress.init(JobInProgress.java:411)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613)
... 12 more
2013-06-25 06:58:27,079 INFO org.apache.hadoop.mapred.JobInitializationPoller:
Passing to Initializer Job Id :job_2013062506
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Plan to create release candidate for 0.23.8
+1 -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On May 17, 2013, at 2:10 PM, Thomas Graves tgra...@yahoo-inc.com wrote: Hello all, We've had a few critical issues come up in 0.23.7 that I think warrants a 0.23.8 release. The main one is MAPREDUCE-5211. There are a couple of other issues that I want finished up and get in before we spin it. Those include HDFS-3875, HDFS-4805, and HDFS-4835. I think those are on track to finish up early next week. So I hope to spin 0.23.8 soon after this vote completes. Please vote '+1' to approve this plan. Voting will close on Friday May 24th at 2:00pm PDT. Thanks, Tom Graves
[jira] [Created] (MAPREDUCE-5198) Race condition in cleanup during task tracker renint with LinuxTaskController
Arpit Gupta created MAPREDUCE-5198:
--
Summary: Race condition in cleanup during task tracker renint with
LinuxTaskController
Key: MAPREDUCE-5198
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5198
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 1.2.0
Reporter: Arpit Gupta
This was noticed when job tracker would be restarted while jobs were running
and would ask the task tracker to reinitialize.
Tasktracker would fail with an error like
{code}
013-04-27 20:19:09,627 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred
local directories are:
/grid/0/hdp/mapred/local,/grid/1/hdp/mapred/local,/grid/2/hdp/mapred/local,/grid/3/hdp/mapred/local,/grid/4/hdp/mapred/local,/grid/5/hdp/mapred/local
2013-04-27 20:19:09,628 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 42075 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:1717)
at org.apache.hadoop.ipc.Server.access$2000(Server.java:98)
at
org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:744)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:808)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)
2013-04-27 20:19:09,628 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 42075: exiting
2013-04-27 20:19:10,414 ERROR org.apache.hadoop.mapred.TaskTracker: Got fatal
exception while reinitializing TaskTracker:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at
org.apache.hadoop.mapred.LinuxTaskController.deleteAsUser(LinuxTaskController.java:281)
at
org.apache.hadoop.mapred.TaskTracker.deleteUserDirectories(TaskTracker.java:779)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:816)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2704)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3934)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5169) Job recovery fails if job tracker is restarted after the job is submitted but before its initialized
Arpit Gupta created MAPREDUCE-5169: -- Summary: Job recovery fails if job tracker is restarted after the job is submitted but before its initialized Key: MAPREDUCE-5169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5169 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.0 Reporter: Arpit Gupta This was noticed when within 5 seconds of submitting a word count job, the job tracker was restarted. Upon restart the job failed to recover -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3826) RM UI when loaded throws a message stating Data Tables warning and then the column sorting stops working
RM UI when loaded throws a message stating Data Tables warning and then the column sorting stops working Key: MAPREDUCE-3826 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3826 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3097) archive does not archive if the content specified is a file
archive does not archive if the content specified is a file --- Key: MAPREDUCE-3097 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3097 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.203.0, 0.20.205.0 Reporter: Arpit Gupta Priority: Minor archive command only archives directories. when the content specified is a file it proceeds with the archive job but does not archive the content this can be misleading as the user might think that archive was successful. We should change it to either throw an error or make it archive files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3100) distcp with an hftp destination url fails if the destination directory does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta resolved MAPREDUCE-3100. Resolution: Invalid hftp is read only distcp with an hftp destination url fails if the destination directory does not exist - Key: MAPREDUCE-3100 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3100 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.20.205.0 Reporter: Arpit Gupta bash-3.2$ bin/hadoop --config HADOOP_CONF_DIR distcp -i hftp://NN_HOST:50070/input_path hftp://NN_HOST:50070/output_path 11/09/27 04:36:01 INFO tools.DistCp: srcPaths=[hftp://hftp://NN_HOST:50070/input_path] 11/09/27 04:36:01 INFO tools.DistCp: destPath=hftp://hftp://NN_HOST:50070/output_path 11/09/27 04:36:02 INFO tools.DistCp: hftp://hftp://NN_HOST:50070/output_path does not exist. With failures, global counters are inaccurate; consider running with -i Copy failed: java.io.IOException: Not supported at org.apache.hadoop.hdfs.HftpFileSystem.mkdirs(HftpFileSystem.java:558) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1066) at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1170) at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666) at org.apache.hadoop.tools.DistCp.run(DistCp.java:881) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.tools.DistCp.main(DistCp.java:908) The same command works if instead of hftp default filesystem (hdfs) is used. It creates the dir if it does not exist. We should do the same for hftp. I also suspect that we have this issue with webhdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
