[jira] [Reopened] (MAPREDUCE-6987) JHS Log Scanner and Cleaner blocked
[
https://issues.apache.org/jira/browse/MAPREDUCE-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened MAPREDUCE-6987:
> JHS Log Scanner and Cleaner blocked
> ---
>
> Key: MAPREDUCE-6987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6987
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
>
> {code}
> "Log Scanner/Cleaner #1" #81 prio=5 os_prio=0 tid=0x7fd6c010f000
> nid=0x11db waiting on condition [0x7fd6aa859000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd6c88a80> (a
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
> at
> org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "Log Scanner/Cleaner #0" #80 prio=5 os_prio=0 tid=0x7fd6c010c800
> nid=0x11da waiting on condition [0x7fd6aa95a000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd6c8> (a
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
> at
> org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Both threads waiting on {{FutureTask.get()}} for infinite time after first
> execution
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6987) JHS Log Scanner and Cleaner blocked
[
https://issues.apache.org/jira/browse/MAPREDUCE-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved MAPREDUCE-6987.
Resolution: Duplicate
Fix Version/s: (was: 3.1.0)
(was: 3.0.0)
(was: 2.9.0)
> JHS Log Scanner and Cleaner blocked
> ---
>
> Key: MAPREDUCE-6987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6987
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
>
> {code}
> "Log Scanner/Cleaner #1" #81 prio=5 os_prio=0 tid=0x7fd6c010f000
> nid=0x11db waiting on condition [0x7fd6aa859000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd6c88a80> (a
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
> at
> org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "Log Scanner/Cleaner #0" #80 prio=5 os_prio=0 tid=0x7fd6c010c800
> nid=0x11da waiting on condition [0x7fd6aa95a000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd6c8> (a
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
> at
> org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Both threads waiting on {{FutureTask.get()}} for infinite time after first
> execution
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6941) The default setting doesn't work for MapReduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved MAPREDUCE-6941.
Resolution: Not A Problem
I'm going to close this based on Ray's analysis. Junping, if you disagree,
please re-open the JIRA.
> The default setting doesn't work for MapReduce job
> --
>
> Key: MAPREDUCE-6941
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6941
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Junping Du
>Priority: Blocker
>
> On the deployment of hadoop 3 cluster (based on current trunk branch) with
> default settings, the MR job will get failed as following exceptions:
> {noformat}
> 2017-08-16 13:00:03,846 INFO mapreduce.Job: Job job_1502913552390_0001
> running in uber mode : false
> 2017-08-16 13:00:03,847 INFO mapreduce.Job: map 0% reduce 0%
> 2017-08-16 13:00:03,864 INFO mapreduce.Job: Job job_1502913552390_0001 failed
> with state FAILED due to: Application application_1502913552390_0001 failed 2
> times due to AM Container for appattempt_1502913552390_0001_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: [2017-08-16 13:00:02.963]Exception from
> container-launch.
> Container id: container_1502913552390_0001_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
> at org.apache.hadoop.util.Shell.run(Shell.java:887)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:295)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:455)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:275)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:90)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This is because mapreduce related jar are not added into yarn setup by
> default. To make MR job run successful, we need to add following
> configurations to yarn-site.xml now:
> {noformat}
>
> yarn.application.classpath
>
> ...
> /share/hadoop/mapreduce/*,
> /share/hadoop/mapreduce/lib/*
> ...
>
> {noformat}
> But this config is not necessary for previous version of Hadoop. We should
> fix this issue before beta release otherwise it will be a regression for
> configuration changes.
> This could be more like a YARN issue (if so, we should move), depends on how
> we fix it finally.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6924) Revert MAPREDUCE-6199 MAPREDUCE-6286 and MAPREDUCE-5875
Andrew Wang created MAPREDUCE-6924: -- Summary: Revert MAPREDUCE-6199 MAPREDUCE-6286 and MAPREDUCE-5875 Key: MAPREDUCE-6924 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6924 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Assignee: Junping Du Filing this JIRA so the reverts show up in the changelog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6924) Revert MAPREDUCE-6199 MAPREDUCE-6286 and MAPREDUCE-5875
[ https://issues.apache.org/jira/browse/MAPREDUCE-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-6924. Resolution: Fixed Fix Version/s: 3.0.0-beta1 Resolving this changelog tracking JIRA. Thanks to [~djp] for doing the reverts! > Revert MAPREDUCE-6199 MAPREDUCE-6286 and MAPREDUCE-5875 > --- > > Key: MAPREDUCE-6924 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6924 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Junping Du > Fix For: 3.0.0-beta1 > > > Filing this JIRA so the reverts show up in the changelog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-5506) Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-5506. Resolution: Won't Fix Resolving as WONTFIX since mr1 has been removed. > Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner > -- > > Key: MAPREDUCE-5506 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5506 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.1 > Environment: RHEL 6.3 x86_64 >Reporter: sam liu >Priority: Blocker > > After I set: > - 'jobConf.setMapRunnerClass(MultithreadedMapRunner.class);' in MR app > - 'mapred.map.multithreadedrunner.threads = 2' in mapred-site.xml > A simple MR app failed as its Map task encountered > ArrayIndexOutOfBoundsException as below(please ignore the line numbers in the > exception as I added some log print codes): > java.lang.ArrayIndexOutOfBoundsException > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1331) > at java.io.DataOutputStream.write(DataOutputStream.java:101) > at org.apache.hadoop.io.Text.write(Text.java:282) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1060) > at > org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:591) > at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:41) > at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:1) > at > org.apache.hadoop.mapred.lib.MultithreadedMapRunner$MapperInvokeRunable.run(MultithreadedMapRunner.java:231) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) > at java.lang.Thread.run(Thread.java:738) > And the exception happens on line 'System.arraycopy(b, off, kvbuffer, > bufindex, len)' in MapTask.java#MapOutputBuffer#Buffer#write(). When the > exception occurs, 'b.length=4' but 'len=9'. > Btw, if I set 'mapred.map.multithreadedrunner.threads = 1', no exception > happened. So it should be an issue caused by multiple threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Reopened] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened MAPREDUCE-6462: > JobHistoryServer to support JvmPauseMonitor as a service > > > Key: MAPREDUCE-6462 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: 0001-MAPREDUCE-6462.patch, 0002-MAPREDUCE-6462.patch, > HADOOP-12321-003.patch, HADOOP-12321-005-aggregated.patch > > > As JvmPauseMonitor is made as an AbstractService, subsequent method changes > are needed in all places which uses the monitor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-6462. Resolution: Duplicate Fix Version/s: (was: 2.9.0) > JobHistoryServer to support JvmPauseMonitor as a service > > > Key: MAPREDUCE-6462 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: 0001-MAPREDUCE-6462.patch, 0002-MAPREDUCE-6462.patch, > HADOOP-12321-003.patch, HADOOP-12321-005-aggregated.patch > > > As JvmPauseMonitor is made as an AbstractService, subsequent method changes > are needed in all places which uses the monitor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6171) The visibilities of the distributed cache files and archives should be determined by both their permissions and if they are located in HDFS encryption zone
[
https://issues.apache.org/jira/browse/MAPREDUCE-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved MAPREDUCE-6171.
Resolution: Duplicate
Fix Version/s: 2.7.0
Duping this to HADOOP-11341 since Dian reports that it fixes this issue. Thanks
again Dian/Arun for finding and working on this.
> The visibilities of the distributed cache files and archives should be
> determined by both their permissions and if they are located in HDFS
> encryption zone
> ---
>
> Key: MAPREDUCE-6171
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6171
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
>Reporter: Dian Fu
> Fix For: 2.7.0
>
>
> The visibilities of the distributed cache files and archives are currently
> determined by the permission of these files or archives.
> The following is the logic of method isPublic() in class
> ClientDistributedCacheManager:
> {code}
> static boolean isPublic(Configuration conf, URI uri,
> Map statCache) throws IOException {
> FileSystem fs = FileSystem.get(uri, conf);
> Path current = new Path(uri.getPath());
> //the leaf level file should be readable by others
> if (!checkPermissionOfOther(fs, current, FsAction.READ, statCache)) {
> return false;
> }
> return ancestorsHaveExecutePermissions(fs, current.getParent(),
> statCache);
> }
> {code}
> At NodeManager side, it will use "yarn" user to download public files and use
> the user who submits the job to download private files. In normal cases,
> there is no problem with this. However, if the files are located in an
> encryption zone(HDFS-6134) and yarn user are configured to be disallowed to
> fetch the DataEncryptionKey(DEK) of this encryption zone by KMS, the download
> process of this file will fail.
> You can reproduce this issue with the following steps (assume you submit job
> with user "testUser"):
> # create a clean cluster which has HDFS cryptographic FileSystem feature
> # create directory "/data/" in HDFS and make it as an encryption zone with
> keyName "testKey"
> # configure KMS to only allow user "testUser" can decrypt DEK of key
> "testKey" in KMS
> {code}
>
> key.acl.testKey.DECRYPT_EEK
> testUser
>
> {code}
> # execute job "teragen" with user "testUser":
> {code}
> su -s /bin/bash testUser -c "hadoop jar hadoop-mapreduce-examples*.jar
> teragen 1 /data/terasort-input"
> {code}
> # execute job "terasort" with user "testUser":
> {code}
> su -s /bin/bash testUser -c "hadoop jar hadoop-mapreduce-examples*.jar
> terasort /data/terasort-input /data/terasort-output"
> {code}
> You will see logs like this at the job submitter's console:
> {code}
> INFO mapreduce.Job: Job job_1416860917658_0002 failed with state FAILED due
> to: Application application_1416860917658_0002 failed 2 times due to AM
> Container for appattempt_1416860917658_0002_02 exited with exitCode:
> -1000 due to: org.apache.hadoop.security.authorize.AuthorizationException:
> User [yarn] is not authorized to perform [DECRYPT_EEK] on key with ACL name
> [testKey]!!
> {code}
> The initial idea to solve this issue is to modify the logic in
> ClientDistributedCacheManager.isPublic to consider also whether this file is
> in an encryption zone. If it is in an encryption zone, this file should be
> considered as private. Then at NodeManager side, it will use user who submits
> the job to fetch the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6040) Automatically use /.reserved/raw when run by the superuser
Andrew Wang created MAPREDUCE-6040: -- Summary: Automatically use /.reserved/raw when run by the superuser Key: MAPREDUCE-6040 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6040 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: fs-encryption Reporter: Andrew Wang Assignee: Charles Lamb On HDFS-6134, [~sanjay.radia] asked for distcp to automatically prepend /.reserved/raw if the distcp is being performed by the superuser and /.reserved/raw is supported by both the source and destination filesystems. Naturally, we'd also want a flag to disable this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-6008) Update distcp docs to include new option that suppresses preservation of RAW.* namespace extended attributes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-6008. Resolution: Not a Problem We took care of the docs in parent JIRA, no need for this one. > Update distcp docs to include new option that suppresses preservation of > RAW.* namespace extended attributes > > > Key: MAPREDUCE-6008 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6008 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: distcp >Affects Versions: fs-encryption >Reporter: Charles Lamb >Assignee: Charles Lamb > > Update the docs to include this new option. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5790) Default map hprof profile options do not work
Andrew Wang created MAPREDUCE-5790:
--
Summary: Default map hprof profile options do not work
Key: MAPREDUCE-5790
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5790
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.3.0
Environment: java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
Reporter: Andrew Wang
I have an MR job doing the following:
{code}
Job job = Job.getInstance(conf);
// Enable profiling
job.setProfileEnabled(true);
job.setProfileTaskRange(true, "0");
job.setProfileTaskRange(false, "0");
{code}
When I run this job, some of my map tasks fail with this error message:
{noformat}
org.apache.hadoop.util.Shell$ExitCodeException:
/data/5/yarn/nm/usercache/hdfs/appcache/application_1394482121761_0012/container_1394482121761_0012_01_41/launch_container.sh:
line 32: $JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -Xmx825955249 -Djava.io.tmpdir=$PWD/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1394482121761_0012/container_1394482121761_0012_01_41
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
${mapreduce.task.profile.params} org.apache.hadoop.mapred.YarnChild
10.20.212.12 43135 attempt_1394482121761_0012_r_00_0 41
1>/var/log/hadoop-yarn/container/application_1394482121761_0012/container_1394482121761_0012_01_41/stdout
2>/var/log/hadoop-yarn/container/application_1394482121761_0012/container_1394482121761_0012_01_41/stderr
: bad substitution
{noformat}
It looks like ${mapreduce.task.profile.params} is not getting subbed in
correctly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5620) distcp1 -delete fails when target directory contains files with percent signs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-5620. Resolution: Invalid Turns out this was due to running distcp1 with hadoop 2's FsShell. I couldn't repro this on a pure branch-1 setup, so resolving as invalid. > distcp1 -delete fails when target directory contains files with percent signs > - > > Key: MAPREDUCE-5620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5620 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > > Debugging a distcp1 issue, it fails to delete extra files in the target > directory when there is a percent sign in the filename. I'm pretty sure this > is an issue with how percent encoding is handled in FsShell (reproduced with > just "hadoop fs -rmr"), but we can also fix this in distcp1 by using > FileSystem instead of FsShell. This is what distcp2 does. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5620) distcp1 -delete fails when target directory contains files with percent signs
Andrew Wang created MAPREDUCE-5620: -- Summary: distcp1 -delete fails when target directory contains files with percent signs Key: MAPREDUCE-5620 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5620 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.1 Reporter: Andrew Wang Assignee: Andrew Wang Debugging a distcp1 issue, it fails to delete extra files in the target directory when there is a percent sign in the filename. I'm pretty sure this is an issue with how percent encoding is handled in FsShell (reproduced with just "hadoop fs -rmr"), but we can also fix this in distcp1 by using FileSystem instead of FsShell. This is what distcp2 does. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5033) mapred shell script should respect usage flags (--help -help -h)
Andrew Wang created MAPREDUCE-5033: -- Summary: mapred shell script should respect usage flags (--help -help -h) Key: MAPREDUCE-5033 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5033 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor Like in HADOOP-9267, the mapred shell script should respect the normal Unix-y help flags. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
