date:20141101


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193003#comment-14193003
 ] 

Zhijie Shen commented on MAPREDUCE-6052:


bq. Let's keep it here. We do can have flexibility in future for switching job 
working directory or changing replication number. Also, combining two methods 
don't have significant benefits, let's along existing copyAndConfigureFiles() 
except we have strong justification for change.
bq. It is safety to keep checking directory existing before using it given the 
possible refactoring or other operations in parallel.

There's unnecessary code duplication though executing it multiple times doesn't 
 result in bug, which is better to be cleanup. In fact, the replication number 
and the working dir is read once, and used for all the following files/jars in 
distributed caches. It's should have no problem to reuse it for log4j. And 
during copying files/jars, this configured replication number and working dir 
shouldn't be changed, and don't make sense to be concurrently modified by 
others.

After the redundant code is removed, one method contains so few statements that 
can be put directly into the other one, but I'm okay if you want to keep 
separate them.

bq. two files with the same name on classpath (one is loading in conf directory 
by default, the other is ), which one would take effective? 

If so, probably we are going to still use the default 
{{container-log4j.properties}} because it comes with 
hadoop-yarn-server-nodemanager-2.x.y.jar. It will be good if we can document 
this issue somewhere, in case we run into this issue later.


 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193016#comment-14193016
 ] 

Zhijie Shen commented on MAPREDUCE-6052:


Have a quick offline chat shortly with Junping. As there already exist code 
duplication between copying regular files and tar ball files, let's just leave 
the patch as what it is not to move fast for 2.6. Will file a separate 
code-refactoring Jira to consolidate copying of regular files(jars)/log4j 
file/tar balls.

Will go ahead to commit the current patch.

 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193019#comment-14193019
 ] 

Hudson commented on MAPREDUCE-6052:
---

FAILURE: Integrated in Hadoop-trunk-Commit #6419 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6419/])
MAPREDUCE-6052. Supported overriding the default container-log4j.properties 
file per job. Contributed by Junping Du. (zjshen: rev 
ed63b116465290fdb0acdf89170025f47b307599)
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java
* hadoop-mapreduce-project/CHANGES.txt
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java


 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6052) Support overriding log4j.properties per job


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated MAPREDUCE-6052:
---
   Resolution: Fixed
Fix Version/s: 2.6.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to trunk, branch-2 and branch-2.6. Thanks, Junping! Also thanks 
contributing ur ideas, Xuan and Vinod!

 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6148) Consolidate copying regular files/jars, log4j file and tar ball files

Zhijie Shen created MAPREDUCE-6148:
--

 Summary: Consolidate copying regular files/jars, log4j file and 
tar ball files
 Key: MAPREDUCE-6148
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6148
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Reporter: Zhijie Shen
Assignee: Junping Du


There're code duplication among copying regular files/jars, log4j file and tar 
ball files. We need to consolidate them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193114#comment-14193114
 ] 

Hudson commented on MAPREDUCE-6052:
---

SUCCESS: Integrated in Hadoop-Yarn-trunk #730 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/730/])
MAPREDUCE-6052. Supported overriding the default container-log4j.properties 
file per job. Contributed by Junping Du. (zjshen: rev 
ed63b116465290fdb0acdf89170025f47b307599)
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* hadoop-mapreduce-project/CHANGES.txt
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java


 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193200#comment-14193200
 ] 

Hudson commented on MAPREDUCE-6052:
---

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1919 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1919/])
MAPREDUCE-6052. Supported overriding the default container-log4j.properties 
file per job. Contributed by Junping Du. (zjshen: rev 
ed63b116465290fdb0acdf89170025f47b307599)
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* hadoop-mapreduce-project/CHANGES.txt
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java


 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193232#comment-14193232
 ] 

Hudson commented on MAPREDUCE-6052:
---

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1944 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1944/])
MAPREDUCE-6052. Supported overriding the default container-log4j.properties 
file per job. Contributed by Junping Du. (zjshen: rev 
ed63b116465290fdb0acdf89170025f47b307599)
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* hadoop-mapreduce-project/CHANGES.txt
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java


 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (MAPREDUCE-6052) Support overriding log4j.properties per job

2014-11-01 Thread Vinod Kumar Vavilapalli (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened MAPREDUCE-6052:


I am reopening this JIRA for few minor issues, please post an addendum patch.

h4. Code
 - Users can use URI fragments in distributed-cache - for e.g. 
hdfs://a/b/c#symlink. Either we should explicitly reject fragments or support 
them. See MRApps.parseDistributedCacheArtifacts() for similar changes that we 
do for regular dist-cache files.
 - MRApps.addLog4jSystemProperties(): If the user passes his own log4j file, 
setting YARN_APP_CONTAINER_LOG_SIZE, YARN_APP_CONTAINER_LOG_BACKUPS and 
logLevel may not matter at all. But let's set them as you do now, just leave a 
code comment that setting them may or may not really take effect.
 - job.setWorkingDirectory() getting called repeatedly?
 - For the null-scheme input, File  + file +  does not exist. - File  + 
file +  does not exist on the local file-system.
 - Delete this log statement that you might have added for your testing: 
LOG.debug(default FileSystem:  + jtFs.getUri());
 - What happens if the user passes a relative path for 
MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE? I think it works. If so, let's put this in 
the documentation too.

h4. Documentation
 - Document the new config in mapred-default.xml
 - Mention in that documentation that if no-scheme is given in the path, it 
defaults to a log4j file on the local FS.
 - Modify the documentation of log-level configs to say that if you override to 
have your own log4j.properties file, the log-level configs may not work

 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MAPREDUCE-6052) Support overriding log4j.properties per job

2014-11-01 Thread Vinod Kumar Vavilapalli (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193562#comment-14193562
]

Vinod Kumar Vavilapalli edited comment on MAPREDUCE-6052 at 11/1/14 11:20 PM:
--

I am reopening this JIRA for few issues, please post an addendum patch.

h4. Code
- Users can use URI fragments in distributed-cache - for e.g.
hdfs://a/b/c#symlink. Either we should explicitly reject fragments or support
them. See MRApps.parseDistributedCacheArtifacts() for similar changes that we
do for regular dist-cache files.
- MRApps.addLog4jSystemProperties(): If the user passes his own log4j file,
setting YARN_APP_CONTAINER_LOG_SIZE, YARN_APP_CONTAINER_LOG_BACKUPS and
logLevel may not matter at all. But let's set them as you do now, just leave a
code comment that setting them may or may not really take effect.
- job.setWorkingDirectory() getting called repeatedly?
- For the null-scheme input, File + file + does not exist. - File +
file + does not exist on the local file-system.
- Delete this log statement that you might have added for your testing:
LOG.debug(default FileSystem: + jtFs.getUri());
- What happens if the user passes a relative path for
MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE? I think it works. If so, let's put this in
the documentation too.

h4. Documentation
- Document the new config in mapred-default.xml
- Mention in that documentation that if no-scheme is given in the path, it
defaults to a log4j file on the local FS.
- Modify the documentation of log-level configs to say that if you override to
have your own log4j.properties file, the log-level configs may not work

was (Author: vinodkv):
I am reopening this JIRA for few minor issues, please post an addendum patch.

Support overriding log4j.properties per job
---

Key: MAPREDUCE-6052
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
Fix For: 2.6.0

Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch,
MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch

For current MR application, the log4j.configuration is hard coded to
container-log4j.properties within each node. We still need flexibility to
override it per job like what we do in MRV1.
{code}
public static void addLog4jSystemProperties(
String logLevel, long logSize, int numBackups, ListString vargs) {
vargs.add(-Dlog4j.configuration=container-log4j.properties);
{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193689#comment-14193689
 ] 

Junping Du commented on MAPREDUCE-6052:
---

Thanks [~vinodkv] for comments!
bq. Users can use URI fragments in distributed-cache - for e.g. 
hdfs://a/b/c#symlink. Either we should explicitly reject fragments or support 
them. See MRApps.parseDistributedCacheArtifacts() for similar changes that we 
do for regular dist-cache files.
IMO, URI fragment doesn't make too much sense here as we are hiding details for 
how log4j.properties upload to hdfs and get download to each node's local cache 
and put it on classpath. MR over distributed cache support fragment as user 
need to explicitly set class path that included in tar ball and symlink for 
directory is easily to reference. In this case, user only need to set one value 
here to point to local filesystem. If you still think fragment is required, 
please file a separated JIRA instead of reopening this.

bq. If the user passes his own log4j file, setting YARN_APP_CONTAINER_LOG_SIZE, 
YARN_APP_CONTAINER_LOG_BACKUPS and logLevel may not matter at all. But let's 
set them as you do now, just leave a code comment that setting them may or may 
not really take effect.
Agree we should put this on documentation.

bq. job.setWorkingDirectory() getting called repeatedly?
As Zhijie's raised above, we decided to address the consolidation of some code 
flow here in MAPREDUCE-6148.

bq. For the null-scheme input, File  + file +  does not exist. - File  + 
file +  does not exist on the local file-system.
Nice catch! Will address this minor issue in refactoring JIRA.

bq. Delete this log statement that you might have added for your testing: 
LOG.debug(default FileSystem:  + jtFs.getUri());
I think it is helpful and prefer to leave it here which doesn't affect system 
runtime performance. Right?

bq. What happens if the user passes a relative path for 
MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE? I think it works. If so, let's put this in 
the documentation too.
Relative path will base on current user's working directory. Let's document it.

Other documentation comments make sense to me. Given most of comments are 
documentation, let's resolve this and file a separated document JIRA to track.

 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6149) Document override log4j.properties in MR job

Junping Du created MAPREDUCE-6149:
-

 Summary: Document override log4j.properties in MR job
 Key: MAPREDUCE-6149
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Reporter: Junping Du
Assignee: Junping Du






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6149) Document override log4j.properties in MR job


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated MAPREDUCE-6149:
--
Description: 
This new feature comes from MAPREDUCE-6052, some documentation requirements 
from Vinod below:
Document the new config in mapred-default.xml
Mention in that documentation that if no-scheme is given in the path, it 
defaults to a log4j file on the local FS.
Modify the documentation of log-level configs to say that if you override 
to have your own log4j.properties file, the log-level configs may not work.


 Document override log4j.properties in MR job
 

 Key: MAPREDUCE-6149
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Reporter: Junping Du
Assignee: Junping Du

 This new feature comes from MAPREDUCE-6052, some documentation requirements 
 from Vinod below:
 Document the new config in mapred-default.xml
 Mention in that documentation that if no-scheme is given in the path, it 
 defaults to a log4j file on the local FS.
 Modify the documentation of log-level configs to say that if you override 
 to have your own log4j.properties file, the log-level configs may not work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-6052) Support overriding log4j.properties per job


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved MAPREDUCE-6052.
---
Resolution: Fixed

 Support overriding log4j.properties per job
 ---

 Key: MAPREDUCE-6052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Junping Du
Assignee: Junping Du
 Fix For: 2.6.0

 Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, 
 MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch


 For current MR application, the log4j.configuration is hard coded to 
 container-log4j.properties within each node. We still need flexibility to 
 override it per job like what we do in MRV1.
 {code}
   public static void addLog4jSystemProperties(
   String logLevel, long logSize, int numBackups, ListString vargs) {
 vargs.add(-Dlog4j.configuration=container-log4j.properties);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6052) Support overriding log4j.properties per job