[jira] [Updated] (HADOOP-8224) Don't hardcode hdfs.audit.logger in the scripts
[ https://issues.apache.org/jira/browse/HADOOP-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomohiko Kinebuchi updated HADOOP-8224: --- Attachment: HADOOP-8224.txt Attached a patch. Could someone give me contributor permission and assign to me? Don't hardcode hdfs.audit.logger in the scripts --- Key: HADOOP-8224 URL: https://issues.apache.org/jira/browse/HADOOP-8224 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 2.0.0 Reporter: Eli Collins Attachments: HADOOP-8224.txt The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the hdfs.audit.logger (is explicitly set via -Dhdfs.audit.logger=INFO,RFAAUDIT) so it's not overridable. Let's allow someone to override it as we do the other parameters by introducing HADOOP_AUDIT_LOGGER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8301) Common (hadoop-tools) side of MAPREDUCE-4172
Harsh J created HADOOP-8301: --- Summary: Common (hadoop-tools) side of MAPREDUCE-4172 Key: HADOOP-8301 URL: https://issues.apache.org/jira/browse/HADOOP-8301 Project: Hadoop Common Issue Type: Task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Patches on MAPREDUCE-4172 (for MR-relevant projects) that requires to run off of Hadoop Common project for Hadoop QA. One sub-task per hadoop-tools submodule will be added here for reviews. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8302) Clean up hadoop-rumen
Harsh J created HADOOP-8302: --- Summary: Clean up hadoop-rumen Key: HADOOP-8302 URL: https://issues.apache.org/jira/browse/HADOOP-8302 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8303) Clean up hadoop-streaming
Harsh J created HADOOP-8303: --- Summary: Clean up hadoop-streaming Key: HADOOP-8303 URL: https://issues.apache.org/jira/browse/HADOOP-8303 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8302) Clean up hadoop-rumen
[ https://issues.apache.org/jira/browse/HADOOP-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8302: Status: Patch Available (was: Open) Clean up hadoop-rumen - Key: HADOOP-8302 URL: https://issues.apache.org/jira/browse/HADOOP-8302 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0007-MapReduce-Rumen-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8303) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8303: Attachment: 0008-MapReduce-Streaming-Cleanup.patch Patch that cleans all warnings of hadoop-streaming sub-module save for usage of MiniMRCluster and DistributedCache and one instance of StreamJob's non usual constructor cause a job reference is still required there. Clean up hadoop-streaming - Key: HADOOP-8303 URL: https://issues.apache.org/jira/browse/HADOOP-8303 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0008-MapReduce-Streaming-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8303) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8303: Status: Patch Available (was: Open) Clean up hadoop-streaming - Key: HADOOP-8303 URL: https://issues.apache.org/jira/browse/HADOOP-8303 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0008-MapReduce-Streaming-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8303) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259198#comment-13259198 ] Hadoop QA commented on HADOOP-8303: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523700/0008-MapReduce-Streaming-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 25 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/876//console This message is automatically generated. Clean up hadoop-streaming - Key: HADOOP-8303 URL: https://issues.apache.org/jira/browse/HADOOP-8303 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0008-MapReduce-Streaming-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8302) Clean up hadoop-rumen
[ https://issues.apache.org/jira/browse/HADOOP-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259199#comment-13259199 ] Hadoop QA commented on HADOOP-8302: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523699/0007-MapReduce-Rumen-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/877//console This message is automatically generated. Clean up hadoop-rumen - Key: HADOOP-8302 URL: https://issues.apache.org/jira/browse/HADOOP-8302 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0007-MapReduce-Rumen-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8303) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8303: Attachment: 0008-MapReduce-Streaming-Cleanup.patch A few things: # Looks like even Hadoop-Common QA does not pick up hadoop-tools changes. # The eclipse's auto unused import fixer removed RecRecord imports which caused TestIO not to compile. I fixed those manually in this new patch. # TestStreaming had a change in original patch that caused it to lose info about the running jobs and such. I tweaked the change to use a non ToolRunner approach, and TestStreamingCounters/Combiner now work nicely again - minus deprecation warnings. Since Hadoop QA bot can't run the tests, here is the test result manually run, with patch applied: {{cd hadoop-tools/hadoop-streaming; mvn clean test}} {code} --- T E S T S --- Running org.apache.hadoop.streaming.TestAutoInputFormat Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.663 sec Running org.apache.hadoop.streaming.TestClassWithNoPackage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.261 sec Running org.apache.hadoop.streaming.TestDumpTypedBytes Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.21 sec Running org.apache.hadoop.streaming.TestFileArgs Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.49 sec Running org.apache.hadoop.streaming.TestGzipInput Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.188 sec Running org.apache.hadoop.streaming.TestLoadTypedBytes Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.486 sec Running org.apache.hadoop.streaming.TestMRFramework Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.265 sec Running org.apache.hadoop.streaming.TestMultipleArchiveFiles Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.727 sec Running org.apache.hadoop.streaming.TestMultipleCachefiles Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.324 sec Running org.apache.hadoop.streaming.TestRawBytesStreaming Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.07 sec Running org.apache.hadoop.streaming.TestStreamAggregate Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.062 sec Running org.apache.hadoop.streaming.TestStreamDataProtocol Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.058 sec Running org.apache.hadoop.streaming.TestStreaming Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.165 sec Running org.apache.hadoop.streaming.TestStreamingBackground Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.09 sec Running org.apache.hadoop.streaming.TestStreamingBadRecords Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.769 sec Running org.apache.hadoop.streaming.TestStreamingCombiner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.21 sec Running org.apache.hadoop.streaming.TestStreamingCounters Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.191 sec Running org.apache.hadoop.streaming.TestStreamingExitStatus Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.031 sec Running org.apache.hadoop.streaming.TestStreamingFailure Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.788 sec Running org.apache.hadoop.streaming.TestStreamingKeyValue Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.324 sec Running org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.626 sec Running org.apache.hadoop.streaming.TestStreamingSeparator Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.068 sec Running org.apache.hadoop.streaming.TestStreamingStderr Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.579 sec Running org.apache.hadoop.streaming.TestStreamingTaskLog Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.659 sec Running org.apache.hadoop.streaming.TestStreamJob Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.761 sec Running org.apache.hadoop.streaming.TestStreamReduceNone Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec Running org.apache.hadoop.streaming.TestStreamXmlMultipleRecords Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.238 sec Running org.apache.hadoop.streaming.TestStreamXmlRecordReader Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.152 sec Running org.apache.hadoop.streaming.TestSymLink Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.418 sec Running org.apache.hadoop.streaming.TestTypedBytesStreaming Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec Running
[jira] [Commented] (HADOOP-8302) Clean up hadoop-rumen
[ https://issues.apache.org/jira/browse/HADOOP-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259253#comment-13259253 ] Harsh J commented on HADOOP-8302: - Since it looks like Hadoop QA is unable to run hadoop-tools patches, I ran the Rumen tests manually with this cleanup patch applied locally: {{cd hadoop-tools/hadoop-rumen; mvn clean test}} {code} --- T E S T S --- Running org.apache.hadoop.tools.rumen.TestHistograms Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.033 sec Running org.apache.hadoop.tools.rumen.TestPiecewiseLinearInterpolation Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.037 sec Running org.apache.hadoop.tools.rumen.TestRandomSeedGenerator Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec Results : Tests run: 3, Failures: 0, Errors: 0, Skipped: 1 {code} Clean up hadoop-rumen - Key: HADOOP-8302 URL: https://issues.apache.org/jira/browse/HADOOP-8302 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0007-MapReduce-Rumen-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8303) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259255#comment-13259255 ] Hadoop QA commented on HADOOP-8303: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523715/0008-MapReduce-Streaming-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 25 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/878//console This message is automatically generated. Clean up hadoop-streaming - Key: HADOOP-8303 URL: https://issues.apache.org/jira/browse/HADOOP-8303 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0008-MapReduce-Streaming-Cleanup.patch, 0008-MapReduce-Streaming-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8224) Don't hardcode hdfs.audit.logger in the scripts
[ https://issues.apache.org/jira/browse/HADOOP-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HADOOP-8224: --- Assignee: Tomohiko Kinebuchi Thanks Tomohiko, I've added you as a contributor and assigned this to you. How did you test this change? Looks like your change should be sufficient, it's not necessary to update hadoop-daemon.sh with an export similar to the other _LOGGER variables. Don't hardcode hdfs.audit.logger in the scripts --- Key: HADOOP-8224 URL: https://issues.apache.org/jira/browse/HADOOP-8224 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Tomohiko Kinebuchi Attachments: HADOOP-8224.txt The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the hdfs.audit.logger (is explicitly set via -Dhdfs.audit.logger=INFO,RFAAUDIT) so it's not overridable. Let's allow someone to override it as we do the other parameters by introducing HADOOP_AUDIT_LOGGER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8224) Don't hardcode hdfs.audit.logger in the scripts
[ https://issues.apache.org/jira/browse/HADOOP-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259312#comment-13259312 ] Eli Collins commented on HADOOP-8224: - Actually, it does look like the export in hadoop-daemon.sh is necessary, this is for the case where people don't have hadoop-env.sh in their conf dir. Don't hardcode hdfs.audit.logger in the scripts --- Key: HADOOP-8224 URL: https://issues.apache.org/jira/browse/HADOOP-8224 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Tomohiko Kinebuchi Attachments: HADOOP-8224.txt The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the hdfs.audit.logger (is explicitly set via -Dhdfs.audit.logger=INFO,RFAAUDIT) so it's not overridable. Let's allow someone to override it as we do the other parameters by introducing HADOOP_AUDIT_LOGGER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8224) Don't hardcode hdfs.audit.logger in the scripts
[ https://issues.apache.org/jira/browse/HADOOP-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomohiko Kinebuchi updated HADOOP-8224: --- Attachment: HADOOP-8224.txt Don't hardcode hdfs.audit.logger in the scripts --- Key: HADOOP-8224 URL: https://issues.apache.org/jira/browse/HADOOP-8224 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Tomohiko Kinebuchi Attachments: HADOOP-8224.txt, HADOOP-8224.txt The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the hdfs.audit.logger (is explicitly set via -Dhdfs.audit.logger=INFO,RFAAUDIT) so it's not overridable. Let's allow someone to override it as we do the other parameters by introducing HADOOP_AUDIT_LOGGER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8224) Don't hardcode hdfs.audit.logger in the scripts
[ https://issues.apache.org/jira/browse/HADOOP-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259381#comment-13259381 ] Tomohiko Kinebuchi commented on HADOOP-8224: Thank you for your comment. I created and attached a new patch including first fix and a fix for hadoop-daemon.sh. I am thinking how do I test these fixes. 1. Kick a Hadoop cluster with this shell, and check existence of log messages in hdfs-audit.log. 2. Kick a Hadoop cluster with this shell, and get configuration values from Configuration class. Do you have some other testing processes? Don't hardcode hdfs.audit.logger in the scripts --- Key: HADOOP-8224 URL: https://issues.apache.org/jira/browse/HADOOP-8224 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Tomohiko Kinebuchi Attachments: HADOOP-8224.txt, HADOOP-8224.txt The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the hdfs.audit.logger (is explicitly set via -Dhdfs.audit.logger=INFO,RFAAUDIT) so it's not overridable. Let's allow someone to override it as we do the other parameters by introducing HADOOP_AUDIT_LOGGER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira