[jira] [Commented] (HADOOP-7470) move up to Jackson 1.8.2
[ https://issues.apache.org/jira/browse/HADOOP-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067472#comment-13067472 ] Amar Kamat commented on HADOOP-7470: Steve, MAPREDUCE-778 depends on Jackson 1.8 jars and I have modified the ivy scripts to pull the latest Jackson jars. We are actively working on MAPREDUCE-778 and will hopefully complete it soon. move up to Jackson 1.8.2 Key: HADOOP-7470 URL: https://issues.apache.org/jira/browse/HADOOP-7470 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 0.20.203.0 Reporter: Steve Loughran Priority: Minor Original Estimate: 0.25h Remaining Estimate: 0.25h I see that hadoop-core still depends on Jackson 1.0.1 -but that project is now up to 1.8.2 in releases. Upgrading will make it easier for other Jackson-using apps that are more up to date to keep their classpath consistent. The patch would be updating the ivy file to pull in the later version; no test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HADOOP-6866) Tool interface should also support getUsage()
Tool interface should also support getUsage()
-
Key: HADOOP-6866
URL: https://issues.apache.org/jira/browse/HADOOP-6866
Project: Hadoop Common
Issue Type: Improvement
Reporter: Amar Kamat
Currently each and every _tool_ implementing the {{Tool}} interface is forced
to manage their usage string. Since its a common piece of code, its better we
factor it out. This can be useful in the following ways
# A proper lib like support for usage strings
# Forcing _tools_ (implementers of {{Tool}}) to expose their usage string
# Test cases can now use these well defined and exposed usage strings to test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6826) Revert FileSystem create method that takes CreateFlags
[ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12881069#action_12881069 ] Amar Kamat commented on HADOOP-6826: Tom/Dhruba, I think this commit should be reverted. FileSystem.create() is a public api and hence cannot be removed like this. Ideally it should be deprecated. Do you agree? Revert FileSystem create method that takes CreateFlags -- Key: HADOOP-6826 URL: https://issues.apache.org/jira/browse/HADOOP-6826 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.21.0 Attachments: HADOOP-6826.patch As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Moved: (HADOOP-6757) NullPointerException for hadoop clients launched from streaming tasks
[ https://issues.apache.org/jira/browse/HADOOP-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat moved MAPREDUCE-1771 to HADOOP-6757: --- Component/s: scripts Project: Hadoop Common (was: Hadoop Map/Reduce) Key: HADOOP-6757 (was: MAPREDUCE-1771) Component/s: (was: client) NullPointerException for hadoop clients launched from streaming tasks - Key: HADOOP-6757 URL: https://issues.apache.org/jira/browse/HADOOP-6757 Project: Hadoop Common Issue Type: Bug Components: scripts Reporter: Amar Kamat Assignee: Amar Kamat TaskRunner sets HADOOP_ROOT_LOGGER to info,TLA while launching the child tasks. TLA implicitly assumes that that task-id information will be made available via the 'hadoop.tasklog.taskid' parameter. 'hadoop.tasklog.taskid' is passed to the child task by the TaskRunner via HADOOP_CLIENT_OPTS. When the streaming task launches a hadoop client (say hadoop job -list), the HADOOP_ROOT_LOGGER of the hadoop client is set to 'info,TLA' but hadoop.tasklog.taskid is not set resulting into NPE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6757) NullPointerException for hadoop clients launched from streaming tasks
[ https://issues.apache.org/jira/browse/HADOOP-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6757: --- Attachment: HADOOP-6757-v1.0.patch Attaching a patch for trunk. This patch adds the HADOOP_CLIENT_OPTS to HADOOP_OPTS for the following commands - bin/mapred job - bin/mapred queue - bin/hadoop jar NullPointerException for hadoop clients launched from streaming tasks - Key: HADOOP-6757 URL: https://issues.apache.org/jira/browse/HADOOP-6757 Project: Hadoop Common Issue Type: Bug Components: scripts Reporter: Amar Kamat Assignee: Amar Kamat Attachments: BZ-3620565-v1.0.patch, HADOOP-6757-v1.0.patch TaskRunner sets HADOOP_ROOT_LOGGER to info,TLA while launching the child tasks. TLA implicitly assumes that that task-id information will be made available via the 'hadoop.tasklog.taskid' parameter. 'hadoop.tasklog.taskid' is passed to the child task by the TaskRunner via HADOOP_CLIENT_OPTS. When the streaming task launches a hadoop client (say hadoop job -list), the HADOOP_ROOT_LOGGER of the hadoop client is set to 'info,TLA' but hadoop.tasklog.taskid is not set resulting into NPE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6757) NullPointerException for hadoop clients launched from streaming tasks
[ https://issues.apache.org/jira/browse/HADOOP-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6757: --- Status: Patch Available (was: Open) Running through Hudson. NullPointerException for hadoop clients launched from streaming tasks - Key: HADOOP-6757 URL: https://issues.apache.org/jira/browse/HADOOP-6757 Project: Hadoop Common Issue Type: Bug Components: scripts Reporter: Amar Kamat Assignee: Amar Kamat Attachments: BZ-3620565-v1.0.patch, HADOOP-6757-v1.0.patch TaskRunner sets HADOOP_ROOT_LOGGER to info,TLA while launching the child tasks. TLA implicitly assumes that that task-id information will be made available via the 'hadoop.tasklog.taskid' parameter. 'hadoop.tasklog.taskid' is passed to the child task by the TaskRunner via HADOOP_CLIENT_OPTS. When the streaming task launches a hadoop client (say hadoop job -list), the HADOOP_ROOT_LOGGER of the hadoop client is set to 'info,TLA' but hadoop.tasklog.taskid is not set resulting into NPE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6711) Configuration should support list of values
Configuration should support list of values
---
Key: HADOOP-6711
URL: https://issues.apache.org/jira/browse/HADOOP-6711
Project: Hadoop Common
Issue Type: New Feature
Components: conf
Reporter: Amar Kamat
Configuration supports 2 operations namely _set()_ and _get()_. It would be
nice to have an inbuild support for lists where there can be multiple values
(i.e list of values) assigned to one key. A workaround could be
{code}
// Assume Key be the parameter key and newValue be the value to be
added/appended
Configuration c = new Configuration();
String value = c.get(Key);
value = value + + newValue
c.set(Key, value);
{code}
One common usecase is that in a production enviroment, some user facing params
(e.g mapred.child.java.opts) are set to default values (say for performance
reasons). Users themselves might want to *add* to this list. Doing a set would
overwrite the previous values. The above mentioned hack is doable via code but
not via command line. Hence we need the framework to support lists.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12837135#action_12837135 ] Amar Kamat commented on HADOOP-6592: Isnt this a duplicate of MAPREDUCE-1227? Scheduler: Pause button desirable - Key: HADOOP-6592 URL: https://issues.apache.org/jira/browse/HADOOP-6592 Project: Hadoop Common Issue Type: Wish Reporter: Adam Kramer Priority: Minor It would be lovely if, from the jobtracker page, I could click a button that's not kill or fail but ...pause. The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the pending stage until the job is un-paused. Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs. This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could catch up, and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2141) speculative execution start up condition based on completion time
[ https://issues.apache.org/jira/browse/HADOOP-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-2141: --- Attachment: hadoop-2141-yahoo-v1.4.8.patch Attaching a patch for Yahoo!'s distribution of hadoop-0.20 not to be committed. speculative execution start up condition based on completion time - Key: HADOOP-2141 URL: https://issues.apache.org/jira/browse/HADOOP-2141 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Koji Noguchi Assignee: Andy Konwinski Fix For: 0.21.0 Attachments: 2141.4.patch, 2141.5.patch, 2141.6.patch, 2141.7.patch, 2141.8.2.patch, 2141.8.3.patch, 2141.patch, HADOOP-2141-v2.patch, HADOOP-2141-v3.patch, HADOOP-2141-v4.patch, HADOOP-2141-v5.patch, HADOOP-2141-v6.patch, hadoop-2141-yahoo-v1.4.1.patch, hadoop-2141-yahoo-v1.4.8.patch, HADOOP-2141.patch, HADOOP-2141.v7.patch, HADOOP-2141.v8.patch We had one job with speculative execution hang. 4 reduce tasks were stuck with 95% completion because of a bad disk. Devaraj pointed out bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here. It would be nice if speculative execution also starts up when tasks stop making progress. Devaraj suggested bq. Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5879) GzipCodec should read compression level etc from configuration
[
https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated HADOOP-5879:
---
Attachment: hadoop-5879-yahoo-0.20-v1.0.patch
Attaching a patch for Yahoo!'s distribution of hadoop 0.20, not to be committed
here.
GzipCodec should read compression level etc from configuration
--
Key: HADOOP-5879
URL: https://issues.apache.org/jira/browse/HADOOP-5879
Project: Hadoop Common
Issue Type: Improvement
Components: io
Reporter: Zheng Shao
Assignee: He Yongqiang
Fix For: 0.21.0
Attachments: H5879-5.patch, hadoop-5879-5-21.patch,
hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch, hadoop-5879-7-18-3.patch,
hadoop-5879-7-26.patch, hadoop-5879-yahoo-0.20-v1.0.patch
GzipCodec currently uses the default compression level. We should allow
overriding the default value from Configuration.
{code}
static final class GzipZlibCompressor extends ZlibCompressor {
public GzipZlibCompressor() {
super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
}
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6550) Document the various compression levels supported for GzipCodec
Document the various compression levels supported for GzipCodec --- Key: HADOOP-6550 URL: https://issues.apache.org/jira/browse/HADOOP-6550 Project: Hadoop Common Issue Type: Improvement Components: io Reporter: Amar Kamat HADOOP-5879 made compression levels configurable for GzipCodec. It would be nice to document them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6161) Add get/setEnum to Configuration
[ https://issues.apache.org/jira/browse/HADOOP-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6161: --- Attachment: hadoop-6161-yahoo-20-v1.patch Attaching an example patch for Yahoo!'s internal 0.20 branch not to be committed. Add get/setEnum to Configuration Key: HADOOP-6161 URL: https://issues.apache.org/jira/browse/HADOOP-6161 Project: Hadoop Common Issue Type: Improvement Components: conf Reporter: Chris Douglas Assignee: Chris Douglas Priority: Minor Fix For: 0.21.0 Attachments: C6161-0.patch, hadoop-6161-yahoo-20-v1.patch It would be useful if Configuration had helper get/set methods for enumerated types. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2141) speculative execution start up condition based on completion time
[ https://issues.apache.org/jira/browse/HADOOP-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-2141: --- Attachment: hadoop-2141-yahoo-v1.4.1.patch Attaching a patch for Yahoo!'s distribution of Hadoop 0.20 not to be committed. test-patch and ant tests passed on my box. speculative execution start up condition based on completion time - Key: HADOOP-2141 URL: https://issues.apache.org/jira/browse/HADOOP-2141 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Koji Noguchi Assignee: Andy Konwinski Fix For: 0.21.0 Attachments: 2141.4.patch, 2141.5.patch, 2141.6.patch, 2141.7.patch, 2141.8.2.patch, 2141.8.3.patch, 2141.patch, HADOOP-2141-v2.patch, HADOOP-2141-v3.patch, HADOOP-2141-v4.patch, HADOOP-2141-v5.patch, HADOOP-2141-v6.patch, hadoop-2141-yahoo-v1.4.1.patch, HADOOP-2141.patch, HADOOP-2141.v7.patch, HADOOP-2141.v8.patch We had one job with speculative execution hang. 4 reduce tasks were stuck with 95% completion because of a bad disk. Devaraj pointed out bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here. It would be nice if speculative execution also starts up when tasks stop making progress. Devaraj suggested bq. Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6487) Support ranges in hosts file
Support ranges in hosts file
Key: HADOOP-6487
URL: https://issues.apache.org/jira/browse/HADOOP-6487
Project: Hadoop Common
Issue Type: Improvement
Components: util
Reporter: Amar Kamat
It would be nice to have a range support for hosts file. For example host files
could look like
{code:title=hosts.include|borderStyle=solid}
abc1-10.com
abc21-30.com
{code}
{code:title=hosts.exclude|borderStyle=solid}
abc1-5.com
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6357) Reducers fail with OutOfMemoryError while copying Map outputs
[ https://issues.apache.org/jira/browse/HADOOP-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12773008#action_12773008 ] Amar Kamat commented on HADOOP-6357: Amazing!! Reducers fail with OutOfMemoryError while copying Map outputs - Key: HADOOP-6357 URL: https://issues.apache.org/jira/browse/HADOOP-6357 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.20.0 Reporter: Chandra Prakash Bhagtani Fix For: 0.20.0 Attachments: HADOOP-6357.patch Reducers fail while copying Map outputs with following exception java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1539) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216) ,Error: Reducer's memory usage keeps on increasing and ultimately exceeds -Xmx value I even tried with -Xmx6.5g to each reducer but it's still failing While looking into the reducer logs, I found that reducers were doing shuffleInMemory every time, rather than doing shuffleOnDisk -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6352) Streaming with -file non-existant-file just exits without a good message
[ https://issues.apache.org/jira/browse/HADOOP-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat resolved HADOOP-6352. Resolution: Duplicate MAPREDUCE-1068 is already opened to track this. Streaming with -file non-existant-file just exits without a good message Key: HADOOP-6352 URL: https://issues.apache.org/jira/browse/HADOOP-6352 Project: Hadoop Common Issue Type: Bug Reporter: Paul Tarjan $ hadoop jar $HADOOP_HOME/hadoop-streaming.jar ... -file non-existant-file Streaming Job Failed! $ Maybe a little message that says the file doesn't exist? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4933) ConcurrentModificationException in JobHistory.java
[
https://issues.apache.org/jira/browse/HADOOP-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12771896#action_12771896
]
Amar Kamat commented on HADOOP-4933:
This should also be Yahoo's distribution of hadoop. The uploaded patch applies.
Result of test-patch
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no tests are needed for
this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to
differ from the contents of the lib directories.
The eclipse class path issues is because jackson-core-asl-1.0.1.jar and
jackson-mapper-asl-1.0.1.jar info is not updated.
Running ant tests.
ConcurrentModificationException in JobHistory.java
--
Key: HADOOP-4933
URL: https://issues.apache.org/jira/browse/HADOOP-4933
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Priority: Blocker
Fix For: 0.21.0
Attachments: HADOOP-4933-v1.1.patch
{{JobHistory.java}} throws {{ConcurrentModificationException}} while finding
out the job history version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4933) ConcurrentModificationException in JobHistory.java
[
https://issues.apache.org/jira/browse/HADOOP-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12771908#action_12771908
]
Amar Kamat commented on HADOOP-4933:
TestLoadGenerator FAILED, TestBlockReplacement FAILED (timeout), TestHdfsProxy
FAILED, TestJobTrackerRestartWithCS FAILED (timeout).
None of these seems related to the patch.
ConcurrentModificationException in JobHistory.java
--
Key: HADOOP-4933
URL: https://issues.apache.org/jira/browse/HADOOP-4933
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Priority: Blocker
Fix For: 0.21.0
Attachments: HADOOP-4933-v1.1.patch
{{JobHistory.java}} throws {{ConcurrentModificationException}} while finding
out the job history version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal with -D with '=' in the value
[ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12748358#action_12748358 ] Amar Kamat commented on HADOOP-6215: All core tests except TestDistributedFileSystem and TestReduceFetch failed on core. All contrib tests except TestStreamingExitStatus, TestJobInitialization and TestQueueCapacities passed. fix GenericOptionParser to deal with -D with '=' in the value - Key: HADOOP-6215 URL: https://issues.apache.org/jira/browse/HADOOP-6215 Project: Hadoop Common Issue Type: Bug Reporter: Owen O'Malley Assignee: Amar Kamat Fix For: 0.20.1 Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch Currently the code ignores any -D option that has a = in the value. This is clearly wrong. In particular, -Dx=y=z should work and assign y=z to x. This is a regression from HADOOP-6213. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6213) Remove commons dependency on commons-cli2
Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Reporter: Amar Kamat GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6213: --- Affects Version/s: 0.20.0 Fix Version/s: 0.20.0 Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Fix For: 0.20.0 GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat reassigned HADOOP-6213: -- Assignee: Amar Kamat Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Assignee: Amar Kamat Fix For: 0.20.0 GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6213: --- Priority: Blocker (was: Major) Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Assignee: Amar Kamat Priority: Blocker Fix For: 0.20.0 GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6213: --- Attachment: HADOOP-6213-v1.0.patch Attaching a patch that removes the dependency on GenericOptionsParser on commons-cli2. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. Eclipse classpath check fails because commons-cli2 has to be manually deleted. Running ant tests. Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Assignee: Amar Kamat Priority: Blocker Fix For: 0.20.0 Attachments: HADOOP-6213-v1.0.patch GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12747892#action_12747892 ] Amar Kamat commented on HADOOP-6213: All test except TestDistributedFileSystem and TestReduceFetch passed. Testing contrib now. Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Assignee: Amar Kamat Priority: Blocker Fix For: 0.20.0 Attachments: HADOOP-6213-v1.0.patch GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12747910#action_12747910 ] Amar Kamat commented on HADOOP-6213: TestStreamingWithExitStatus, TestJobInitialization (timeout) and TestQueueCapacities (timeout) failed in contrib. Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Assignee: Amar Kamat Priority: Blocker Fix For: 0.20.0 Attachments: HADOOP-6213-v1.0.patch GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6213: --- Release Note: GenericOptionsParser depends on commons-cli2. This jira removed the dependency of branch 0.20 on commons-cli2. Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Assignee: Amar Kamat Priority: Blocker Fix For: 0.20.1 Attachments: HADOOP-6213-v1.0.patch GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6213) Remove commons dependency on commons-cli2
[ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-6213: --- Release Note: GenericOptionsParser in branch 0.20 depends on commons-cli2. This jira removes the dependency of branch 0.20 on commons-cli2 completely. The problem is seen after 'ant binary' where all the library files are copied to '$hadoop-home/lib' which already has commons-cli2. (was: GenericOptionsParser depends on commons-cli2. This jira removed the dependency of branch 0.20 on commons-cli2.) Remove commons dependency on commons-cli2 - Key: HADOOP-6213 URL: https://issues.apache.org/jira/browse/HADOOP-6213 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: 0.20.0 Reporter: Amar Kamat Assignee: Amar Kamat Priority: Blocker Fix For: 0.20.1 Attachments: HADOOP-6213-v1.0.patch GenericOptionsParser depends on commons-cli2 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal with -D with '=' in the value
[ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12748041#action_12748041 ] Amar Kamat commented on HADOOP-6215: The patch in HADOOP-6213 makes GenericOptionsParser same as in commons-trunk. We can fix both here. fix GenericOptionParser to deal with -D with '=' in the value - Key: HADOOP-6215 URL: https://issues.apache.org/jira/browse/HADOOP-6215 Project: Hadoop Common Issue Type: Bug Reporter: Owen O'Malley Assignee: Amar Kamat Fix For: 0.20.1 Currently the code ignores any -D option that has a = in the value. This is clearly wrong. In particular, -Dx=y=z should work and assign y=z to x. This is a regression from HADOOP-6213. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal with -D with '=' in the value
[ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12748100#action_12748100 ] Amar Kamat commented on HADOOP-6215: Testing on branch 20 fix GenericOptionParser to deal with -D with '=' in the value - Key: HADOOP-6215 URL: https://issues.apache.org/jira/browse/HADOOP-6215 Project: Hadoop Common Issue Type: Bug Reporter: Owen O'Malley Assignee: Amar Kamat Fix For: 0.20.1 Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch Currently the code ignores any -D option that has a = in the value. This is clearly wrong. In particular, -Dx=y=z should work and assign y=z to x. This is a regression from HADOOP-6213. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6178) GenericOptionsParser should ignore symlinks from archive filenames
GenericOptionsParser should ignore symlinks from archive filenames
--
Key: HADOOP-6178
URL: https://issues.apache.org/jira/browse/HADOOP-6178
Project: Hadoop Common
Issue Type: Bug
Components: util
Reporter: Amar Kamat
http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#Large+files+and+archives+in+Hadoop+Streaming
example doesnt work as expected.
Stack trace :
{code}
trunk$ ./bin/hadoop jar
build/contrib/streaming/hadoop-0.21.0-dev-streaming.jar -D mapred.map.tasks=1 \
-D mapred.reduce.tasks=1 -D mapred.job.name=Experiment \
-archives 'hdfs://namenode:port/user/me/cachedir.jar#testlink' -input
input -output output -mapper xargs cat -reducer cat
java.io.FileNotFoundException: File
hdfs://namenode:port/user/me/cachedir.jar#testlink does not exist.
at
org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:349)
at
org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:275)
at
org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:375)
at
org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:153)
at
org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:138)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at
org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4490) Map and Reduce tasks should run as the user who submitted the job
[ https://issues.apache.org/jira/browse/HADOOP-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-4490: --- Attachment: 2701949.4490.9.patch Attaching an example patch for branch 20 not to be committed. Map and Reduce tasks should run as the user who submitted the job - Key: HADOOP-4490 URL: https://issues.apache.org/jira/browse/HADOOP-4490 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Arun C Murthy Assignee: Hemanth Yamijala Fix For: 0.21.0 Attachments: 2701949.4490.9.patch, cluster_setup.pdf, cluster_setup.pdf, HADOOP-4490-1.patch, HADOOP-4490-1.patch, hadoop-4490-10.patch, hadoop-4490-11.patch, hadoop-4490-12.patch, hadoop-4490-13.patch, hadoop-4490-14.patch, HADOOP-4490-2.patch, HADOOP-4490-3.patch, hadoop-4490-4.patch, hadoop-4490-5.patch, hadoop-4490-6.patch, hadoop-4490-7.patch, hadoop-4490-8.patch, hadoop-4490-9.patch, hadoop-4490-br20-3.2.patch, hadoop-4490-br20-3.patch, hadoop-4490-design.pdf, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490_streaming.patch Currently the TaskTracker spawns the map/reduce tasks, resulting in them running as the user who started the TaskTracker. For security and accounting purposes the tasks should be run as the job-owner. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4490) Map and Reduce tasks should run as the user who submitted the job
[ https://issues.apache.org/jira/browse/HADOOP-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated HADOOP-4490: --- Attachment: (was: 2701949.4490.9.patch) Map and Reduce tasks should run as the user who submitted the job - Key: HADOOP-4490 URL: https://issues.apache.org/jira/browse/HADOOP-4490 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Arun C Murthy Assignee: Hemanth Yamijala Fix For: 0.21.0 Attachments: cluster_setup.pdf, cluster_setup.pdf, HADOOP-4490-1.patch, HADOOP-4490-1.patch, hadoop-4490-10.patch, hadoop-4490-11.patch, hadoop-4490-12.patch, hadoop-4490-13.patch, hadoop-4490-14.patch, HADOOP-4490-2.patch, HADOOP-4490-3.patch, hadoop-4490-4.patch, hadoop-4490-5.patch, hadoop-4490-6.patch, hadoop-4490-7.patch, hadoop-4490-8.patch, hadoop-4490-9.patch, hadoop-4490-br20-3.2.patch, hadoop-4490-br20-3.patch, hadoop-4490-design.pdf, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490_streaming.patch Currently the TaskTracker spawns the map/reduce tasks, resulting in them running as the user who started the TaskTracker. For security and accounting purposes the tasks should be run as the job-owner. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
