[jira] Commented: (PIG-660) Integration with Hadoop 0.20
[
https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740241#action_12740241
]
Dmitriy V. Ryaboy commented on PIG-660:
---
The shim patch posted above doesn't work as cleanly as desired; the current
build.xml has junit.hadoop.conf points to a directory in ${user.home}
This has an undesired effect -- a hadoop config file gets created the first
time you run ant, which among other things sets what class implements the
FileSytem interface. When ant gets re-run with a different hadoop version, 'ant
clean' does not clean out this file -- so an incorrect fs class name gets used.
Deleting the directory created by junit.hadoop.conf before rerunning fixes the
problem; so does putting the value of junit.hadoop.conf relative to
${build.dir} instead of ${user.home}.
As I am not sure how the Y! developers use their pigconf directories this thing
references, I do not know the appropriate way to proceed. Comments?
Integration with Hadoop 0.20
Key: PIG-660
URL: https://issues.apache.org/jira/browse/PIG-660
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.2.0
Environment: Hadoop 0.20
Reporter: Santhosh Srinivasan
Assignee: Santhosh Srinivasan
Fix For: 0.4.0
Attachments: PIG-660-for-branch-0.3.patch, PIG-660.patch,
PIG-660_1.patch, PIG-660_2.patch, PIG-660_3.patch, PIG-660_4.patch,
PIG-660_5.patch, pig_660_shims.patch, pig_660_shims_2.patch
With Hadoop 0.20, it will be possible to query the status of each map and
reduce in a map reduce job. This will allow better error reporting. Some of
the other items that could be on Hadoop's feature requests/bugs are
documented here for tracking.
1. Hadoop should return objects instead of strings when exceptions are thrown
2. The JobControl should handle all exceptions and report them appropriately.
For example, when the JobControl fails to launch jobs, it should handle
exceptions appropriately and should support APIs that query this state, i.e.,
failure to launch jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-660) Integration with Hadoop 0.20
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12740339#action_12740339 ] Dmitriy V. Ryaboy commented on PIG-660: --- Nate, Your stacktrace shows hadoop.dfs calls (as opposed to hdfs) which tells me it's looking for -- and finding -- hadoop 18 classes. Can you do this: export PIG_HADOOP_VERSION=20 ant clean; ant -Dhadoop.version=20 any try again? Just to be sure, try moving hadoop1* out of the lib directory (so that it for sure fails if it's trying to look for 18). Integration with Hadoop 0.20 Key: PIG-660 URL: https://issues.apache.org/jira/browse/PIG-660 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Environment: Hadoop 0.20 Reporter: Santhosh Srinivasan Assignee: Santhosh Srinivasan Fix For: 0.4.0 Attachments: PIG-660-for-branch-0.3.patch, PIG-660.patch, PIG-660_1.patch, PIG-660_2.patch, PIG-660_3.patch, PIG-660_4.patch, PIG-660_5.patch, pig_660_shims.patch, pig_660_shims_2.patch, pig_660_shims_3.patch With Hadoop 0.20, it will be possible to query the status of each map and reduce in a map reduce job. This will allow better error reporting. Some of the other items that could be on Hadoop's feature requests/bugs are documented here for tracking. 1. Hadoop should return objects instead of strings when exceptions are thrown 2. The JobControl should handle all exceptions and report them appropriately. For example, when the JobControl fails to launch jobs, it should handle exceptions appropriately and should support APIs that query this state, i.e., failure to launch jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-660) Integration with Hadoop 0.20
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12736264#action_12736264 ] Raghu Angadi commented on PIG-660: -- Currently, hadoop jar for 0.18 under lib/ is called hadoop18.jar. Should we change build.xml to use hadoop20.jar instead of hadoop18.jar? I can file a jira to commit hadoop20.jar. This might be replaced by updated jar when this jira is committed. Integration with Hadoop 0.20 Key: PIG-660 URL: https://issues.apache.org/jira/browse/PIG-660 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Environment: Hadoop 0.20 Reporter: Santhosh Srinivasan Assignee: Santhosh Srinivasan Fix For: 0.4.0 Attachments: PIG-660.patch, PIG-660_1.patch, PIG-660_2.patch, PIG-660_3.patch, PIG-660_4.patch, PIG-660_5.patch With Hadoop 0.20, it will be possible to query the status of each map and reduce in a map reduce job. This will allow better error reporting. Some of the other items that could be on Hadoop's feature requests/bugs are documented here for tracking. 1. Hadoop should return objects instead of strings when exceptions are thrown 2. The JobControl should handle all exceptions and report them appropriately. For example, when the JobControl fails to launch jobs, it should handle exceptions appropriately and should support APIs that query this state, i.e., failure to launch jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-660) Integration with Hadoop 0.20
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12736283#action_12736283 ] Santhosh Srinivasan commented on PIG-660: - The build.xml in the patch(es) have the reference to hadoop20.jar. The missing part is the hadoop20.jar that Pig can use to build its sources. Pig cannot use the hadoop20.jar coming from the Hadoop release. Integration with Hadoop 0.20 Key: PIG-660 URL: https://issues.apache.org/jira/browse/PIG-660 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Environment: Hadoop 0.20 Reporter: Santhosh Srinivasan Assignee: Santhosh Srinivasan Fix For: 0.4.0 Attachments: PIG-660.patch, PIG-660_1.patch, PIG-660_2.patch, PIG-660_3.patch, PIG-660_4.patch, PIG-660_5.patch With Hadoop 0.20, it will be possible to query the status of each map and reduce in a map reduce job. This will allow better error reporting. Some of the other items that could be on Hadoop's feature requests/bugs are documented here for tracking. 1. Hadoop should return objects instead of strings when exceptions are thrown 2. The JobControl should handle all exceptions and report them appropriately. For example, when the JobControl fails to launch jobs, it should handle exceptions appropriately and should support APIs that query this state, i.e., failure to launch jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-660) Integration with Hadoop 0.20
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12736297#action_12736297 ] Raghu Angadi commented on PIG-660: -- Thanks Olga and Santosh. build.xml change is already in the patch. Thanks. I will attach hadoop20.jar that works with PIG. This is useful for anyone to tryout the patch. This will also be used by zebra (PIG-833). Please commit the jar file to PIG trunk. It could be updated with a later version of hadoop-0.20 branch. Integration with Hadoop 0.20 Key: PIG-660 URL: https://issues.apache.org/jira/browse/PIG-660 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Environment: Hadoop 0.20 Reporter: Santhosh Srinivasan Assignee: Santhosh Srinivasan Fix For: 0.4.0 Attachments: PIG-660.patch, PIG-660_1.patch, PIG-660_2.patch, PIG-660_3.patch, PIG-660_4.patch, PIG-660_5.patch With Hadoop 0.20, it will be possible to query the status of each map and reduce in a map reduce job. This will allow better error reporting. Some of the other items that could be on Hadoop's feature requests/bugs are documented here for tracking. 1. Hadoop should return objects instead of strings when exceptions are thrown 2. The JobControl should handle all exceptions and report them appropriately. For example, when the JobControl fails to launch jobs, it should handle exceptions appropriately and should support APIs that query this state, i.e., failure to launch jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-660) Integration with Hadoop 0.20
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12671936#action_12671936 ] Santhosh Srinivasan commented on PIG-660: - JIRAs in Hadoop corresponding to items 1 and 2. 1. https://issues.apache.org/jira/browse/HADOOP-5201 2. https://issues.apache.org/jira/browse/HADOOP-5202 Integration with Hadoop 0.20 Key: PIG-660 URL: https://issues.apache.org/jira/browse/PIG-660 Project: Pig Issue Type: Bug Components: impl Affects Versions: types_branch Environment: Hadoop 0.20 Reporter: Santhosh Srinivasan Fix For: 0.1.0 With Hadoop 0.20, it will be possible to query the status of each map and reduce in a map reduce job. This will allow better error reporting. Some of the other items that could be on Hadoop's feature requests/bugs are documented here for tracking. 1. Hadoop should return objects instead of strings when exceptions are thrown 2. The JobControl should handle all exceptions and report them appropriately. For example, when the JobControl fails to launch jobs, it should handle exceptions appropriately and should support APIs that query this state, i.e., failure to launch jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
