[jira] Updated: (MAPREDUCE-1624) Document the job credentials and associated details to do with delegation tokens (on the client side)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-1624: --- Attachment: job-creds.patch Patch for Y20 > Document the job credentials and associated details to do with delegation > tokens (on the client side) > - > > Key: MAPREDUCE-1624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1624 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: job-creds.patch > > > Document the job credentials and associated details to do with delegation > tokens (on the client side) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1624) Document the job credentials and associated details to do with delegation tokens (on the client side)
Document the job credentials and associated details to do with delegation tokens (on the client side) - Key: MAPREDUCE-1624 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1624 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 0.22.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.22.0 Document the job credentials and associated details to do with delegation tokens (on the client side) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1507) The old MapReduce API is only partially deprecated
[ https://issues.apache.org/jira/browse/MAPREDUCE-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849051#action_12849051 ] Tom White commented on MAPREDUCE-1507: -- This change necessarily increases the number of deprecation warnings since it marks existing classes as deprecated. > The old MapReduce API is only partially deprecated > -- > > Key: MAPREDUCE-1507 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1507 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-1507.patch, MAPREDUCE-1507.patch > > > Not all of the old API is currently marked as deprecated. E.g. > org.apache.hadoop.mapred.OutputFormat is deprecated, but > org.apache.hadoop.mapred.FileOutputFormat isn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1618) JobStatus.getJobAcls() and setJobAcls should have javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1618: --- Status: Patch Available (was: Open) re-submitting for hudson > JobStatus.getJobAcls() and setJobAcls should have javadoc > - > > Key: MAPREDUCE-1618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1618 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Priority: Trivial > Fix For: 0.22.0 > > Attachments: patch-1618.txt > > > org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added > in MAPREDUCE-1307. They should have javadoc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1623) Apply audience and stability annotations to classes in mapred package
Apply audience and stability annotations to classes in mapred package - Key: MAPREDUCE-1623 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1623 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Tom White Assignee: Tom White There are lots of implementation classes in org.apache.hadoop.mapred which makes it difficult to see the user-level MapReduce API classes in the Javadoc. (See http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/package-summary.html for example.) By marking these implementation classes with the InterfaceAudience.Private annotation we can exclude them from user Javadoc (using HADOOP-6658). Later work will move the implementation classes into o.a.h.mapreduce.server and related packages (see MAPREDUCE-561), but applying the annotations is a good first step. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1618) JobStatus.getJobAcls() and setJobAcls should have javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1618: --- Status: Open (was: Patch Available) > JobStatus.getJobAcls() and setJobAcls should have javadoc > - > > Key: MAPREDUCE-1618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1618 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Priority: Trivial > Fix For: 0.22.0 > > Attachments: patch-1618.txt > > > org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added > in MAPREDUCE-1307. They should have javadoc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1622) Include slf4j dependencies in binary tarball
[
https://issues.apache.org/jira/browse/MAPREDUCE-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849021#action_12849021
]
Chris Douglas commented on MAPREDUCE-1622:
--
Core failure is caused by HADOOP-6645, but this should not affect unit tests
> Include slf4j dependencies in binary tarball
>
>
> Key: MAPREDUCE-1622
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1622
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Reporter: Chris Douglas
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: M1622-0.patch
>
>
> After MAPREDUCE-1556 and HADOOP-6486, starting \*Trackers from a binary
> tarball produces the following warning:
> {noformat}
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> 2010-03-18 01:11:32.988::INFO: Logging to STDERR via
> org.mortbay.log.StdErrLog
> 2010-03-18 01:11:33.056::INFO: jetty-6.1.14
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
[ https://issues.apache.org/jira/browse/MAPREDUCE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-1545: --- Attachment: mr-1545-y20s-v2.patch > Add 'first-task-launched' to job-summary > > > Key: MAPREDUCE-1545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Arun C Murthy >Assignee: Luke Lu > Fix For: 0.22.0 > > Attachments: mr-1545-y20s-v1.patch, mr-1545-y20s-v2.patch > > > It would be useful to track 'first-task-launched' time to job-summary for > better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1622) Include slf4j dependencies in binary tarball
[
https://issues.apache.org/jira/browse/MAPREDUCE-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849006#action_12849006
]
Hadoop QA commented on MAPREDUCE-1622:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12439609/M1622-0.patch
against trunk revision 926449.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/53/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/53/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/53/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/53/console
This message is automatically generated.
> Include slf4j dependencies in binary tarball
>
>
> Key: MAPREDUCE-1622
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1622
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Reporter: Chris Douglas
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: M1622-0.patch
>
>
> After MAPREDUCE-1556 and HADOOP-6486, starting \*Trackers from a binary
> tarball produces the following warning:
> {noformat}
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> 2010-03-18 01:11:32.988::INFO: Logging to STDERR via
> org.mortbay.log.StdErrLog
> 2010-03-18 01:11:33.056::INFO: jetty-6.1.14
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848980#action_12848980 ] Rodrigo Schmidt commented on MAPREDUCE-1585: You definitely have a point there after the last problems with HADOOP-6591. I'll change the patch. > Create Hadoop Archives version 2 with filenames URL-encoded > --- > > Key: MAPREDUCE-1585 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: harchive >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1585.1.patch, MAPREDUCE-1585.patch > > > Hadoop Archives version 1 don't cope with files that have spaces on their > names. > One proposal is to URLEncode filenames inside the index file (version 2, > refers to HADOOP-6591). > This task is to allow the creation of version 2 files that have file names > encoded appropriately. It currently depends on HADOOP-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848977#action_12848977 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1585: --- TestHadoopArchives contains some useful test cases. How about change it to work with the new version but not removing it? > Create Hadoop Archives version 2 with filenames URL-encoded > --- > > Key: MAPREDUCE-1585 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: harchive >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1585.1.patch, MAPREDUCE-1585.patch > > > Hadoop Archives version 1 don't cope with files that have spaces on their > names. > One proposal is to URLEncode filenames inside the index file (version 2, > refers to HADOOP-6591). > This task is to allow the creation of version 2 files that have file names > encoded appropriately. It currently depends on HADOOP-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1338) need security keys storage solution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated MAPREDUCE-1338: -- Tags: security Release Note: new command line argument: tokensFile - path to the file with clients secret keys in JSON format > need security keys storage solution > --- > > Key: MAPREDUCE-1338 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1338 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: HADOOP-6325.patch, MAPREDUCE-1338-10.patch, > MAPREDUCE-1338-2.patch, MAPREDUCE-1338-4.patch, MAPREDUCE-1338-6.patch, > MAPREDUCE-1338-7.patch, MAPREDUCE-1338-8.patch, MAPREDUCE-1338-9.patch, > MAPREDUCE-1338-BP20-2.patch, MAPREDUCE-1338-BP20-3.patch, MAPREDUCE-1338.patch > > > set, get, store, load security keys > key alias - byte[] > key value - byte[] > store/load from DataInput/Output stream -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1383) Allow storage and caching of delegation token.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated MAPREDUCE-1383: -- Tags: security Release Note: mapreduce.job.hdfs-servers - declares hdfs servers to be used by the job, so client can pre-fetch delegation tokens for thsese servers (comma separated list of NameNodes). > Allow storage and caching of delegation token. > -- > > Key: MAPREDUCE-1383 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1383 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1383-1.patch, MAPREDUCE-1383-10.patch, > MAPREDUCE-1383-11.patch, MAPREDUCE-1383-2.patch, MAPREDUCE-1383-5.patch, > MAPREDUCE-1383-6.patch, MAPREDUCE-1383-9.patch, MAPREDUCE-1383-BP20-3.patch, > MAPREDUCE-1383-BP20-4.patch, MAPREDUCE-1383-BP20-5.patch, > MAPREDUCE-1383-BP20-6.patch, MAPREDUCE-1383-BP20-7.patch > > > Client needs to obtain delegation tokens from all the NameNodes it is going > to work with and pass it to the application. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
[ https://issues.apache.org/jira/browse/MAPREDUCE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu reassigned MAPREDUCE-1545: -- Assignee: Luke Lu (was: Arun C Murthy) > Add 'first-task-launched' to job-summary > > > Key: MAPREDUCE-1545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Arun C Murthy >Assignee: Luke Lu > Fix For: 0.22.0 > > Attachments: mr-1545-y20s-v1.patch > > > It would be useful to track 'first-task-launched' time to job-summary for > better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1430) JobTracker should be able to renew delegation tokens for the jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated MAPREDUCE-1430: -- Release Note: mapreduce.job.complete.cancel.delegation.tokens - if false - don't cancel delegation token renewal when the job is complete, because it may be used by some other job. > JobTracker should be able to renew delegation tokens for the jobs > - > > Key: MAPREDUCE-1430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1430 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: 1430-BP20-Add.patch, 1430-bp20-bugfix.patch, > 1430-dd1.patch, 1430-dd2.patch, 1430-dd3.patch, 1430-dd4-BP20.patch, > 1430-dd4.patch, MAPREDUCE-1430-12.patch, MAPREDUCE-1430-13.patch, > MAPREDUCE-1430-14.patch, MAPREDUCE-1430-5.patch, MAPREDUCE-1430-6.patch, > MAPREDUCE-1430-8.patch > > > JobTracker should automatically renew delegation tokens for the jobs it is > currently running. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
[ https://issues.apache.org/jira/browse/MAPREDUCE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-1545: --- Attachment: mr-1545-y20s-v1.patch Patch for yahoo 0.20.1xx (y20s) branch. > Add 'first-task-launched' to job-summary > > > Key: MAPREDUCE-1545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.22.0 > > Attachments: mr-1545-y20s-v1.patch > > > It would be useful to track 'first-task-launched' time to job-summary for > better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1558) specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated MAPREDUCE-1558: -- Release Note: new config: hadoop.security.service.user.name.key this setting points to the server principal for RefreshUserToGroupMappingsProtocol. The value should be either NN or JT principal depending if it is used in DFAdmin or MRAdmin. The value is set by the application. No need for default value. was: adds new conf setting: CommonConfigurationKeys.HADOOP_SECURITY_SERVICE_USER_NAME_KEY this setting points to the server principal for RefreshUserToGroupMappingsProtocol. The value should be either NN or JT principal depending if it is used in DFAdmin or MRAdmin. > specify correct server principal for RefreshAuthorizationPolicyProtocol and > RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612) > - > > Key: MAPREDUCE-1558 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1558 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1558.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1489) DataDrivenDBInputFormat should not query the database when generating only one split
[ https://issues.apache.org/jira/browse/MAPREDUCE-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848938#action_12848938 ] Aaron Kimball commented on MAPREDUCE-1489: -- Test failure in HAR is unrelated > DataDrivenDBInputFormat should not query the database when generating only > one split > > > Key: MAPREDUCE-1489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1489 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1489.patch > > > DataDrivenDBInputFormat runs a query to establish bounding values for each > split it generates; but if it's going to generate only one split > (mapreduce.job.maps == 1), then there's no reason to do this. This will > remove overhead associated with a single-threaded import of a non-indexed > table since it avoids a full table scan. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1489) DataDrivenDBInputFormat should not query the database when generating only one split
[ https://issues.apache.org/jira/browse/MAPREDUCE-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848928#action_12848928 ] Hadoop QA commented on MAPREDUCE-1489: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435756/MAPREDUCE-1489.patch against trunk revision 926449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/52/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/52/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/52/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/52/console This message is automatically generated. > DataDrivenDBInputFormat should not query the database when generating only > one split > > > Key: MAPREDUCE-1489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1489 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1489.patch > > > DataDrivenDBInputFormat runs a query to establish bounding values for each > split it generates; but if it's going to generate only one split > (mapreduce.job.maps == 1), then there's no reason to do this. This will > remove overhead associated with a single-threaded import of a non-indexed > table since it avoids a full table scan. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1622) Include slf4j dependencies in binary tarball
[
https://issues.apache.org/jira/browse/MAPREDUCE-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1622:
-
Attachment: M1622-0.patch
This resolves the missing dependency by adding the slf4j deps to {{common}}
instead of {{test}}
> Include slf4j dependencies in binary tarball
>
>
> Key: MAPREDUCE-1622
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1622
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Reporter: Chris Douglas
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: M1622-0.patch
>
>
> After MAPREDUCE-1556 and HADOOP-6486, starting \*Trackers from a binary
> tarball produces the following warning:
> {noformat}
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> 2010-03-18 01:11:32.988::INFO: Logging to STDERR via
> org.mortbay.log.StdErrLog
> 2010-03-18 01:11:33.056::INFO: jetty-6.1.14
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1556) upgrade to Avro 1.3.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848902#action_12848902 ] Chris Douglas commented on MAPREDUCE-1556: -- Yea, that worked. Filed MAPREDUCE-1622 > upgrade to Avro 1.3.0 > - > > Key: MAPREDUCE-1556 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1556 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Doug Cutting >Assignee: Doug Cutting > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1556.patch, MAPREDUCE-1556.patch, > MAPREDUCE-1556.patch > > > Avro 1.3.0 has now been released. HADOOP-6486 and HDFS-892 require it, and > the version of Avro used by MapReduce should be synchronized with these > projects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1622) Include slf4j dependencies in binary tarball
[
https://issues.apache.org/jira/browse/MAPREDUCE-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1622:
-
Status: Patch Available (was: Open)
> Include slf4j dependencies in binary tarball
>
>
> Key: MAPREDUCE-1622
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1622
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Reporter: Chris Douglas
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: M1622-0.patch
>
>
> After MAPREDUCE-1556 and HADOOP-6486, starting \*Trackers from a binary
> tarball produces the following warning:
> {noformat}
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> 2010-03-18 01:11:32.988::INFO: Logging to STDERR via
> org.mortbay.log.StdErrLog
> 2010-03-18 01:11:33.056::INFO: jetty-6.1.14
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1622) Include slf4j dependencies in binary tarball
Include slf4j dependencies in binary tarball
Key: MAPREDUCE-1622
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1622
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Reporter: Chris Douglas
Priority: Minor
Fix For: 0.22.0
After MAPREDUCE-1556 and HADOOP-6486, starting \*Trackers from a binary tarball
produces the following warning:
{noformat}
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
2010-03-18 01:11:32.988::INFO: Logging to STDERR via org.mortbay.log.StdErrLog
2010-03-18 01:11:33.056::INFO: jetty-6.1.14
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1556) upgrade to Avro 1.3.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848893#action_12848893
]
Chris Douglas commented on MAPREDUCE-1556:
--
Should it be in {{common->master}} instead of {{test->master}} ?
> upgrade to Avro 1.3.0
> -
>
> Key: MAPREDUCE-1556
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1556
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Reporter: Doug Cutting
>Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1556.patch, MAPREDUCE-1556.patch,
> MAPREDUCE-1556.patch
>
>
> Avro 1.3.0 has now been released. HADOOP-6486 and HDFS-892 require it, and
> the version of Avro used by MapReduce should be synchronized with these
> projects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1556) upgrade to Avro 1.3.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848892#action_12848892 ] Chris Douglas commented on MAPREDUCE-1556: -- bq. It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts? I think Avro marking it as a dependency masked MR/HDFS/Common's dependence on it, which isn't explicit? It's printed whenever I try to start a MR \*Tracker from a binary tarball. > upgrade to Avro 1.3.0 > - > > Key: MAPREDUCE-1556 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1556 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Doug Cutting >Assignee: Doug Cutting > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1556.patch, MAPREDUCE-1556.patch, > MAPREDUCE-1556.patch > > > Avro 1.3.0 has now been released. HADOOP-6486 and HDFS-892 require it, and > the version of Avro used by MapReduce should be synchronized with these > projects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1621) Streaming's TextOutputReader.getLastOutput throws NPE if it has never read any output
Streaming's TextOutputReader.getLastOutput throws NPE if it has never read any output - Key: MAPREDUCE-1621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1621 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 0.20.2, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon If TextOutputReader.readKeyValue() has never successfully read a line, then its bytes member will be left null. Thus when logging a task failure, PipeMapRed.getContext() can trigger an NPE when it calls outReader_.getLastOutput(). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1620) Hadoop should serialize the Configration after the call to getSplits() to the backend such that any changes to the Configuration in getSplits() is serialized to the ba
Hadoop should serialize the Configration after the call to getSplits() to the backend such that any changes to the Configuration in getSplits() is serialized to the backend Key: MAPREDUCE-1620 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1620 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 0.20.1 Reporter: Pradeep Kamath In 0.20.1 and 0.20.2, when using the new API, while working on the next pig release we discovered that the hadoop code makes a copy of the Configuration and hands a copy to the getSplits() call. Any changes to the Configuration made in getSplits() are on that copy. However the original Configuraiton is the one which gets serialized to the backend - hence any changes made to the Configuration in the getSplits() implementation does not get serialized to the backend. In a framework like Pig, there are usecases for writing information into the Configuration during getSplits - it would be helpful if Hadoop would ensure that these changes get serialized to the backend. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1556) upgrade to Avro 1.3.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848865#action_12848865 ] Doug Cutting commented on MAPREDUCE-1556: - > The slf4j-simple jar is no longer included. It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts? That error message is triggered when no slf4j implementation is on the classpath. When is this printed? > upgrade to Avro 1.3.0 > - > > Key: MAPREDUCE-1556 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1556 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Doug Cutting >Assignee: Doug Cutting > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1556.patch, MAPREDUCE-1556.patch, > MAPREDUCE-1556.patch > > > Avro 1.3.0 has now been released. HADOOP-6486 and HDFS-892 require it, and > the version of Avro used by MapReduce should be synchronized with these > projects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1556) upgrade to Avro 1.3.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848854#action_12848854
]
Chris Douglas commented on MAPREDUCE-1556:
--
With HADOOP-6486, the following warnings show up on trunk using a binary
tarball:
{noformat}
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
2010-03-18 01:11:32.988::INFO: Logging to STDERR via org.mortbay.log.StdErrLog
2010-03-18 01:11:33.056::INFO: jetty-6.1.14
{noformat}
The slf4j-simple jar is no longer included.
> upgrade to Avro 1.3.0
> -
>
> Key: MAPREDUCE-1556
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1556
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Reporter: Doug Cutting
>Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1556.patch, MAPREDUCE-1556.patch,
> MAPREDUCE-1556.patch
>
>
> Avro 1.3.0 has now been released. HADOOP-6486 and HDFS-892 require it, and
> the version of Avro used by MapReduce should be synchronized with these
> projects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1489) DataDrivenDBInputFormat should not query the database when generating only one split
[ https://issues.apache.org/jira/browse/MAPREDUCE-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1489: - Status: Patch Available (was: Open) Hudson failed to test; recycling. > DataDrivenDBInputFormat should not query the database when generating only > one split > > > Key: MAPREDUCE-1489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1489 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1489.patch > > > DataDrivenDBInputFormat runs a query to establish bounding values for each > split it generates; but if it's going to generate only one split > (mapreduce.job.maps == 1), then there's no reason to do this. This will > remove overhead associated with a single-threaded import of a non-indexed > table since it avoids a full table scan. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1489) DataDrivenDBInputFormat should not query the database when generating only one split
[ https://issues.apache.org/jira/browse/MAPREDUCE-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1489: - Status: Open (was: Patch Available) > DataDrivenDBInputFormat should not query the database when generating only > one split > > > Key: MAPREDUCE-1489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1489 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1489.patch > > > DataDrivenDBInputFormat runs a query to establish bounding values for each > split it generates; but if it's going to generate only one split > (mapreduce.job.maps == 1), then there's no reason to do this. This will > remove overhead associated with a single-threaded import of a non-indexed > table since it avoids a full table scan. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1618) JobStatus.getJobAcls() and setJobAcls should have javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848714#action_12848714 ] Hadoop QA commented on MAPREDUCE-1618: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439554/patch-1618.txt against trunk revision 926449. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/51/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/51/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/51/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/51/console This message is automatically generated. > JobStatus.getJobAcls() and setJobAcls should have javadoc > - > > Key: MAPREDUCE-1618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1618 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Priority: Trivial > Fix For: 0.22.0 > > Attachments: patch-1618.txt > > > org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added > in MAPREDUCE-1307. They should have javadoc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1570) Shuffle stage - Key and Group Comparators
[ https://issues.apache.org/jira/browse/MAPREDUCE-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848664#action_12848664 ] Hadoop QA commented on MAPREDUCE-1570: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439534/MAPREDUCE-1570.patch against trunk revision 926449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/350/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/350/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/350/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/350/console This message is automatically generated. > Shuffle stage - Key and Group Comparators > - > > Key: MAPREDUCE-1570 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1570 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mrunit >Reporter: Chris White >Assignee: Chris White >Priority: Minor > Attachments: MAPREDUCE-1570.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Shuffle method in org.apache.hadoop.mrunit.MapReduceDriverBase doesn't > currently allow the use of custom GroupingComparator and SortComparator. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1420: - Release Note: Fixed a bug in the testcase TestTTResourceReporting. > TestTTResourceReporting failing in trunk > > > Key: MAPREDUCE-1420 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, > output.rtf > > > TestTTResourceReporting failing in trunk. > The most specific issue from the logs seems to be : Error executing shell > command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process > Link : > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ > Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (MAPREDUCE-1561) mapreduce patch tests hung with "java.lang.OutOfMemoryError: Java heap space"
[
https://issues.apache.org/jira/browse/MAPREDUCE-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V reopened MAPREDUCE-1561:
--
Reopening for setting proper resolution status.
> mapreduce patch tests hung with "java.lang.OutOfMemoryError: Java heap space"
> -
>
> Key: MAPREDUCE-1561
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1561
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Giridharan Kesavan
>Assignee: Doug Cutting
>
> http://hudson.zones.apache.org/hudson/view/Mapreduce/job/Mapreduce-Patch-h9.grid.sp2.yahoo.net/4/console
> Error form the console:
> [exec] [junit] 10/03/05 04:08:29 INFO datanode.DataNode: PacketResponder
> 2 for block blk_-3280111748864197295_19758 terminating
> [exec] [junit] 10/03/05 04:08:29 INFO hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:46067 is added to
> blk_-3280111748864197295_19758{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:46067|RBW],
> ReplicaUnderConstruction[127.0.0.1:37626|RBW],
> ReplicaUnderConstruction[127.0.0.1:48886|RBW]]} size 0
> [exec] [junit] 10/03/05 04:08:29 INFO hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /tmp/hadoop-hudson/mapred/system/job_20100304162726530_3751/job-info is
> closed by DFSClient_79157028
> [exec] [junit] 10/03/05 04:08:29 INFO mapred.JobTracker: Job
> job_20100304162726530_3751 added successfully for user 'hudson' to queue
> 'default'
> [exec] [junit] 10/03/05 04:08:29 INFO mapred.JobTracker:
> Initializing job_20100304162726530_3751
> [exec] [junit] 10/03/05 04:08:29 INFO mapred.JobInProgress:
> Initializing job_20100304162726530_3751
> [exec] [junit] 10/03/05 04:08:29 INFO mapreduce.Job: Running job:
> job_20100304162726530_3751
> [exec] [junit] 10/03/05 04:08:29 INFO jobhistory.JobHistory:
> SetupWriter, creating file
> file:/grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h9.grid.sp2.yahoo.net/trunk/build/contrib/raid/test/logs/history/job_20100304162726530_3751_hudson
> [exec] [junit] 10/03/05 04:08:29 ERROR mapred.JobTracker: Job
> initialization failed:
> [exec] [junit] org.apache.avro.AvroRuntimeException:
> java.lang.NoSuchFieldException: _SCHEMA
> [exec] [junit] at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:50)
> [exec] [junit] at
> org.apache.avro.reflect.ReflectData.getSchema(ReflectData.java:210)
> [exec] [junit] at
> org.apache.avro.specific.SpecificDatumWriter.(SpecificDatumWriter.java:28)
> [exec] [junit] at
> org.apache.hadoop.mapreduce.jobhistory.EventWriter.(EventWriter.java:47)
> [exec] [junit] at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.setupEventWriter(JobHistory.java:252)
> [exec] [junit] at
> org.apache.hadoop.mapred.JobInProgress.logSubmissionToJobHistory(JobInProgress.java:710)
> [exec] [junit] at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:619)
> [exec] [junit] at
> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3256)
> [exec] [junit] at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
> [exec] [junit] at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [exec] [junit] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [exec] [junit] at java.lang.Thread.run(Thread.java:619)
> [exec] [junit] Caused by: java.lang.NoSuchFieldException: _SCHEMA
> [exec] [junit] at
> java.lang.Class.getDeclaredField(Class.java:1882)
> [exec] [junit] at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:48)
> [exec] [junit] ... 11 more
> [exec] [junit]
> [exec] [junit] Exception in thread "pool-1-thread-3"
> java.lang.OutOfMemoryError: Java heap space
> [exec] [junit] at java.util.Arrays.copyOf(Arrays.java:2786)
> [exec] [junit] at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> [exec] [junit] at
> java.io.PrintStream.write(PrintStream.java:430)
> [exec] [junit] at
> org.apache.tools.ant.util.TeeOutputStream.write(TeeOutputStream.java:81)
> [exec] [junit] at
> java.io.PrintStream.write(PrintStream.java:430)
> [exec] [junit] at
> sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
> [exec] [junit] at
> sun.nio.cs.StreamEncoder.implFlushBuffer(S
[jira] Resolved: (MAPREDUCE-1561) mapreduce patch tests hung with "java.lang.OutOfMemoryError: Java heap space"
[
https://issues.apache.org/jira/browse/MAPREDUCE-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V resolved MAPREDUCE-1561.
--
Resolution: Duplicate
> mapreduce patch tests hung with "java.lang.OutOfMemoryError: Java heap space"
> -
>
> Key: MAPREDUCE-1561
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1561
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Giridharan Kesavan
>Assignee: Doug Cutting
>
> http://hudson.zones.apache.org/hudson/view/Mapreduce/job/Mapreduce-Patch-h9.grid.sp2.yahoo.net/4/console
> Error form the console:
> [exec] [junit] 10/03/05 04:08:29 INFO datanode.DataNode: PacketResponder
> 2 for block blk_-3280111748864197295_19758 terminating
> [exec] [junit] 10/03/05 04:08:29 INFO hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:46067 is added to
> blk_-3280111748864197295_19758{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:46067|RBW],
> ReplicaUnderConstruction[127.0.0.1:37626|RBW],
> ReplicaUnderConstruction[127.0.0.1:48886|RBW]]} size 0
> [exec] [junit] 10/03/05 04:08:29 INFO hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /tmp/hadoop-hudson/mapred/system/job_20100304162726530_3751/job-info is
> closed by DFSClient_79157028
> [exec] [junit] 10/03/05 04:08:29 INFO mapred.JobTracker: Job
> job_20100304162726530_3751 added successfully for user 'hudson' to queue
> 'default'
> [exec] [junit] 10/03/05 04:08:29 INFO mapred.JobTracker:
> Initializing job_20100304162726530_3751
> [exec] [junit] 10/03/05 04:08:29 INFO mapred.JobInProgress:
> Initializing job_20100304162726530_3751
> [exec] [junit] 10/03/05 04:08:29 INFO mapreduce.Job: Running job:
> job_20100304162726530_3751
> [exec] [junit] 10/03/05 04:08:29 INFO jobhistory.JobHistory:
> SetupWriter, creating file
> file:/grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h9.grid.sp2.yahoo.net/trunk/build/contrib/raid/test/logs/history/job_20100304162726530_3751_hudson
> [exec] [junit] 10/03/05 04:08:29 ERROR mapred.JobTracker: Job
> initialization failed:
> [exec] [junit] org.apache.avro.AvroRuntimeException:
> java.lang.NoSuchFieldException: _SCHEMA
> [exec] [junit] at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:50)
> [exec] [junit] at
> org.apache.avro.reflect.ReflectData.getSchema(ReflectData.java:210)
> [exec] [junit] at
> org.apache.avro.specific.SpecificDatumWriter.(SpecificDatumWriter.java:28)
> [exec] [junit] at
> org.apache.hadoop.mapreduce.jobhistory.EventWriter.(EventWriter.java:47)
> [exec] [junit] at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.setupEventWriter(JobHistory.java:252)
> [exec] [junit] at
> org.apache.hadoop.mapred.JobInProgress.logSubmissionToJobHistory(JobInProgress.java:710)
> [exec] [junit] at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:619)
> [exec] [junit] at
> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3256)
> [exec] [junit] at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
> [exec] [junit] at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [exec] [junit] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [exec] [junit] at java.lang.Thread.run(Thread.java:619)
> [exec] [junit] Caused by: java.lang.NoSuchFieldException: _SCHEMA
> [exec] [junit] at
> java.lang.Class.getDeclaredField(Class.java:1882)
> [exec] [junit] at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:48)
> [exec] [junit] ... 11 more
> [exec] [junit]
> [exec] [junit] Exception in thread "pool-1-thread-3"
> java.lang.OutOfMemoryError: Java heap space
> [exec] [junit] at java.util.Arrays.copyOf(Arrays.java:2786)
> [exec] [junit] at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> [exec] [junit] at
> java.io.PrintStream.write(PrintStream.java:430)
> [exec] [junit] at
> org.apache.tools.ant.util.TeeOutputStream.write(TeeOutputStream.java:81)
> [exec] [junit] at
> java.io.PrintStream.write(PrintStream.java:430)
> [exec] [junit] at
> sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
> [exec] [junit] at
> sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
[jira] Updated: (MAPREDUCE-1611) Refresh nodes and refresh queues doesnt work with service authorization enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-1611:
-
Release Note: Fixed a bug that caused all the AdminOperationsProtocol
operations to fail when service-level authorization is enabled. The problem is
solved by registering AdminOperationsProtocol also with
MapReducePolicyProvider. (was: Fixes the bug wherein if service-level
authorization is enabled then all AdminOperationsProtocol operations fail. If
service-level authorization is enabled, then all services returned by
MapReducePolicyProvider are authenticated by ServiceAuthorizationManager. Prior
to this patch, AdminOperationsProtocol was not registered with
MapReducePolicyProvider causing it to fail.)
> Refresh nodes and refresh queues doesnt work with service authorization
> enabled
> ---
>
> Key: MAPREDUCE-1611
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1611
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1611-20100319-ydist.txt, mr-1611-v1.0.patch
>
>
> If service-level authorization enabled (i.e _hadoop.security.authorization_
> set to *true*) for MapReduce then refreshing the node and queue fails with
> the following message
> {noformat}
> Protocol interface org.apache.hadoop.mapred.AdminOperationsProtocol is not
> known.
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1515) need to pass down java5 and forrest home variables
[ https://issues.apache.org/jira/browse/MAPREDUCE-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848643#action_12848643 ] Hadoop QA commented on MAPREDUCE-1515: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439536/MAPREDUCE-1515.patch against trunk revision 926449. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/50/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/50/console This message is automatically generated. > need to pass down java5 and forrest home variables > -- > > Key: MAPREDUCE-1515 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1515 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.22.0 > > Attachments: atx20100224a.patch, m-1515.patch, MAPREDUCE-1515.patch > > > Currently, the build script doesn't pass down the variables for java5 and > forrest, so the build breaks unless they are on the command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-890: Release Note: Fixed a bug that failed jobs that are run by the same user who started the mapreduce system(cluster). (was: Allows the user who started the mapreduce system(cluster) to run jobs) > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.22.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, > MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-476) extend DistributedCache to work locally (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-476: Release Note: Extended DistributedCache to work with LocalJobRunner. > extend DistributedCache to work locally (LocalJobRunner) > > > Key: MAPREDUCE-476 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-476 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: sam rash >Assignee: Philip Zeyliger >Priority: Minor > Fix For: 0.21.0 > > Attachments: 476.20S-2.patch, HADOOP-2914-v1-full.patch, > HADOOP-2914-v1-since-4041.patch, HADOOP-2914-v2.patch, HADOOP-2914-v3.patch, > MAPREDUCE-476-20090814.1.txt, MAPREDUCE-476-20090818.txt, > MAPREDUCE-476-v2-vs-v3.patch, MAPREDUCE-476-v2-vs-v3.try2.patch, > MAPREDUCE-476-v2-vs-v4.txt, MAPREDUCE-476-v2.patch, MAPREDUCE-476-v3.patch, > MAPREDUCE-476-v3.try2.patch, MAPREDUCE-476-v4-requires-MR711.patch, > MAPREDUCE-476-v5-requires-MR711.patch, MAPREDUCE-476-v7.patch, > MAPREDUCE-476-v8.patch, MAPREDUCE-476-v9.patch, MAPREDUCE-476.patch, > v6-to-v7.patch > > > The DistributedCache does not work locally when using the outlined recipe at > http://hadoop.apache.org/core/docs/r0.16.0/api/org/apache/hadoop/filecache/DistributedCache.html > > Ideally, LocalJobRunner would take care of populating the JobConf and copying > remote files to the local file sytem (http, assume hdfs = default fs = local > fs when doing local development. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-408) TestKillSubProcesses fails with assertion failure sometimes
[
https://issues.apache.org/jira/browse/MAPREDUCE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-408:
Release Note: Fixed a bug in the testcase TestKillSubProcesses. (was:
Fixes a bug in testcase TestKillSubProcesses.)
> TestKillSubProcesses fails with assertion failure sometimes
> ---
>
> Key: MAPREDUCE-408
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-408
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Ravi Gummadi
> Fix For: 0.21.0
>
> Attachments: MR-408-yhadoop20.patch, MR-408.patch, MR-408.v1.1.patch,
> MR-408.v1.1.y20.patch, MR-408.v1.patch
>
>
> org.apache.hadoop.mapred.TestKillSubProcesses.testJobKillFailAndSucceed fails
> sometimes with following error Message:
> {noformat}
> Unexpected: The subprocess at level 3 in the subtree is not alive before Job
> completion
> {noformat}
> Stacktrace
> {noformat}
> junit.framework.AssertionFailedError: Unexpected: The subprocess at level 3
> in the subtree is not alive before Job completion
> at
> org.apache.hadoop.mapred.TestKillSubProcesses.runJobAndSetProcessHandle(TestKillSubProcesses.java:221)
> at
> org.apache.hadoop.mapred.TestKillSubProcesses.runFailingJobAndValidate(TestKillSubProcesses.java:112)
> at
> org.apache.hadoop.mapred.TestKillSubProcesses.runTests(TestKillSubProcesses.java:327)
> at
> org.apache.hadoop.mapred.TestKillSubProcesses.testJobKillFailAndSucceed(TestKillSubProcesses.java:310)
> {noformat}
> one such failure at
> http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/testReport/org.apache.hadoop.mapred/TestKillSubProcesses/testJobKillFailAndSucceed/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1517) streaming should support running on background
[ https://issues.apache.org/jira/browse/MAPREDUCE-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bochun Bai updated MAPREDUCE-1517: -- Attachment: contrib-streaming-background.patch > streaming should support running on background > -- > > Key: MAPREDUCE-1517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1517 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/streaming >Reporter: Bochun Bai > Attachments: contrib-streaming-background.patch, > contrib-streaming-background.patch > > > StreamJob submit the job and use a while loop monitor the progress. > I prefer it running on background. > Just add "&" at the end of command is a alternative solution, but it keeps a > java process on client machine. > When submit hundreds jobs at the same time, the client machine is overloaded. > Adding a -background option to StreamJob, tell it only submit and don't > monitor the progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1619) Eclipse .classpath file should be generated from Ivy files to avoid duplicating dependencies
Eclipse .classpath file should be generated from Ivy files to avoid duplicating dependencies Key: MAPREDUCE-1619 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1619 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.20.3 Reporter: Paolo Castagna Priority: Trivial Fix For: 0.21.0 Using Ant-Eclipse (http://sourceforge.net/projects/ant-eclipse/) it's possible to generate Eclipse files from Ivy files, this avoids unnecessary duplication of dependencies information. See: HADOOP-6407 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1618) JobStatus.getJobAcls() and setJobAcls should have javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1618: --- Status: Patch Available (was: Open) > JobStatus.getJobAcls() and setJobAcls should have javadoc > - > > Key: MAPREDUCE-1618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1618 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Priority: Trivial > Fix For: 0.22.0 > > Attachments: patch-1618.txt > > > org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added > in MAPREDUCE-1307. They should have javadoc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1618) JobStatus.getJobAcls() and setJobAcls should have javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1618: --- Attachment: patch-1618.txt Patch adding the javadoc > JobStatus.getJobAcls() and setJobAcls should have javadoc > - > > Key: MAPREDUCE-1618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1618 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Priority: Trivial > Fix For: 0.22.0 > > Attachments: patch-1618.txt > > > org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added > in MAPREDUCE-1307. They should have javadoc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1618) JobStatus.getJobAcls() and setJobAcls should have javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1618: --- Description: org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added in MAPREDUCE-1307. They should have javadoc. (was: org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added in MAPREDUCE-1307 should have javadoc.) > JobStatus.getJobAcls() and setJobAcls should have javadoc > - > > Key: MAPREDUCE-1618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1618 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Priority: Trivial > Fix For: 0.22.0 > > > org.apache.hadoop.mapreduce.JobStatus.getJobAcls() and setJobAcls are added > in MAPREDUCE-1307. They should have javadoc. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1489) DataDrivenDBInputFormat should not query the database when generating only one split
[ https://issues.apache.org/jira/browse/MAPREDUCE-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848605#action_12848605 ] Hadoop QA commented on MAPREDUCE-1489: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435756/MAPREDUCE-1489.patch against trunk revision 926449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/49/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/49/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/49/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/49/console This message is automatically generated. > DataDrivenDBInputFormat should not query the database when generating only > one split > > > Key: MAPREDUCE-1489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1489 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1489.patch > > > DataDrivenDBInputFormat runs a query to establish bounding values for each > split it generates; but if it's going to generate only one split > (mapreduce.job.maps == 1), then there's no reason to do this. This will > remove overhead associated with a single-threaded import of a non-indexed > table since it avoids a full table scan. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
