[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: MAPREDUCE-5860.patch A properly named and dev-support/test-patch tested patch. This patch is ready for code review. Hadoop pipes Combiner is closed before all of its reduce calls -- Key: MAPREDUCE-5860 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.23.0 Environment: 0.23.0 on 64 bit linux Reporter: Joe Mudd Priority: Minor Attachments: HadoopPipes.cc.patch, MAPREDUCE-5860.patch When a Combiner is specified to runTask() its reduce() method may be called after its close() method has been called due to how the Combiner's containing object, CombineRunner, is closed after the TaskContextImpl's reducer member is closed (see TaskContextImpl::closeAll()). I believe the fix is to delegate the Combiner's ownership to CombineRunner, making it responsible for calling the Combiner's close() method and deleting the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-3914: Attachment: MAPREDUCE-3914.patch Trunk version of the patch that passes dev-support/test-patch and is ready for code review. Mismatched free() / delete / delete [] in HadoopPipes - Key: MAPREDUCE-3914 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 Reporter: Charles Earl Attachments: MAPREDUCE-3914-branch-0.23.patch, MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch Original Estimate: 1h Remaining Estimate: 1h When running valgrind on a simple MapReduce pipes job, valgrind identifies a mismatched new / delete: ==20394== Mismatched free() / delete / delete [] ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1171) ==20394==by 0x424C33: main (ProcessRow.cpp:118) ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd ==20394==at 0x4C2864B: operator new[](unsigned long) (vg_replace_malloc.c:305) ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1121) ==20394==by 0x424C33: main (ProcessRow.cpp:118) ==20394== ==20394== Mismatched free() / delete / delete [] ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1172) ==20394==by 0x424C33: main (ProcessRow.cpp:118) ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd ==20394==at 0x4C2864B: operator new[](unsigned long) (vg_replace_malloc.c:305) ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1122) ==20394==by 0x424C33: main (ProcessRow.cpp:118) The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: bufin = new char[bufsize]; bufout = new char[bufsize]; should have matching delete [] calls but are instead bracketed my delete on lines 1171 and 1172: delete bufin; delete bufout; So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5915) Pipes ping thread should sleep in intervals to allow for isDone() to be checked
Joe Mudd created MAPREDUCE-5915: --- Summary: Pipes ping thread should sleep in intervals to allow for isDone() to be checked Key: MAPREDUCE-5915 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5915 Project: Hadoop Map/Reduce Issue Type: Improvement Components: pipes Affects Versions: trunk Reporter: Joe Mudd Priority: Minor The ping() thread sleeps for 5 seconds at a time causing up to a 5 second delay in testing if the job is finished. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5915) Pipes ping thread should sleep in intervals to allow for isDone() to be checked
[ https://issues.apache.org/jira/browse/MAPREDUCE-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5915: Attachment: MAPREDUCE-5915.patch The trunk patch sleeps 5 times for a second and between each sleep the job status is tested. The provides a more timely exit of the ping() thread. The patch passes dev-support/test-patch and is ready for code review. Pipes ping thread should sleep in intervals to allow for isDone() to be checked --- Key: MAPREDUCE-5915 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5915 Project: Hadoop Map/Reduce Issue Type: Improvement Components: pipes Affects Versions: trunk Reporter: Joe Mudd Priority: Minor Attachments: MAPREDUCE-5915.patch The ping() thread sleeps for 5 seconds at a time causing up to a 5 second delay in testing if the job is finished. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
Joe Mudd created MAPREDUCE-5916: --- Summary: The authenticate response is not sent when password is empty (LocalJobRunner) Key: MAPREDUCE-5916 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: trunk Reporter: Joe Mudd When running in a mode where there are no credentials associated with the pipes submission and the password is empty, the C++ verifyDigestAndRespond() does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5916: Attachment: MAPREDUCE-5916.patch Trunk patch that passes dev-support/test-patch and is ready for code review. The patch was verified after MAPREDUCE-1362 was applied which allows for LocalJobRunner() testing. The authenticate response is not sent when password is empty (LocalJobRunner) - Key: MAPREDUCE-5916 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: trunk Reporter: Joe Mudd Attachments: MAPREDUCE-5916.patch When running in a mode where there are no credentials associated with the pipes submission and the password is empty, the C++ verifyDigestAndRespond() does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
Joe Mudd created MAPREDUCE-5917: --- Summary: Be able to retrieve configuration keys by index Key: MAPREDUCE-5917 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 Project: Hadoop Map/Reduce Issue Type: New Feature Components: pipes Affects Versions: trunk Reporter: Joe Mudd Priority: Minor The pipes C++ side does not have a configuration key/value pair iterator. It is useful to be able to iterate through all of the configuration keys without having to expose a C++ map iterator since that is specific to the JobConf internals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
[ https://issues.apache.org/jira/browse/MAPREDUCE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5917: Attachment: MAPREDUCE-5917.patch Trunk patch that passes dev-support/test-patch and is ready for code review. The patch adds two methods to the JobConf() class: getCount() and getKey(size_t index). This avoids having to expose an implementation specific mapstring, string::const_iterator. Be able to retrieve configuration keys by index --- Key: MAPREDUCE-5917 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 Project: Hadoop Map/Reduce Issue Type: New Feature Components: pipes Affects Versions: trunk Reporter: Joe Mudd Priority: Minor Attachments: MAPREDUCE-5917.patch The pipes C++ side does not have a configuration key/value pair iterator. It is useful to be able to iterate through all of the configuration keys without having to expose a C++ map iterator since that is specific to the JobConf internals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019845#comment-14019845 ] Joe Mudd commented on MAPREDUCE-5860: - Bumped up to major since this issue could cause rows to be lost or a crash due to close() cleaning up before all of the Combiner's reduce() calls. Hadoop pipes Combiner is closed before all of its reduce calls -- Key: MAPREDUCE-5860 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.23.0 Environment: 0.23.0 on 64 bit linux Reporter: Joe Mudd Attachments: HadoopPipes.cc.patch, MAPREDUCE-5860.patch When a Combiner is specified to runTask() its reduce() method may be called after its close() method has been called due to how the Combiner's containing object, CombineRunner, is closed after the TaskContextImpl's reducer member is closed (see TaskContextImpl::closeAll()). I believe the fix is to delegate the Combiner's ownership to CombineRunner, making it responsible for calling the Combiner's close() method and deleting the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Priority: Major (was: Minor) Hadoop pipes Combiner is closed before all of its reduce calls -- Key: MAPREDUCE-5860 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.23.0 Environment: 0.23.0 on 64 bit linux Reporter: Joe Mudd Attachments: HadoopPipes.cc.patch, MAPREDUCE-5860.patch When a Combiner is specified to runTask() its reduce() method may be called after its close() method has been called due to how the Combiner's containing object, CombineRunner, is closed after the TaskContextImpl's reducer member is closed (see TaskContextImpl::closeAll()). I believe the fix is to delegate the Combiner's ownership to CombineRunner, making it responsible for calling the Combiner's close() method and deleting the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5898) distcp to support preserving HDFS extended attributes(XAttrs)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated MAPREDUCE-5898: --- Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have just committed this to trunk! distcp to support preserving HDFS extended attributes(XAttrs) - Key: MAPREDUCE-5898 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5898 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Yi Liu Fix For: 3.0.0 Attachments: MAPREDUCE-5898.1.patch, MAPREDUCE-5898.patch This JIRA to track the distcp support to handle the Xattrs with preserving options. Add new command line argument to support that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5898) distcp to support preserving HDFS extended attributes(XAttrs)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019902#comment-14019902 ] Hudson commented on MAPREDUCE-5898: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5654 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5654/]) MAPREDUCE-5898. distcp to support preserving HDFS extended attributes(XAttrs). Contributed by Yi Liu. (umamahesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1600900) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListingFileStatus.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpWithXAttrs.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java distcp to support preserving HDFS extended attributes(XAttrs) - Key: MAPREDUCE-5898 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5898 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Yi Liu Fix For: 3.0.0 Attachments: MAPREDUCE-5898.1.patch, MAPREDUCE-5898.patch This JIRA to track the distcp support to handle the Xattrs with preserving options. Add new command line argument to support that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5777) Support utf-8 text with BOM (byte order marker)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020177#comment-14020177 ] Karthik Kambatla commented on MAPREDUCE-5777: - Thanks Zhihai. If everything is duplicated, I think it makes sense to handle all of them together in a different JIRA. The patch looks good to me. +1. Committing shortly. Support utf-8 text with BOM (byte order marker) --- Key: MAPREDUCE-5777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5777 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0, 2.2.0 Reporter: bc Wong Assignee: zhihai xu Attachments: MAPREDUCE-5777.000.patch, MAPREDUCE-5777.001.patch, MAPREDUCE-5777.002.patch, MAPREDUCE-5777.003.patch, MAPREDUCE-5777.004.patch UTF-8 text may have a BOM. TextInputFormat, KeyValueTextInputFormat and friends should recognize the BOM and not treat it as actual data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5777) Support utf-8 text with BOM (byte order marker)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5777: Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Zhihai. Just committed this to trunk and branch-2. Support utf-8 text with BOM (byte order marker) --- Key: MAPREDUCE-5777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5777 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0, 2.2.0 Reporter: bc Wong Assignee: zhihai xu Fix For: 2.5.0 Attachments: MAPREDUCE-5777.000.patch, MAPREDUCE-5777.001.patch, MAPREDUCE-5777.002.patch, MAPREDUCE-5777.003.patch, MAPREDUCE-5777.004.patch UTF-8 text may have a BOM. TextInputFormat, KeyValueTextInputFormat and friends should recognize the BOM and not treat it as actual data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5777) Support utf-8 text with BOM (byte order marker)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020217#comment-14020217 ] Hudson commented on MAPREDUCE-5777: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5657 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5657/]) MAPREDUCE-5777. Support utf-8 text with Byte Order Marker. (Zhihai Xu via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1600977) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LineRecordReader.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/LineRecordReader.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/testBOM.txt Support utf-8 text with BOM (byte order marker) --- Key: MAPREDUCE-5777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5777 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0, 2.2.0 Reporter: bc Wong Assignee: zhihai xu Fix For: 2.5.0 Attachments: MAPREDUCE-5777.000.patch, MAPREDUCE-5777.001.patch, MAPREDUCE-5777.002.patch, MAPREDUCE-5777.003.patch, MAPREDUCE-5777.004.patch UTF-8 text may have a BOM. TextInputFormat, KeyValueTextInputFormat and friends should recognize the BOM and not treat it as actual data. -- This message was sent by Atlassian JIRA (v6.2#6252)
