[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: (was: MAPREDUCE-5860.patch) > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0, trunk > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Fix For: trunk > > Attachments: MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: MAPREDUCE-5860.patch Updated patch to ensure the CombineContext remained valid during the combiner->close() method called during CombineRunner::close() processing. > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0, trunk > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Fix For: trunk > > Attachments: MAPREDUCE-5860.patch, MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
[ https://issues.apache.org/jira/browse/MAPREDUCE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5917: Fix Version/s: trunk Target Version/s: trunk Release Note: Add method to retrieve configuration key by index. Status: Patch Available (was: Open) > Be able to retrieve configuration keys by index > --- > > Key: MAPREDUCE-5917 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-5917.patch > > > The pipes C++ side does not have a configuration key/value pair iterator. It > is useful to be able to iterate through all of the configuration keys without > having to expose a C++ map iterator since that is specific to the JobConf > internals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
[ https://issues.apache.org/jira/browse/MAPREDUCE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5917: Attachment: (was: MAPREDUCE-5917.patch) > Be able to retrieve configuration keys by index > --- > > Key: MAPREDUCE-5917 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-5917.patch > > > The pipes C++ side does not have a configuration key/value pair iterator. It > is useful to be able to iterate through all of the configuration keys without > having to expose a C++ map iterator since that is specific to the JobConf > internals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
[ https://issues.apache.org/jira/browse/MAPREDUCE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5917: Attachment: MAPREDUCE-5917.patch Update patch to be trunk root based. > Be able to retrieve configuration keys by index > --- > > Key: MAPREDUCE-5917 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-5917.patch > > > The pipes C++ side does not have a configuration key/value pair iterator. It > is useful to be able to iterate through all of the configuration keys without > having to expose a C++ map iterator since that is specific to the JobConf > internals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5916: Fix Version/s: trunk Target Version/s: trunk Status: Patch Available (was: Open) > The authenticate response is not sent when password is empty (LocalJobRunner) > - > > Key: MAPREDUCE-5916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd > Fix For: trunk > > Attachments: MAPREDUCE-5916.patch > > > When running in a mode where there are no credentials associated with the > pipes submission and the password is empty, the C++ verifyDigestAndRespond() > does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5916: Attachment: (was: MAPREDUCE-5916.patch) > The authenticate response is not sent when password is empty (LocalJobRunner) > - > > Key: MAPREDUCE-5916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd > Attachments: MAPREDUCE-5916.patch > > > When running in a mode where there are no credentials associated with the > pipes submission and the password is empty, the C++ verifyDigestAndRespond() > does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5916: Attachment: MAPREDUCE-5916.patch Update patch to be trunk root based. > The authenticate response is not sent when password is empty (LocalJobRunner) > - > > Key: MAPREDUCE-5916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd > Attachments: MAPREDUCE-5916.patch > > > When running in a mode where there are no credentials associated with the > pipes submission and the password is empty, the C++ verifyDigestAndRespond() > does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5915) Pipes ping thread should sleep in intervals to allow for isDone() to be checked
[ https://issues.apache.org/jira/browse/MAPREDUCE-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5915: Attachment: (was: MAPREDUCE-5915.patch) > Pipes ping thread should sleep in intervals to allow for isDone() to be > checked > --- > > Key: MAPREDUCE-5915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5915 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-5915.patch > > > The ping() thread sleeps for 5 seconds at a time causing up to a 5 second > delay in testing if the job is finished. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5915) Pipes ping thread should sleep in intervals to allow for isDone() to be checked
[ https://issues.apache.org/jira/browse/MAPREDUCE-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5915: Fix Version/s: trunk Target Version/s: trunk Status: Patch Available (was: Open) > Pipes ping thread should sleep in intervals to allow for isDone() to be > checked > --- > > Key: MAPREDUCE-5915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5915 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-5915.patch > > > The ping() thread sleeps for 5 seconds at a time causing up to a 5 second > delay in testing if the job is finished. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5915) Pipes ping thread should sleep in intervals to allow for isDone() to be checked
[ https://issues.apache.org/jira/browse/MAPREDUCE-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5915: Attachment: MAPREDUCE-5915.patch Update patch to be trunk root based. > Pipes ping thread should sleep in intervals to allow for isDone() to be > checked > --- > > Key: MAPREDUCE-5915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5915 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Attachments: MAPREDUCE-5915.patch, MAPREDUCE-5915.patch > > > The ping() thread sleeps for 5 seconds at a time causing up to a 5 second > delay in testing if the job is finished. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: MAPREDUCE-5860.patch Update patch to be based off the root. > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0, trunk > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Fix For: trunk > > Attachments: MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: (was: HadoopPipes.cc.patch) > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0, trunk > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Fix For: trunk > > Attachments: MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: (was: MAPREDUCE-5860.patch) > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0, trunk > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Fix For: trunk > > Attachments: MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Fix Version/s: trunk Target Version/s: trunk Affects Version/s: trunk Status: Patch Available (was: Open) > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0, trunk > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Fix For: trunk > > Attachments: MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-3914: Fix Version/s: trunk Target Version/s: trunk Affects Version/s: trunk Status: Patch Available (was: Open) > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 1.0.0, 0.23.0, 0.20.205.0, trunk > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl > Fix For: trunk > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-3914: Attachment: (was: MAPREDUCE-3914.patch) > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0, trunk > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl > Fix For: trunk > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-3914: Attachment: MAPREDUCE-3914.patch Update patch to be relative to the root > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043295#comment-14043295 ] Joe Mudd commented on MAPREDUCE-1362: - The additional warning is due to the use of getLocalCacheFiles(). There didn't appear to be an alternative. > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: (was: MAPREDUCE-1362.patch) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: (was: MAPREDUCE-1362.patch) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: (was: MAPREDUCE-1362.patch) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: MAPREDUCE-1362.patch Address the mkdirs findbug issue > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: MAPREDUCE-1362.patch Repackage patch from branch root. > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch, MAPREDUCE-1362.patch, MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Status: Patch Available (was: Open) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch, MAPREDUCE-1362.patch, MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Status: Open (was: Patch Available) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch, MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Fix Version/s: trunk Target Version/s: trunk Affects Version/s: trunk Release Note: Update Hadoop Pipes to support MRv2 API Status: Patch Available (was: Open) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Bassam Tabbara > Fix For: trunk > > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch, MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: MAPREDUCE-1362.patch > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Reporter: Bassam Tabbara > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch, MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: (was: MAPREDUCE-1362.patch) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Reporter: Bassam Tabbara > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: MAPREDUCE-1362.patch Fixed typo where "outputFormat" was used to look up output format class instead of documented "writer" option. Patch is ready for code review. > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Reporter: Bassam Tabbara > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch, MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Priority: Major (was: Minor) > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0 > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Attachments: HadoopPipes.cc.patch, MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019845#comment-14019845 ] Joe Mudd commented on MAPREDUCE-5860: - Bumped up to major since this issue could cause rows to be lost or a crash due to close() cleaning up before all of the Combiner's reduce() calls. > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0 > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd > Attachments: HadoopPipes.cc.patch, MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
[ https://issues.apache.org/jira/browse/MAPREDUCE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5917: Attachment: MAPREDUCE-5917.patch Trunk patch that passes dev-support/test-patch and is ready for code review. The patch adds two methods to the JobConf() class: getCount() and getKey(size_t index). This avoids having to expose an implementation specific map::const_iterator. > Be able to retrieve configuration keys by index > --- > > Key: MAPREDUCE-5917 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Attachments: MAPREDUCE-5917.patch > > > The pipes C++ side does not have a configuration key/value pair iterator. It > is useful to be able to iterate through all of the configuration keys without > having to expose a C++ map iterator since that is specific to the JobConf > internals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
Joe Mudd created MAPREDUCE-5917: --- Summary: Be able to retrieve configuration keys by index Key: MAPREDUCE-5917 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 Project: Hadoop Map/Reduce Issue Type: New Feature Components: pipes Affects Versions: trunk Reporter: Joe Mudd Priority: Minor The pipes C++ side does not have a configuration key/value pair iterator. It is useful to be able to iterate through all of the configuration keys without having to expose a C++ map iterator since that is specific to the JobConf internals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5916: Attachment: MAPREDUCE-5916.patch Trunk patch that passes dev-support/test-patch and is ready for code review. The patch was verified after MAPREDUCE-1362 was applied which allows for LocalJobRunner() testing. > The authenticate response is not sent when password is empty (LocalJobRunner) > - > > Key: MAPREDUCE-5916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd > Attachments: MAPREDUCE-5916.patch > > > When running in a mode where there are no credentials associated with the > pipes submission and the password is empty, the C++ verifyDigestAndRespond() > does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
Joe Mudd created MAPREDUCE-5916: --- Summary: The authenticate response is not sent when password is empty (LocalJobRunner) Key: MAPREDUCE-5916 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: trunk Reporter: Joe Mudd When running in a mode where there are no credentials associated with the pipes submission and the password is empty, the C++ verifyDigestAndRespond() does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5915) Pipes ping thread should sleep in intervals to allow for isDone() to be checked
[ https://issues.apache.org/jira/browse/MAPREDUCE-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5915: Attachment: MAPREDUCE-5915.patch The trunk patch sleeps 5 times for a second and between each sleep the job status is tested. The provides a more timely exit of the ping() thread. The patch passes dev-support/test-patch and is ready for code review. > Pipes ping thread should sleep in intervals to allow for isDone() to be > checked > --- > > Key: MAPREDUCE-5915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5915 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Affects Versions: trunk >Reporter: Joe Mudd >Priority: Minor > Attachments: MAPREDUCE-5915.patch > > > The ping() thread sleeps for 5 seconds at a time causing up to a 5 second > delay in testing if the job is finished. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5915) Pipes ping thread should sleep in intervals to allow for isDone() to be checked
Joe Mudd created MAPREDUCE-5915: --- Summary: Pipes ping thread should sleep in intervals to allow for isDone() to be checked Key: MAPREDUCE-5915 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5915 Project: Hadoop Map/Reduce Issue Type: Improvement Components: pipes Affects Versions: trunk Reporter: Joe Mudd Priority: Minor The ping() thread sleeps for 5 seconds at a time causing up to a 5 second delay in testing if the job is finished. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-3914: Attachment: MAPREDUCE-3914.patch Trunk version of the patch that passes dev-support/test-patch and is ready for code review. > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: MAPREDUCE-5860.patch A properly named and dev-support/test-patch tested patch. This patch is ready for code review. > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0 > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd >Priority: Minor > Attachments: HadoopPipes.cc.patch, MAPREDUCE-5860.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019168#comment-14019168 ] Joe Mudd commented on MAPREDUCE-1362: - I've rebuilt the patch against the latest trunk. The latest MAPREDUCE-1362.patch is ready for a code review. > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Reporter: Bassam Tabbara > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-1362: Attachment: MAPREDUCE-1362.patch Resync'd trunk patch > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Reporter: Bassam Tabbara > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Mudd updated MAPREDUCE-5860: Attachment: HadoopPipes.cc.patch A patch that works for me. > Hadoop pipes Combiner is closed before all of its reduce calls > -- > > Key: MAPREDUCE-5860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.23.0 > Environment: 0.23.0 on 64 bit linux >Reporter: Joe Mudd >Priority: Minor > Attachments: HadoopPipes.cc.patch > > > When a Combiner is specified to runTask() its reduce() method may be called > after its close() method has been called due to how the Combiner's containing > object, CombineRunner, is closed after the TaskContextImpl's reducer member > is closed (see TaskContextImpl::closeAll()). > I believe the fix is to delegate the Combiner's ownership to CombineRunner, > making it responsible for calling the Combiner's close() method and deleting > the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
Joe Mudd created MAPREDUCE-5860: --- Summary: Hadoop pipes Combiner is closed before all of its reduce calls Key: MAPREDUCE-5860 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.23.0 Environment: 0.23.0 on 64 bit linux Reporter: Joe Mudd Priority: Minor When a Combiner is specified to runTask() its reduce() method may be called after its close() method has been called due to how the Combiner's containing object, CombineRunner, is closed after the TaskContextImpl's reducer member is closed (see TaskContextImpl::closeAll()). I believe the fix is to delegate the Combiner's ownership to CombineRunner, making it responsible for calling the Combiner's close() method and deleting the Combiner instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
