[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132761#comment-14132761 ] Hudson commented on MAPREDUCE-6005: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/]) MAPREDUCE-6005. native-task: Fix some valgrind errors (Binglin Chang) (bchang: rev 83a396733ea43564beee928f41041a341e26db05) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestByteBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucketIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCounter.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadWriteBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestComparatorForDualPivotQuickSort.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestComparatorForStdSort.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestFileSystem.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestPartitionBucket.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/StringUtil.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestMain.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeObjectFactory.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/StringUtil.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestMemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/GzipCodec.cc * hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCommand.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestMemBlockIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/util/TestStringUtil.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestFixSizeContainer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeObjectFactory.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestKVBuffer.cc > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132722#comment-14132722 ] Hudson commented on MAPREDUCE-6005: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/]) MAPREDUCE-6005. native-task: Fix some valgrind errors (Binglin Chang) (bchang: rev 83a396733ea43564beee928f41041a341e26db05) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCommand.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestMain.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestByteBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestComparatorForStdSort.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestFileSystem.cc * hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/StringUtil.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestMemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/util/TestStringUtil.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeObjectFactory.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestMemBlockIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucketIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCounter.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestFixSizeContainer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadWriteBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/StringUtil.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestPartitionBucket.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/GzipCodec.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestComparatorForDualPivotQuickSort.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeObjectFactory.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestKVBuffer.cc > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132664#comment-14132664 ] Hudson commented on MAPREDUCE-6005: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/679/]) MAPREDUCE-6005. native-task: Fix some valgrind errors (Binglin Chang) (bchang: rev 83a396733ea43564beee928f41041a341e26db05) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestComparatorForStdSort.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucketIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/GzipCodec.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestMemBlockIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestReadWriteBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/StringUtil.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCommand.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestIterator.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestComparatorForDualPivotQuickSort.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestPartitionBucket.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestCounter.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/StringUtil.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeObjectFactory.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestFixSizeContainer.cc * hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestMain.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestByteBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestMemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/lib/TestKVBuffer.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestFileSystem.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/NativeObjectFactory.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/util/TestStringUtil.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.h > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084351#comment-14084351 ] Binglin Chang commented on MAPREDUCE-6005: -- I have committed this, thanks Sean. > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080651#comment-14080651 ] Sean Zhong commented on MAPREDUCE-6005: --- Thanks. +1 > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080567#comment-14080567
]
Sean Zhong commented on MAPREDUCE-6005:
---
One more:
Can you also fix
{quote}
string StringUtil::ToString(int32_t v) {
char tmp[32];
snprintf(tmp, 32, "%d", v);
return tmp;
}
string StringUtil::ToString(uint32_t v) {
char tmp[32];
snprintf(tmp, 32, "%u", v);
return tmp;
}
string StringUtil::ToString(int64_t v) {
char tmp[32];
snprintf(tmp, 32, "%lld", (long long int)v);
return tmp;
}
string StringUtil::ToString(int64_t v, char pad, int64_t len) {
char tmp[32];
snprintf(tmp, 32, "%%%c%lldlld", pad, len);
return Format(tmp, v);
}
string StringUtil::ToString(uint64_t v) {
char tmp[32];
snprintf(tmp, 32, "%llu", (long long unsigned int)v);
return tmp;
}
string StringUtil::ToString(bool v) {
if (v) {
return "true";
} else {
return "false";
}
}
string StringUtil::ToString(float v) {
char tmp[32];
snprintf(tmp, 32, "%f", v);
return tmp;
}
string StringUtil::ToString(double v) {
char tmp[32];
snprintf(tmp, 32, "%lf", v);
return tmp;
}
{quote}
1) it is not safe to convert a char array to a string like this. It will
trigger a copy contructor. But by
http://www.cplusplus.com/reference/string/string/string/,
{quote}
string (const char* s);
the string need to be null terminated.
{quote}
2) snprintf(tmp, 32, "%lf", v) impl is platform dependant when size “32”
equals the v length. It may truncate the raw data, or may ignore the null
terminitor. http://linux.die.net/man/3/snprintf,
{quote}
The functions snprintf() and vsnprintf() write at most size bytes (including
the terminating null byte ('\0')) to str.
{quote}
> native-task: fix some valgrind errors
> --
>
> Key: MAPREDUCE-6005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch,
> MAPREDUCE-6005.v3.patch
>
>
> Running test with valgrind shows there are some bugs, this jira try to fix
> them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080546#comment-14080546
]
Sean Zhong commented on MAPREDUCE-6005:
---
{quote}
In merger, all MergeEntryPtr is owned by Merger::_entries, and is deleted in
~Merger at end, so it doesn't require additional care.
{quote}
you are right, +1.
> native-task: fix some valgrind errors
> --
>
> Key: MAPREDUCE-6005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch,
> MAPREDUCE-6005.v3.patch
>
>
> Running test with valgrind shows there are some bugs, this jira try to fix
> them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080524#comment-14080524 ] Binglin Chang commented on MAPREDUCE-6005: -- Hi Sean, bq. About the leak, at src / main / native / src / lib / Merge.cc, there is a similar mem leak, you only fixed the leak in PartitionBucketIterator.cc In merger, all MergeEntryPtr is owned by Merger::_entries, and is deleted in ~Merger at end, so it doesn't require additional care. bq. maybe better use snsprintf(buf_ptr, "%02X", ...) Sure, I will update the patch > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080500#comment-14080500
]
Sean Zhong commented on MAPREDUCE-6005:
---
Hi Binling,
About the leak, at src / main / native / src / lib / Merge.cc, there is a
similar mem leak, you only fixed the leak in PartitionBucketIterator.cc in
patch
https://issues.apache.org/jira/secure/attachment/12658416/MAPREDUCE-6005.v2.patch
About toHexString, the name is good. However, maybe better use
snsprintf(buf_ptr, "%02X", ...)
for (i = 0; i < size; i++)
{
buf_ptr += snsprintf(buf_ptr, "%02X", buf[i]);
}
> native-task: fix some valgrind errors
> --
>
> Key: MAPREDUCE-6005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch
>
>
> Running test with valgrind shows there are some bugs, this jira try to fix
> them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080476#comment-14080476 ] Binglin Chang commented on MAPREDUCE-6005: -- Thanks for the review Sean. bq. I believe we can rename StringUtil::ToString(const void * v, uint32_t len) to StringUtil::md5BinaryToString(const void * v, uint32_t len), and also make ToHex(uint8_t v) private or inlined to md5BinaryToString. I make the function name from ToString to ToHexString, hex means hex format binary string, which is md5's format, this format is not restricted to md5 only, so I think ToHexString is more suitable. About ToHex, using static prefix makes this method private, thats why I add static. bq. There is another leak at Merge What do you mean? I already add delete in the code you specified. Do we need to adding comments only? > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080460#comment-14080460
]
Sean Zhong commented on MAPREDUCE-6005:
---
1. About toHex
{quote}
-string StringUtil::ToString(const void * v, uint32_t len) {
+static char ToHex(uint8_t v) {
+ return v < 10 ? (v + '0') : (v - 10 + 'a');
+}
{quote}
It is not safe to not doing range check for a public function, besides the
correct implementation which convert binary to string should use base64
encoding. Since StringUtil::ToString(const void * v, uint32_t len) is only
used for md5 conversion,
{quote}
case MD5HashType:
dest.append(StringUtil::ToString(data, length));
{quote}
I believe we can rename StringUtil::ToString(const void * v, uint32_t len) to
StringUtil::md5BinaryToString(const void * v, uint32_t len), and also make
ToHex(uint8_t v) private or inlined to md5BinaryToString.
2. memmov replace memcpy is good, thanks
3. About
{quote}
} else { // no more, pop heap
+delete _heap[0];
{quote}
There is another leak at Merge
{quote}
MergeEntryPtr * base = &(_heap[0]);
popHeap(base, base + cur_heap_size, _comparator);
_heap.pop_back();
{quote}
And, I suggest we can add a comments in source about why we delete _heap[0]
Others looks good, +1
> native-task: fix some valgrind errors
> --
>
> Key: MAPREDUCE-6005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch
>
>
> Running test with valgrind shows there are some bugs, this jira try to fix
> them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077684#comment-14077684 ] Binglin Chang commented on MAPREDUCE-6005: -- update patch, changes: 1. fix more valgrind errors in nttest, run nttest now only show errors in libz and libjvm 2. remove some excessive log in test 3. remove java filesystem and related factory methods (getDefaultFsUri, FileSystem::get) 4. FileInputStream/FileOutputStream use open instead of fopen, this is mostly an optimization. 5. add unittest for CounterSet 6. add unittest for ToHexString 7. fix memory leak in PartitionBucket > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075879#comment-14075879 ] Manu Zhang commented on MAPREDUCE-6005: --- Hi Binglin, could you explain a bit about "2. use open instead of fopen in FileInputStream/FileOutputStream" ? I've also run nttest with valgrind and there is a bunch of "blocks still reachable or indirectly lost" in the outputs. Is there a serious bug in fopen ? > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[
https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075715#comment-14075715
]
Todd Lipcon commented on MAPREDUCE-6005:
{code}
+ } else {
+return RawFileSystemInstance;
{code}
This doesn't seem quite right. How about we THROW instead?
Can you add a test for the bug fix in ToString, and the fix in CounterSet?
> native-task: fix some valgrind errors
> --
>
> Key: MAPREDUCE-6005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6005.v1.patch
>
>
> Running test with valgrind shows there are some bugs, this jira try to fix
> them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
