[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10188: --- Resolution: Fixed Status: Resolved (was: Patch Available) > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch, > HDFS-10188.HDFS-8707.003.patch, HDFS-10188.HDFS-8707.004.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[
https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10188:
---
Attachment: HDFS-10188.HDFS-8707.004.patch
New patch looks good, I went to fix a typo where it was doing something like
{code}
memset(ptr, 0, sizeof(ptr))
{code}
and tried to change to
{code}
memset(ptr, 0, sizeof(decltype(this)))
{code}
I forgot that member new/delete needed to be static, and because of that they
don't get to see 'this', so sizeof(decltype(this)) doesn't compile. Sorry for
the bad advice there.
The quick fix was to pass in the class name as a macro argument; not ideal but
not terrible. The API would look like
{code}
class UsedAfterFreeSometimes {
MEMCHECKED_CLASS(UsedAfterFreeSometimes)
}
{code}
Unfortunately it looks like getting an implicit typename isn't really possible
in static methods short of even more macro tricks.
Xiaowei, could you take a look at these changes and see if they are still in
line with what you did?
> libhdfs++: Implement debug allocators
> -
>
> Key: HDFS-10188
> URL: https://issues.apache.org/jira/browse/HDFS-10188
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Xiaowei Zhu
> Attachments: HDFS-10188.HDFS-8707.000.patch,
> HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch,
> HDFS-10188.HDFS-8707.003.patch, HDFS-10188.HDFS-8707.004.patch
>
>
> I propose implementing a set of memory new/delete pairs with additional
> checking to detect double deletes, read-after-delete, and write-after-deletes
> to help debug resource ownership issues and prevent new ones from entering
> the library.
> One of the most common issues we have is use-after-free issues. The
> continuation pattern makes these really tricky to debug because by the time a
> segsegv is raised the context of what has caused the error is long gone.
> The plan is to add allocators that can be turned on that can do the
> following, in order of runtime cost.
> 1: no-op, forward through to default new/delete
> 2: make sure the memory given to the constructor is dirty, memset free'd
> memory to 0
> 3: implement operator new with mmap, lock that region of memory once it's
> been deleted; obviously this can't be left to run forever because the memory
> is never unmapped
> This should also put some groundwork in place for implementing specialized
> allocators for tiny objects that we churn through like std::string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: HDFS-10188.HDFS-8707.003.patch > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch, > HDFS-10188.HDFS-8707.003.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: (was: HDFS-10188.HDFS-8707.003.patch) > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch, > HDFS-10188.HDFS-8707.003.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: HDFS-10188.HDFS-8707.003.patch New patch using decltype as suggested by James. For new[] and delete, keep the mem_struct usage. > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch, > HDFS-10188.HDFS-8707.003.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: (was: HDFS-10188.HDFS-8707.002.patch) > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: HDFS-10188.HDFS-8707.002.patch > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: HDFS-10188.HDFS-8707.002.patch > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: HDFS-10188.HDFS-8707.001.patch > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch, > HDFS-10188.HDFS-8707.001.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Attachment: HDFS-10188.HDFS-8707.000.patch > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-10188.HDFS-8707.000.patch > > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-10188: --- Status: Patch Available (was: Open) First patch overrides new/delete globally w/ memset before free in delete call. > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators
[ https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10188: --- Description: I propose implementing a set of memory new/delete pairs with additional checking to detect double deletes, read-after-delete, and write-after-deletes to help debug resource ownership issues and prevent new ones from entering the library. One of the most common issues we have is use-after-free issues. The continuation pattern makes these really tricky to debug because by the time a segsegv is raised the context of what has caused the error is long gone. The plan is to add allocators that can be turned on that can do the following, in order of runtime cost. 1: no-op, forward through to default new/delete 2: make sure the memory given to the constructor is dirty, memset free'd memory to 0 3: implement operator new with mmap, lock that region of memory once it's been deleted; obviously this can't be left to run forever because the memory is never unmapped This should also put some groundwork in place for implementing specialized allocators for tiny objects that we churn through like std::string. was: I propose implementing a set of memory new/delete pairs with additional checking to detect double deletes, read-after-delete, and write-after-deletes to help debug resource ownership issues and prevent new ones from entering the library. One of the most common issues we have is use-after-free issues. The continuation pattern makes these really tricky to debug because by the time a segsegv is raised the context of what has caused the error is long gone. The plan is to add allocators that can be turned on that can do the following, in order of runtime cost. 1: no-op, forward through to default new/delete 2: memset free'd memory to 0 3: implement operator new with mmap, lock that region of memory once it's been deleted; obviously this can't be left to run forever because the memory is never unmapped This should also put some groundwork in place for implementing specialized allocators for tiny objects that we churn through like std::string. > libhdfs++: Implement debug allocators > - > > Key: HDFS-10188 > URL: https://issues.apache.org/jira/browse/HDFS-10188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > > I propose implementing a set of memory new/delete pairs with additional > checking to detect double deletes, read-after-delete, and write-after-deletes > to help debug resource ownership issues and prevent new ones from entering > the library. > One of the most common issues we have is use-after-free issues. The > continuation pattern makes these really tricky to debug because by the time a > segsegv is raised the context of what has caused the error is long gone. > The plan is to add allocators that can be turned on that can do the > following, in order of runtime cost. > 1: no-op, forward through to default new/delete > 2: make sure the memory given to the constructor is dirty, memset free'd > memory to 0 > 3: implement operator new with mmap, lock that region of memory once it's > been deleted; obviously this can't be left to run forever because the memory > is never unmapped > This should also put some groundwork in place for implementing specialized > allocators for tiny objects that we churn through like std::string. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
