[
https://issues.apache.org/jira/browse/DISPATCH-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17341267#comment-17341267
]
Jiri Daněk commented on DISPATCH-2039:
--------------------------------------
The poisoning has been merged. There are still the outstanding ideas for
improving the basic poisoning with redzones and quarantine, which haven't been
implemented.
The idea of being able to directly replace pool with malloc/free suffers from
the little problem of DISPATCH-2060. There are two possibilities. First, be
able to turn off the pool and use malloc/free, which is a lot of work, but has
the added advantage that off-the-shelf allocators could then be used (tcmalloc,
jemalloc, mimalloc; from Google, Facebook, Microsoft, respectively; each has
means of being queried for allocation stats). Second, set pool free_list max
size to 0. This is easy, but it must be done sensitively and turned on only for
tests.
https://travis-ci.com/github/apache/qpid-dispatch/jobs/503899597#L7856
{noformat}
62: test_999_check_for_leaks (system_tests_multicast.MulticastLinearTest) ...
OOPS!!! INT.A: (qd_message_t) - held=2 max=0
62: {u'qdr_delivery_t': {u'heldByThreads': int32(0), u'typeSize':
int32(312), u'transferBatchSize': int32(1), u'globalFreeListMax': int32(0),
u'batchesRebalancedToGlobal': int32(2695), u'typeName': u'qdr_delivery_t',
u'batchesRebalancedToThreads': int32(0), u'totalFreeToHeap': int32(2695),
u'totalAllocFromHeap': int32(2695), u'localFreeListMax': int32(0), u'type':
u'org.apache.qpid.dispatch.allocator', u'identity':
u'allocator/qdr_delivery_t', u'name': u'allocator/qdr_delivery_t'},
u'qd_buffer_t': {u'heldByThreads': int32(7), u'typeSize': int32(536),
u'transferBatchSize': int32(1), u'globalFreeListMax': int32(0),
u'batchesRebalancedToGlobal': int32(30731), u'typeName': u'qd_buffer_t',
u'batchesRebalancedToThreads': int32(0), u'totalFreeToHeap': int32(30731),
u'totalAllocFromHeap': int32(30738), u'localFreeListMax': int32(0), u'type':
u'org.apache.qpid.dispatch.allocator', u'identity': u'allocator/qd_buffer_t',
u'name': u'allocator/qd_buffer_t'}, u'qd_message_t': {u'heldByThreads':
int32(2), u'typeSize': int32(200), u'transferBatchSize': int32(1),
u'globalFreeListMax': int32(0), u'batchesRebalancedToGlobal': int32(4472),
u'typeName': u'qd_message_t', u'batchesRebalancedToThreads': int32(0),
u'totalFreeToHeap': int32(4472), u'totalAllocFromHeap': int32(4474),
u'localFreeListMax': int32(0), u'type': u'org.apache.qpid.dispatch.allocator',
u'identity': u'allocator/qd_message_t', u'name': u'allocator/qd_message_t'}}
{noformat}
{noformat}
62: ======================================================================
62: FAIL: test_999_check_for_leaks (system_tests_multicast.MulticastLinearTest)
62: ----------------------------------------------------------------------
62: Traceback (most recent call last):
62: File
"/home/travis/build/apache/qpid-dispatch/tests/system_tests_multicast.py", line
437, in test_999_check_for_leaks
62: self._check_for_leaks()
62: File
"/home/travis/build/apache/qpid-dispatch/tests/system_tests_multicast.py", line
214, in _check_for_leaks
62: self.assertFalse(held >= (2 * max_allowed))
62: AssertionError: True is not false
62:
62: ----------------------------------------------------------------------
62: Ran 16 tests in 67.300s
62:
62: FAILED (failures=1)
62/72 Test #62: system_tests_multicast ............................***Failed
67.38 sec
{noformat}
https://travis-ci.com/github/apache/qpid-dispatch/jobs/503899597#L2285
{noformat}
9: Test Case alloc_tests.test_alloc_basic: FAIL: Incorrect alloc-from-heap
9: Test Case alloc_tests.test_safe_references: PASS
9: Test Case policy_tests.test_link_name_lookup: PASS
9: Test Case policy_tests.test_link_name_tree_lookup: PASS
9: Test Case policy_tests.test_link_name_csv_parser: PASS
9: Test Case failover_tests.test_failover_list_empty: PASS
9: Test Case failover_tests.test_failover_list_single: PASS
9: Test Case failover_tests.test_failover_list_multiple: PASS
9: Test Case parse_tree_tests.test_add_remove: PASS
9: Test Case parse_tree_tests.test_add_and_match_str: PASS
9: Test Case parse_tree_tests.test_duplicate_error_str: PASS
9: Test Case parse_tree_tests.test_normalization: PASS
9: Test Case parse_tree_tests.test_matches: PASS
9: Test Case parse_tree_tests.test_multiple_matches: PASS
9: Test Case parse_tree_tests.test_validation: PASS
9: Test Case proton_utils_tests.test_data_as_string: PASS
9: Test Case core_timer_tests.test_core_timer: PASS
9: Test Case hash_tests.test_iter_keys: PASS
9: Test Case hash_tests.test_str_keys: PASS
9: Test Case hash_tests.test_iter_bad: PASS
9: Test Case hash_tests.test_str_bad: PASS
9: Test Case thread_tests.test_thread_id: PASS
9: Test Case thread_tests.test_condition: PASS
9: -----------------------------------------------------
9: Suppressions used:
9: count bytes template
9: 1 56 ^IoAdapter_init$
9: 37 120 _ctypes_alloc_format_string
9: 4962 3497107 /libpython2.*.so
9: -----------------------------------------------------
9:
9/72 Test #9: unit_tests ........................................***Failed
1.28 sec
{noformat}
https://travis-ci.com/github/apache/qpid-dispatch/jobs/503899598#L4094
{noformat}
34: Process 13827 error: exit code -6, expected 0
34: qdrouterd -c ssl-test-router.conf -I
/home/travis/build/apache/qpid-dispatch/python
34:
/home/travis/build/apache/qpid-dispatch/build/tests/system_test.dir/system_tests_user_id_proxy/QdSSLUseridProxy/setUpClass/ssl-test-router-1.cmd
34: >>>>
34: 2021-05-08 07:33:18.294584 +0000 AGENT (warning) Attribute
'displayNameFile' of entity 'sslProfile' has been deprecated. Use
'uidNameMappingFile' instead
34: qdrouterd: /home/travis/build/apache/qpid-dispatch/src/alloc_pool.c:527:
qd_alloc_sequence: Assertion `item->header == PATTERN_FRONT' failed.
34: <<<<
34:
34: ----------------------------------------------------------------------
34: Ran 2 tests in 0.566s
34:
34: FAILED (errors=1)
34/73 Test #34: system_tests_user_id_proxy ........................***Failed
0.63 sec
{noformat}
> Memory pool should be manually poisoned so that ASAN works with it
> ------------------------------------------------------------------
>
> Key: DISPATCH-2039
> URL: https://issues.apache.org/jira/browse/DISPATCH-2039
> Project: Qpid Dispatch
> Issue Type: Wish
> Affects Versions: 1.15.0
> Reporter: Jiri Daněk
> Assignee: Jiri Daněk
> Priority: Minor
> Fix For: 1.17.0
>
>
> From https://github.com/google/sanitizers/wiki/AddressSanitizerManualPoisoning
> bq. A user may poison/unpoison a region of memory manually. Use this feature
> with caution. In many cases good old malloc+free is a better way to find heap
> bugs than using custom allocators with manual poisoning.
> As far as I can tell, it is nowadays not possible to turn off the pool
> allocation and use malloc/free, because the pool mechanism also implements
> the weak pointers and ref counters. That means giving hints to ASAN is the
> only way to discover memory bugs of the type (if what Chuck speculated is
> true) of DISPATCH-2032.
> bq. If you have a custom allocation arena, the typical workflow would be to
> poison the entire arena first, and then unpoison allocated chunks of memory
> leaving poisoned redzones between them. The allocated chunks should start
> with 8-aligned addresses.
> Alternatively, the current memory debugging machinery for the pool could take
> care of it on its own... but using ASAN seems sensible to me.
> http://blog.hostilefork.com/poison-memory-without-asan/
> h3. Nice to have extra features (which won't be implemented at first)
> * redzones, there should be chunks of poison on either end of a returned
> memory, to detect invalid accesses out of bounds; this means deliberate waste
> of memory (I am thinking 3x increase, to make implementation easy)
> * quarantine, returned chunks should be kept in the pool for some time before
> they are returned as new allocations, to catch use-after-free; this policy
> goes against performance considerations
> h3. Open issues
> Is it necessary to lock around the poison macros? I did not understand the
> thread safety note in API comment fully.
> h3. One thought
> Actually, setting a limit on free_list length == 0 would effectively disable
> pool and turn the calls into simple wrappers over malloc/free. It would be
> enough to make this configurable at build time. Then asan should work just
> fine without poison.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
