[
https://issues.apache.org/jira/browse/DISPATCH-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gordon Sim closed DISPATCH-1504.
--------------------------------
Resolution: Fixed
> CLONE - pn_session_free is inefficient if handling large numbers of links
> -------------------------------------------------------------------------
>
> Key: DISPATCH-1504
> URL: https://issues.apache.org/jira/browse/DISPATCH-1504
> Project: Qpid Dispatch
> Issue Type: Improvement
> Reporter: Gordon Sim
> Priority: Major
>
> The links for a session are kept in a proton list object, which is
> essentially an array.
> Removing a link from this list involves first searching for the link by
> iterating over the array comparing it with the desired object using the
> comparator defined for the class, then cleaning up that element and shuffling
> up any remaining elements.
> In the case of pn_session_free, it loops calling on_link_free on the first
> element in the list (which involves among other things removing it from the
> list as described above) until there are no more elements in the list.
> Looking up the link (though actually unnecessary in the overall scheme of
> things) is not too expensive as it will always be the first item, however
> each of the items in the list (which is large) is then shuffled up, which
> will be relatively expensive.
> The removed links are also added to the session free list (which is also a
> proton list type) and then removed from that again in pn_link_finalize.
> (Also, pni_link_remove is then called in the pn_link_finalize (as well as
> having been called in the pn_link_free), though since the list should be
> emoty by this point that probably doesn't actually matter, its just looks
> odd).
> In more practical terms, this inefficiency can prevent applications using
> proton form properly handling sessions with large numbers of links on them.
> In the case of the dispatch router, if you have N connections (where N ==
> number of work threads) with a large number of links (say 60,000) and
> suddenly all N connections are disconnected, the worker threads will all be
> stuck processing pn_session_free for each of the disconnected connections
> leaving the router unresponsive to any other task for the duration.
> {noformat}
> Thread 5 (Thread 0x7f920cbb6700 (LWP 3624)):
> #0 0x00007f9216372e79 in pn_object_reify (object=0x7f91d0e63480) at
> /home/gordon/projects/proton/c/src/core/object/object.c:214
> #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>,
> a=0x7f91d0e63480, b=0x7f91f95f9300) at
> /home/gordon/projects/proton/c/src/core/object/object.c:158
> #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>,
> a=<optimized out>, b=b@entry=0x7f91f95f9300) at
> /home/gordon/projects/proton/c/src/core/object/object.c:169
> #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f91f801dc40,
> value=value@entry=0x7f91f95f9300) at
> /home/gordon/projects/proton/c/src/core/object/list.c:88
> #4 0x00007f92163737a9 in pn_list_remove (list=0x7f91f801dc40,
> value=value@entry=0x7f91f95f9300) at
> /home/gordon/projects/proton/c/src/core/object/list.c:99
> #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f91f804b870,
> link=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/engine.c:311
> #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:1119
> #7 pn_link_finalize (object=0x7f91f95f9300) at
> /home/gordon/projects/proton/c/src/core/engine.c:1098
> #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>,
> clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f91f95f9300) at
> /home/gordon/projects/proton/c/src/core/object/object.c:95
> #9 0x00007f921637326f in pn_decref (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/object/object.c:253
> #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:366
> #11 0x00007f921637f086 in pn_session_free (session=0x7f91f804b870) at
> /home/gordon/projects/proton/c/src/core/engine.c:271
> #12 0x00007f921637f165 in pn_connection_release (connection=0x7f9204050fa0)
> at /home/gordon/projects/proton/c/src/core/engine.c:125
> #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f9204050fa0) at
> /home/gordon/projects/proton/c/src/core/engine.c:146
> #14 0x00007f921637c296 in pn_connection_driver_destroy
> (d=d@entry=0x7f920405b4b8) at
> /home/gordon/projects/proton/c/src/core/connection_driver.c:94
> #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f920405af10) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:882
> #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:898
> #17 0x00007f921635cb82 in pconnection_done (pc=0x7f920405af10) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:1037
> #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f920405b4b0) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:2159
> #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at
> /home/gordon/projects/dispatch/src/server.c:1037
> #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0
> #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6
> Thread 4 (Thread 0x7f920d3b7700 (LWP 3623)):
> #0 0x00007f9216372e79 in pn_object_reify (object=0x7f9205c699d0) at
> /home/gordon/projects/proton/c/src/core/object/object.c:214
> #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>,
> a=0x7f9205c699d0, b=0x7f9201a87ca0) at
> /home/gordon/projects/proton/c/src/core/object/object.c:158
> #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>,
> a=<optimized out>, b=b@entry=0x7f9201a87ca0) at
> /home/gordon/projects/proton/c/src/core/object/object.c:169
> #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f920400bec0,
> value=value@entry=0x7f9201a87ca0) at
> /home/gordon/projects/proton/c/src/core/object/list.c:88
> #4 0x00007f92163737a9 in pn_list_remove (list=0x7f920400bec0,
> value=value@entry=0x7f9201a87ca0) at
> /home/gordon/projects/proton/c/src/core/object/list.c:99
> #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f9204077fd0,
> link=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/engine.c:311
> #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:1119
> #7 pn_link_finalize (object=0x7f9201a87ca0) at
> /home/gordon/projects/proton/c/src/core/engine.c:1098
> #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>,
> clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f9201a87ca0) at
> /home/gordon/projects/proton/c/src/core/object/object.c:95
> #9 0x00007f921637326f in pn_decref (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/object/object.c:253
> #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:366
> #11 0x00007f921637f086 in pn_session_free (session=0x7f9204077fd0) at
> /home/gordon/projects/proton/c/src/core/engine.c:271
> #12 0x00007f921637f165 in pn_connection_release (connection=0x7f9200053520)
> at /home/gordon/projects/proton/c/src/core/engine.c:125
> #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f9200053520) at
> /home/gordon/projects/proton/c/src/core/engine.c:146
> #14 0x00007f921637c296 in pn_connection_driver_destroy
> (d=d@entry=0x7f9200054258) at
> /home/gordon/projects/proton/c/src/core/connection_driver.c:94
> #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f9200053cb0) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:882
> #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:898
> #17 0x00007f921635cb82 in pconnection_done (pc=0x7f9200053cb0) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:1037
> #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f9200054250) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:2159
> #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at
> /home/gordon/projects/dispatch/src/server.c:1037
> #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0
> #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6
> Thread 3 (Thread 0x7f920dbb8700 (LWP 3622)):
> #0 0x00007f9216372e79 in pn_object_reify (object=0x7f91f9de8050) at
> /home/gordon/projects/proton/c/src/core/object/object.c:214
> #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>,
> a=0x7f91f9de8050, b=0x7f92059f0710) at
> /home/gordon/projects/proton/c/src/core/object/object.c:158
> #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>,
> a=<optimized out>, b=b@entry=0x7f92059f0710) at
> /home/gordon/projects/proton/c/src/core/object/object.c:169
> #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f9200036900,
> value=value@entry=0x7f92059f0710) at
> /home/gordon/projects/proton/c/src/core/object/list.c:88
> #4 0x00007f92163737a9 in pn_list_remove (list=0x7f9200036900,
> value=value@entry=0x7f92059f0710) at
> /home/gordon/projects/proton/c/src/core/object/list.c:99
> #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f920005a670,
> link=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/engine.c:311
> #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:1119
> #7 pn_link_finalize (object=0x7f92059f0710) at
> /home/gordon/projects/proton/c/src/core/engine.c:1098
> #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>,
> clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f92059f0710) at
> /home/gordon/projects/proton/c/src/core/object/object.c:95
> #9 0x00007f921637326f in pn_decref (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/object/object.c:253
> #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:366
> #11 0x00007f921637f086 in pn_session_free (session=0x7f920005a670) at
> /home/gordon/projects/proton/c/src/core/engine.c:271
> #12 0x00007f921637f165 in pn_connection_release (connection=0x7f91f8045d00)
> at /home/gordon/projects/proton/c/src/core/engine.c:125
> #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f91f8045d00) at
> /home/gordon/projects/proton/c/src/core/engine.c:146
> #14 0x00007f921637c296 in pn_connection_driver_destroy
> (d=d@entry=0x7f91f8061898) at
> /home/gordon/projects/proton/c/src/core/connection_driver.c:94
> #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f91f80612f0) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:882
> #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:898
> #17 0x00007f921635cb82 in pconnection_done (pc=0x7f91f80612f0) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:1037
> #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f91f8061890) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:2159
> #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at
> /home/gordon/projects/dispatch/src/server.c:1037
> #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0
> #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6
> Thread 2 (Thread 0x7f920e3f9700 (LWP 3621)):
> #0 0x00007f921631f3c5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007f92163e5199 in sys_cond_wait (cond=<optimized out>,
> held_mutex=<optimized out>) at
> /home/gordon/projects/dispatch/src/posix/threading.c:91
> #2 0x00007f92163fee77 in router_core_thread (arg=0xba9220) at
> /home/gordon/projects/dispatch/src/router_core/router_core_thread.c:176
> #3 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0
> #4 0x00007f9215f75553 in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x7f92156a5240 (LWP 3620)):
> #0 0x00007f9216372e79 in pn_object_reify (object=0x7f9202159b80) at
> /home/gordon/projects/proton/c/src/core/object/object.c:214
> #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>,
> a=0x7f9202159b80, b=0x7f91f9742490) at
> /home/gordon/projects/proton/c/src/core/object/object.c:158
> #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>,
> a=<optimized out>, b=b@entry=0x7f91f9742490) at
> /home/gordon/projects/proton/c/src/core/object/object.c:169
> #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f91f8023a40,
> value=value@entry=0x7f91f9742490) at
> /home/gordon/projects/proton/c/src/core/object/list.c:88
> #4 0x00007f92163737a9 in pn_list_remove (list=0x7f91f8023a40,
> value=value@entry=0x7f91f9742490) at
> /home/gordon/projects/proton/c/src/core/object/list.c:99
> #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f91f8067cb0,
> link=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/engine.c:311
> #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:1119
> #7 pn_link_finalize (object=0x7f91f9742490) at
> /home/gordon/projects/proton/c/src/core/engine.c:1098
> #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>,
> clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f91f9742490) at
> /home/gordon/projects/proton/c/src/core/object/object.c:95
> #9 0x00007f921637326f in pn_decref (object=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/object/object.c:253
> #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at
> /home/gordon/projects/proton/c/src/core/engine.c:366
> #11 0x00007f921637f086 in pn_session_free (session=0x7f91f8067cb0) at
> /home/gordon/projects/proton/c/src/core/engine.c:271
> #12 0x00007f921637f165 in pn_connection_release (connection=0x7f91f8044720)
> at /home/gordon/projects/proton/c/src/core/engine.c:125
> #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f91f8044720) at
> /home/gordon/projects/proton/c/src/core/engine.c:146
> #14 0x00007f921637c296 in pn_connection_driver_destroy
> (d=d@entry=0x7f91f8045458) at
> /home/gordon/projects/proton/c/src/core/connection_driver.c:94
> #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f91f8044eb0) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:882
> #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:898
> #17 0x00007f921635cb82 in pconnection_done (pc=0x7f91f8044eb0) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:1037
> #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f91f8045450) at
> /home/gordon/projects/proton/c/src/proactor/epoll.c:2159
> #19 0x00007f9216411bb5 in thread_run (arg=arg@entry=0xaf8380) at
> /home/gordon/projects/dispatch/src/server.c:1037
> #20 0x00007f9216411e60 in qd_server_run (qd=<optimized out>) at
> /home/gordon/projects/dispatch/src/server.c:1297
> #21 0x00000000004026b2 in main_process (config_path=0x7ffe78e17155
> "/home/gordon/projects/router-config-examples/router_a.conf",
> python_pkgdir=<optimized out>, test_hooks=<optimized out>, fd=2) at
> /home/gordon/projects/dispatch/router/src/main.c:113
> #22 0x0000000000402400 in main (argc=3, argv=0x7ffe78e16cb8) at
> /home/gordon/projects/dispatch/router/src/main.c:367
> {noformat}
> It would seem that pn_session_free could be more efficient in cleaning up the
> links since it knows it is doing them all at once (even if there are other
> references, itwill still remove them all from the session list of active
> links).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
