[ https://issues.apache.org/jira/browse/PROTON-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Ross updated PROTON-2151: -------------------------------- Labels: perf (was: ) > pn_session_free is inefficient if handling large numbers of links > ----------------------------------------------------------------- > > Key: PROTON-2151 > URL: https://issues.apache.org/jira/browse/PROTON-2151 > Project: Qpid Proton > Issue Type: Improvement > Reporter: Gordon Sim > Priority: Major > Labels: perf > > The links for a session are kept in a proton list object, which is > essentially an array. > Removing a link from this list involves first searching for the link by > iterating over the array comparing it with the desired object using the > comparator defined for the class, then cleaning up that element and shuffling > up any remaining elements. > In the case of pn_session_free, it loops calling on_link_free on the first > element in the list (which involves among other things removing it from the > list as described above) until there are no more elements in the list. > Looking up the link (though actually unnecessary in the overall scheme of > things) is not too expensive as it will always be the first item, however > each of the items in the list (which is large) is then shuffled up, which > will be relatively expensive. > The removed links are also added to the session free list (which is also a > proton list type) and then removed from that again in pn_link_finalize. > (Also, pni_link_remove is then called in the pn_link_finalize (as well as > having been called in the pn_link_free), though since the list should be > emoty by this point that probably doesn't actually matter, its just looks > odd). > In more practical terms, this inefficiency can prevent applications using > proton form properly handling sessions with large numbers of links on them. > In the case of the dispatch router, if you have N connections (where N == > number of work threads) with a large number of links (say 60,000) and > suddenly all N connections are disconnected, the worker threads will all be > stuck processing pn_session_free for each of the disconnected connections > leaving the router unresponsive to any other task for the duration. > {noformat} > Thread 5 (Thread 0x7f920cbb6700 (LWP 3624)): > #0 0x00007f9216372e79 in pn_object_reify (object=0x7f91d0e63480) at > /home/gordon/projects/proton/c/src/core/object/object.c:214 > #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, > a=0x7f91d0e63480, b=0x7f91f95f9300) at > /home/gordon/projects/proton/c/src/core/object/object.c:158 > #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, > a=<optimized out>, b=b@entry=0x7f91f95f9300) at > /home/gordon/projects/proton/c/src/core/object/object.c:169 > #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f91f801dc40, > value=value@entry=0x7f91f95f9300) at > /home/gordon/projects/proton/c/src/core/object/list.c:88 > #4 0x00007f92163737a9 in pn_list_remove (list=0x7f91f801dc40, > value=value@entry=0x7f91f95f9300) at > /home/gordon/projects/proton/c/src/core/object/list.c:99 > #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f91f804b870, > link=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/engine.c:311 > #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:1119 > #7 pn_link_finalize (object=0x7f91f95f9300) at > /home/gordon/projects/proton/c/src/core/engine.c:1098 > #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, > clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f91f95f9300) at > /home/gordon/projects/proton/c/src/core/object/object.c:95 > #9 0x00007f921637326f in pn_decref (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/object/object.c:253 > #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:366 > #11 0x00007f921637f086 in pn_session_free (session=0x7f91f804b870) at > /home/gordon/projects/proton/c/src/core/engine.c:271 > #12 0x00007f921637f165 in pn_connection_release (connection=0x7f9204050fa0) > at /home/gordon/projects/proton/c/src/core/engine.c:125 > #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f9204050fa0) at > /home/gordon/projects/proton/c/src/core/engine.c:146 > #14 0x00007f921637c296 in pn_connection_driver_destroy > (d=d@entry=0x7f920405b4b8) at > /home/gordon/projects/proton/c/src/core/connection_driver.c:94 > #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f920405af10) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:882 > #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:898 > #17 0x00007f921635cb82 in pconnection_done (pc=0x7f920405af10) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 > #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f920405b4b0) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 > #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at > /home/gordon/projects/dispatch/src/server.c:1037 > #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x7f920d3b7700 (LWP 3623)): > #0 0x00007f9216372e79 in pn_object_reify (object=0x7f9205c699d0) at > /home/gordon/projects/proton/c/src/core/object/object.c:214 > #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, > a=0x7f9205c699d0, b=0x7f9201a87ca0) at > /home/gordon/projects/proton/c/src/core/object/object.c:158 > #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, > a=<optimized out>, b=b@entry=0x7f9201a87ca0) at > /home/gordon/projects/proton/c/src/core/object/object.c:169 > #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f920400bec0, > value=value@entry=0x7f9201a87ca0) at > /home/gordon/projects/proton/c/src/core/object/list.c:88 > #4 0x00007f92163737a9 in pn_list_remove (list=0x7f920400bec0, > value=value@entry=0x7f9201a87ca0) at > /home/gordon/projects/proton/c/src/core/object/list.c:99 > #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f9204077fd0, > link=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/engine.c:311 > #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:1119 > #7 pn_link_finalize (object=0x7f9201a87ca0) at > /home/gordon/projects/proton/c/src/core/engine.c:1098 > #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, > clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f9201a87ca0) at > /home/gordon/projects/proton/c/src/core/object/object.c:95 > #9 0x00007f921637326f in pn_decref (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/object/object.c:253 > #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:366 > #11 0x00007f921637f086 in pn_session_free (session=0x7f9204077fd0) at > /home/gordon/projects/proton/c/src/core/engine.c:271 > #12 0x00007f921637f165 in pn_connection_release (connection=0x7f9200053520) > at /home/gordon/projects/proton/c/src/core/engine.c:125 > #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f9200053520) at > /home/gordon/projects/proton/c/src/core/engine.c:146 > #14 0x00007f921637c296 in pn_connection_driver_destroy > (d=d@entry=0x7f9200054258) at > /home/gordon/projects/proton/c/src/core/connection_driver.c:94 > #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f9200053cb0) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:882 > #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:898 > #17 0x00007f921635cb82 in pconnection_done (pc=0x7f9200053cb0) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 > #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f9200054250) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 > #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at > /home/gordon/projects/dispatch/src/server.c:1037 > #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7f920dbb8700 (LWP 3622)): > #0 0x00007f9216372e79 in pn_object_reify (object=0x7f91f9de8050) at > /home/gordon/projects/proton/c/src/core/object/object.c:214 > #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, > a=0x7f91f9de8050, b=0x7f92059f0710) at > /home/gordon/projects/proton/c/src/core/object/object.c:158 > #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, > a=<optimized out>, b=b@entry=0x7f92059f0710) at > /home/gordon/projects/proton/c/src/core/object/object.c:169 > #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f9200036900, > value=value@entry=0x7f92059f0710) at > /home/gordon/projects/proton/c/src/core/object/list.c:88 > #4 0x00007f92163737a9 in pn_list_remove (list=0x7f9200036900, > value=value@entry=0x7f92059f0710) at > /home/gordon/projects/proton/c/src/core/object/list.c:99 > #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f920005a670, > link=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/engine.c:311 > #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:1119 > #7 pn_link_finalize (object=0x7f92059f0710) at > /home/gordon/projects/proton/c/src/core/engine.c:1098 > #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, > clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f92059f0710) at > /home/gordon/projects/proton/c/src/core/object/object.c:95 > #9 0x00007f921637326f in pn_decref (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/object/object.c:253 > #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:366 > #11 0x00007f921637f086 in pn_session_free (session=0x7f920005a670) at > /home/gordon/projects/proton/c/src/core/engine.c:271 > #12 0x00007f921637f165 in pn_connection_release (connection=0x7f91f8045d00) > at /home/gordon/projects/proton/c/src/core/engine.c:125 > #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f91f8045d00) at > /home/gordon/projects/proton/c/src/core/engine.c:146 > #14 0x00007f921637c296 in pn_connection_driver_destroy > (d=d@entry=0x7f91f8061898) at > /home/gordon/projects/proton/c/src/core/connection_driver.c:94 > #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f91f80612f0) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:882 > #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:898 > #17 0x00007f921635cb82 in pconnection_done (pc=0x7f91f80612f0) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 > #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f91f8061890) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 > #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at > /home/gordon/projects/dispatch/src/server.c:1037 > #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f920e3f9700 (LWP 3621)): > #0 0x00007f921631f3c5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f92163e5199 in sys_cond_wait (cond=<optimized out>, > held_mutex=<optimized out>) at > /home/gordon/projects/dispatch/src/posix/threading.c:91 > #2 0x00007f92163fee77 in router_core_thread (arg=0xba9220) at > /home/gordon/projects/dispatch/src/router_core/router_core_thread.c:176 > #3 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007f9215f75553 in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f92156a5240 (LWP 3620)): > #0 0x00007f9216372e79 in pn_object_reify (object=0x7f9202159b80) at > /home/gordon/projects/proton/c/src/core/object/object.c:214 > #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, > a=0x7f9202159b80, b=0x7f91f9742490) at > /home/gordon/projects/proton/c/src/core/object/object.c:158 > #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, > a=<optimized out>, b=b@entry=0x7f91f9742490) at > /home/gordon/projects/proton/c/src/core/object/object.c:169 > #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f91f8023a40, > value=value@entry=0x7f91f9742490) at > /home/gordon/projects/proton/c/src/core/object/list.c:88 > #4 0x00007f92163737a9 in pn_list_remove (list=0x7f91f8023a40, > value=value@entry=0x7f91f9742490) at > /home/gordon/projects/proton/c/src/core/object/list.c:99 > #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f91f8067cb0, > link=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/engine.c:311 > #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:1119 > #7 pn_link_finalize (object=0x7f91f9742490) at > /home/gordon/projects/proton/c/src/core/engine.c:1098 > #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, > clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f91f9742490) at > /home/gordon/projects/proton/c/src/core/object/object.c:95 > #9 0x00007f921637326f in pn_decref (object=<optimized out>) at > /home/gordon/projects/proton/c/src/core/object/object.c:253 > #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at > /home/gordon/projects/proton/c/src/core/engine.c:366 > #11 0x00007f921637f086 in pn_session_free (session=0x7f91f8067cb0) at > /home/gordon/projects/proton/c/src/core/engine.c:271 > #12 0x00007f921637f165 in pn_connection_release (connection=0x7f91f8044720) > at /home/gordon/projects/proton/c/src/core/engine.c:125 > #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f91f8044720) at > /home/gordon/projects/proton/c/src/core/engine.c:146 > #14 0x00007f921637c296 in pn_connection_driver_destroy > (d=d@entry=0x7f91f8045458) at > /home/gordon/projects/proton/c/src/core/connection_driver.c:94 > #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f91f8044eb0) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:882 > #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:898 > #17 0x00007f921635cb82 in pconnection_done (pc=0x7f91f8044eb0) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 > #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f91f8045450) at > /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 > #19 0x00007f9216411bb5 in thread_run (arg=arg@entry=0xaf8380) at > /home/gordon/projects/dispatch/src/server.c:1037 > #20 0x00007f9216411e60 in qd_server_run (qd=<optimized out>) at > /home/gordon/projects/dispatch/src/server.c:1297 > #21 0x00000000004026b2 in main_process (config_path=0x7ffe78e17155 > "/home/gordon/projects/router-config-examples/router_a.conf", > python_pkgdir=<optimized out>, test_hooks=<optimized out>, fd=2) at > /home/gordon/projects/dispatch/router/src/main.c:113 > #22 0x0000000000402400 in main (argc=3, argv=0x7ffe78e16cb8) at > /home/gordon/projects/dispatch/router/src/main.c:367 > {noformat} > It would seem that pn_session_free could be more efficient in cleaning up the > links since it knows it is doing them all at once (even if there are other > references, itwill still remove them all from the session list of active > links). -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org