nicob87 commented on a change in pull request #560: URL: https://github.com/apache/qpid-dispatch/pull/560#discussion_r408988308
########## File path: docs/notes/threading-info.txt ########## @@ -0,0 +1,591 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + + +===================================================== +Router Threading and Interprocess Communication Guide +===================================================== + +The Qpid Dispatch Router (qdrouterd) threading architecture is based +on two classes of threads: + + *) A Worker thread which interfaces with the Proton subsystem, and + *) The Core Routing thread which manages the routing database and +performs forwarding. + +In a running router there is a single Core Routing thread (core) and +one or more Worker threads. The number of Worker threads is +determined by the router configuration setting "workerThreads", which +defaults to four. + +IPC between these threads is done using Action queues, Work queues, +and manipulation of shared data. + + +Worker Threads +============== + +The Worker thread is responsible for interfacing with the Proton +subsystem. Only a worker thread can safely call directly into the +Proton library. The core thread must communicate with a worker thread +in order to have the worker manipulate Proton state on the core's +behalf. + +The Proton subsystem limits concurrency to a single connection. That +is, only one thread can be processing a given Proton connection (and +all of its child elements, such as links and deliveries) at a time. +The router honors this requirement by restricting access to a given +Proton connection (and its children) to a single worker thread at a +time. To say this another way, a particular Proton connection can be +processed by any worker threads but not concurrently. + +A worker thread is driven by the Proton proactor API. The worker's +main loop blocks on the proactor waiting for events, processes +incoming events, then blocks again. + + +Core Thread +=========== + +The one core thread has several responsibilities, including: + + *) Managing the forwarding state + *) Forwarding messages across the router + *) Forwarding disposition and settlement state changes across the router + *) Managing link credit flow + *) Responding to management requests for data owned by the core thread + +The core thread can be thought of as sitting in between the worker +threads, moving messages and state between them. + +When a worker thread needs to forward a received message it passes the +message, its associated delivery state, and incoming link identifier +to the core thread. + +The core thread uses the information supplied by the worker thread to +determine the outgoing link(s) for the message. Once the outgoing +link(s) are identified the core creates the necessary outgoing +delivery state(s) for sending the message out each link. + +The core binds together the incoming and outgoing deliveries (or +incoming and outgoing links in the case of link routing) so state +updates at one endpoint can be efficiently communicated to the peer +endpoint. + +The core then queues the message and outgoing delivery state to the +proper outgoing link(s). The core wakes the worker thread(s) (via the +Proton proactor) so the message(s) can be written out the Proton link. + +When delivery disposition or settlement changes are detected by a +worker thread it notifies the core thread. The core thread then uses +the linkage between incoming and outgoing state to propagate the +change to the peer. This results in the core thread setting the new +state in the peer link/delivery and waking a worker thread to update +the new state in Proton. + +The core also manages credit flow. The core grants credit to inbound +links. The core grants an initial batch of credit to a non-anonymous +link (a link with a target address) when the target address is +available for routing. The core will continue granting credit to the +link as long as the address is routable. The core ties the +replenishment of credit with the settlement of messages arriving on +the link: when the message is settled a single credit is granted by +the core. All credit flow operations involve having the core put a +credit flow work event on the proper inbound link then waking the +worker thread to update the credit in proton. + +The core can be the destination or source of AMQP message flows +from/to clients outside the router. This is used by services like the +Management Agent to receive management requests and send responses to +management clients. The core sends and receives these messages in +exactly the same way it forwards routed traffic - via the worker +threads. + +The core thread's main loop is driven by a work queue (the +qdr_core_t's action_list). The core thread blocks on a mutex +condition waiting for work. When a worker thread needs a service from +the core thread, the worker queues a work item to the action_list then +triggers the condition. This causes the core thread to unblock and +service the action_list work items. Review comment: I would just add a comment saying where to find in the code the thread main loop. ########## File path: docs/notes/threading-info.txt ########## @@ -0,0 +1,591 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + + +===================================================== +Router Threading and Interprocess Communication Guide +===================================================== + +The Qpid Dispatch Router (qdrouterd) threading architecture is based +on two classes of threads: + + *) A Worker thread which interfaces with the Proton subsystem, and + *) The Core Routing thread which manages the routing database and +performs forwarding. + +In a running router there is a single Core Routing thread (core) and +one or more Worker threads. The number of Worker threads is +determined by the router configuration setting "workerThreads", which +defaults to four. + +IPC between these threads is done using Action queues, Work queues, +and manipulation of shared data. + + +Worker Threads +============== + +The Worker thread is responsible for interfacing with the Proton +subsystem. Only a worker thread can safely call directly into the +Proton library. The core thread must communicate with a worker thread +in order to have the worker manipulate Proton state on the core's +behalf. + +The Proton subsystem limits concurrency to a single connection. That +is, only one thread can be processing a given Proton connection (and +all of its child elements, such as links and deliveries) at a time. +The router honors this requirement by restricting access to a given +Proton connection (and its children) to a single worker thread at a +time. To say this another way, a particular Proton connection can be +processed by any worker threads but not concurrently. + +A worker thread is driven by the Proton proactor API. The worker's +main loop blocks on the proactor waiting for events, processes +incoming events, then blocks again. + + +Core Thread +=========== + +The one core thread has several responsibilities, including: + + *) Managing the forwarding state + *) Forwarding messages across the router + *) Forwarding disposition and settlement state changes across the router + *) Managing link credit flow + *) Responding to management requests for data owned by the core thread + +The core thread can be thought of as sitting in between the worker +threads, moving messages and state between them. + +When a worker thread needs to forward a received message it passes the +message, its associated delivery state, and incoming link identifier +to the core thread. + +The core thread uses the information supplied by the worker thread to +determine the outgoing link(s) for the message. Once the outgoing +link(s) are identified the core creates the necessary outgoing +delivery state(s) for sending the message out each link. + +The core binds together the incoming and outgoing deliveries (or +incoming and outgoing links in the case of link routing) so state +updates at one endpoint can be efficiently communicated to the peer +endpoint. + +The core then queues the message and outgoing delivery state to the +proper outgoing link(s). The core wakes the worker thread(s) (via the +Proton proactor) so the message(s) can be written out the Proton link. + +When delivery disposition or settlement changes are detected by a +worker thread it notifies the core thread. The core thread then uses +the linkage between incoming and outgoing state to propagate the +change to the peer. This results in the core thread setting the new +state in the peer link/delivery and waking a worker thread to update +the new state in Proton. + +The core also manages credit flow. The core grants credit to inbound +links. The core grants an initial batch of credit to a non-anonymous +link (a link with a target address) when the target address is +available for routing. The core will continue granting credit to the +link as long as the address is routable. The core ties the +replenishment of credit with the settlement of messages arriving on +the link: when the message is settled a single credit is granted by +the core. All credit flow operations involve having the core put a +credit flow work event on the proper inbound link then waking the +worker thread to update the credit in proton. + +The core can be the destination or source of AMQP message flows +from/to clients outside the router. This is used by services like the +Management Agent to receive management requests and send responses to +management clients. The core sends and receives these messages in +exactly the same way it forwards routed traffic - via the worker +threads. + +The core thread's main loop is driven by a work queue (the +qdr_core_t's action_list). The core thread blocks on a mutex +condition waiting for work. When a worker thread needs a service from +the core thread, the worker queues a work item to the action_list then +triggers the condition. This causes the core thread to unblock and +service the action_list work items. + + +Embedded Python +=============== + +Some portions of the dispatch router are implemented in the Python +language. Python is currently used for parts of the routing protocol +implementation and some management. + +The python interpreter cannot be shared between threads - only one +thread can be executing the interpreter at a time. This is enforced +by the use of a single global lock that must be held whenever a thread +executes the Python interpreter. + + +Source Code Conventions +======================= + +In general, core thread code and data structure have names that are +prefixed by "qdr_". Functions that can only be run on the core thread +- that cannot be invoked from the context of a worker thread - are +suffixed with "_CT". For example "qdr_update_delivery_CT()" is a core +function that must never be invoked by a worker thread. + +Code that is private to the Core thread is in the "src/router_core" Review comment: This sentence is strictly true, but It can also suggest the "all the code" in that folder is core_thread code, and I think this is not the case... below in line 328 you can find that: ``` The worker thread code that processes the work_list can be found in src/router_core/connections.c::qdr_connection_process() ``` ########## File path: docs/notes/threading-info.txt ########## @@ -0,0 +1,591 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + + +===================================================== +Router Threading and Interprocess Communication Guide +===================================================== + +The Qpid Dispatch Router (qdrouterd) threading architecture is based +on two classes of threads: + + *) A Worker thread which interfaces with the Proton subsystem, and + *) The Core Routing thread which manages the routing database and +performs forwarding. + +In a running router there is a single Core Routing thread (core) and +one or more Worker threads. The number of Worker threads is +determined by the router configuration setting "workerThreads", which +defaults to four. + +IPC between these threads is done using Action queues, Work queues, +and manipulation of shared data. + + +Worker Threads +============== + +The Worker thread is responsible for interfacing with the Proton +subsystem. Only a worker thread can safely call directly into the +Proton library. The core thread must communicate with a worker thread +in order to have the worker manipulate Proton state on the core's +behalf. + +The Proton subsystem limits concurrency to a single connection. That +is, only one thread can be processing a given Proton connection (and +all of its child elements, such as links and deliveries) at a time. +The router honors this requirement by restricting access to a given +Proton connection (and its children) to a single worker thread at a +time. To say this another way, a particular Proton connection can be +processed by any worker threads but not concurrently. + +A worker thread is driven by the Proton proactor API. The worker's +main loop blocks on the proactor waiting for events, processes +incoming events, then blocks again. + + +Core Thread +=========== + +The one core thread has several responsibilities, including: + + *) Managing the forwarding state + *) Forwarding messages across the router + *) Forwarding disposition and settlement state changes across the router + *) Managing link credit flow + *) Responding to management requests for data owned by the core thread + +The core thread can be thought of as sitting in between the worker +threads, moving messages and state between them. + +When a worker thread needs to forward a received message it passes the Review comment: the worker knows that he needs to forward? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
