Personally from the original mail I think its as likely issue lies in just how the messages are being handled and responses generated. If adding threads is not helping any, it would only reinforce that view for me.
Note is made that a single thread is being used, and that messages are only queued by the thread and "handled elsewhere" "quickly", but "responses" take a long time. What type of queuing is being done? Can the queue block (which would stop the container doing _anything_ )? How are the messages actually then being handled and responses generated exactly? Unless that process is using the appropriate mechanisms for passing work back to the connection (/container if single) thread, it would both be both unsafe and may very well result in delays, because no IO would actually happen until something else entirely caused that connection to process again later (e.g heartbeat checking). On Fri, 3 Jun 2022 at 18:04, Cliff Jansen <[email protected]> wrote: > > Adding threads should allow connection setup (socket creation, accept, and > initial malloc of data structures) to run in parallel with connection > processing (socket read/write, TLS overhead, AMQP encode/decode, your > application on_message callback). > > The epoll proactor scales better with additional threads than the libuv > implementation. If you are seeing no benefit with extra threads, trying > the libuv proactor is a worthwhile idea. > > On Fri, Jun 3, 2022 at 2:38 AM Fredrik Hallenberg <[email protected]> > wrote: > > > Yes, the fd limit is already raised a lot. Increasing backlog has improved > > performance and more file descriptors are in use but still I feel > > connection times are too long. Is there anything else to tune in the > > proactor? Should I try with the libuv proactor instead of epoll? > > I have tried with multiple threads in the past but did not notice any > > difference, but perhaps it is worth trying again with the current backlog > > setting? > > > > > > On Thu, Jun 2, 2022 at 5:11 PM Cliff Jansen <[email protected]> wrote: > > > > > Please try raising your fd limit too. Perhaps doubling it or more. > > > > > > I would also try running your proton::container with more threads, say 4 > > > and then 16, and see if that makes a difference. It shouldn’t if your > > > processing within Proton is as minimal as you describe. However, if > > there > > > is lengthy lock contention as you pass work out and then back in to > > Proton, > > > that may introduce delays. > > > > > > On Thu, Jun 2, 2022 at 7:43 AM Fredrik Hallenberg <[email protected]> > > > wrote: > > > > > > > I have done some experiments raising the backlog value, and it is > > > > possibly a bit better, I have to test it more. Even if it works I would > > > of > > > > course like to avoid having to rely on a patched qpid. Also, maybe some > > > > internal queues or similar should be modified to handle this? > > > > > > > > I have not seen transport errors in the clients, but this may be > > because > > > > reconnection is enabled. I am unsure on what the reconnection feature > > > > actually does, I never seen an on_connection_open where > > > > connection.reconnection() returns true. > > > > Perhaps it is only useful when a connection is established and then > > lost? > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 2, 2022 at 1:44 PM Ted Ross <[email protected]> wrote: > > > > > > > > > On Thu, Jun 2, 2022 at 9:06 AM Fredrik Hallenberg < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi, my application tends to get a lot of short lived incoming > > > > > connections. > > > > > > Messages are very short sync messages that usually can be responded > > > > with > > > > > > very little processing on the server side. It works fine but I feel > > > > > > that the performance is a bit lacking when many connections happen > > at > > > > the > > > > > > same time and would like advice on how to improve it. I am using > > qpid > > > > > > proton c++ 0.37 with epoll proactor. > > > > > > My current design uses a single thread for the listener but it will > > > > > > immediately push incoming messages in on_message to a queue that is > > > > > handled > > > > > > elsewhere. I can see that clients have to wait for a long time (up > > > to a > > > > > > minute) until they get a response, but I don't believe there is an > > > > issue > > > > > on > > > > > > my end as I as will quickly deal with any client messages as soon > > as > > > > they > > > > > > show up. Rather the issues seems to be that messages are not pushed > > > > into > > > > > > the queue quickly enough. > > > > > > I have noticed that the pn_proactor_listen is hardcoded to use a > > > > backlog > > > > > of > > > > > > 16 in the default container implementation, this seems low, but I > > am > > > > not > > > > > > sure if it is correct to change it. > > > > > > Any advice apppreciated. My goal is that a client should never need > > > to > > > > > wait > > > > > > more than a few seconds for a response even under reasonably high > > > load, > > > > > > maybe a few hundred connections per seconds. > > > > > > > > > > > > > > > > I would try increasing the backlog. 16 seems low to me as well. Do > > > you > > > > > know if any of your clients are re-trying the connection setup > > because > > > > they > > > > > overran the server's backlog? > > > > > > > > > > -Ted > > > > > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
