Thanks for the answer. I did some test and found when I have a lot of slow publisher (1 message/second), but 5000 of them. All publisher connect to a single client region(shared among all publisher), that client region use a lot of connections to geode server. is that normal? Because when I use fewer but faster publisher, I can see a lot less connection being used from client region to server region. I think from server point of view, it’s almost the same load. Why more connections? After a while, server max out 800 connections. Is that safe to increase this number to let’s say 4000? Or it’s better to set client cache factory a max connection number?
Thanks, Nan From: Akihiro Kitada [mailto:[email protected]] Sent: Thursday, October 19, 2017 9:30 PM To: [email protected] Subject: Re: max out 800 connections Hello Nan, >[warning 2017/10/19 16:00:47.219 EDT a1 <Handshaker >0.0.0.0/0.0.0.0:40411<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> > Thread 20> tid=0x11e9] Rejected >connection from >/171.135.144.25<https://urldefense.proofpoint.com/v2/url?u=http-3A__171.135.144.25&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=0GmLAzVpw8ar3wV475-XLR93UTxMrEZzEyIDfcaT1qM&e=> > because current connection count of 800 is greater than or equal to the >configured max of 800 : >Is the first line means max-connection is a server side settting? It means that the number of client connections for the server exceeds max-connections setting in the server side setting at cache-server attribute. http://geode.apache.org/docs/guide/12/reference/topics/cache_xml.html#cache-server<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_12_reference_topics_cache-5Fxml.html-23cache-2Dserver&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=EZAOFvnX72Vm8BgIawdHZAkICbw9cKfSWK8Je5SVgE0&e=> >The whole cluster only take 800 client connections? Or it’s a client side >setting, so each client can use up to 800 connections, so if I have 5 clients, >server side will be 4000 connections? It's a server side configuration for the each servers. Each cache servers accepts 800 clients connections by default. >[warning 2017/10/19 16:00:47.220 EDT a1 <Handshaker >0.0.0.0/0.0.0.0:40411<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> > Thread 20> tid=0x11e9] Rejected >connection from Server connection from [client host address=171.135.144.71; >client port=45894] because >incoming request was rejected by pool possibly due to thread exhaustion : >Second line means no more thread on the server? I have about 4000 threads on >server. any way to limit geode use certain number of thread? In this case, "pool" should mean the instance of java.util.concurrent.ThreadPoolExecutor used for server connections. The size of this pool equals to max-connections. So, those both two warning messages basically indicate that the number of client connections exceeds max-connections setting. The difference is that the first one is for client connections used for 'client to server' messages related to a client * queue (register interest, create cq, etc.) and the second one is for other purposes of client connections, according to the source code. You can find exact parts at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl by searching the following words. * AcceptorImpl_REJECTED_CONNECTION_FROM_0_BECAUSE_CURRENT_CONNECTION_COUNT_OF_1_IS_GREATER_THAN_OR_EQUAL_TO_THE_CONFIGURED_MAX_OF_2 * AcceptorImpl_REJECTED_CONNECTION_FROM_0_BECAUSE_REQUEST_REJECTED_BY_POOL Thanks. -- Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=5FY7L2xbyCREkTX91bYnim5siS9b8s0xShxiOX56F4M&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 [Image removed by sender. support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=4egEtrCETGSJB6Yj1FBvcAoaJddujRQ76REMB1CjOiQ&e=> [Image removed by sender. twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=sDNN2JmaGJFGYNQ8SPRpNJzMXWG4cAsRq8jTwBj5Oso&e=> [Image removed by sender. linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=27JxI5iQLaAHxQAjf4U8ZT5jFTdArFc12nsN2WKC6LA&e=> [Image removed by sender. facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=j6uWCSv_WdfZwG62Zc30Dn4cqwEkGlF6A3RMecWs7hA&e=> [Image removed by sender. google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=scc6bteFZUCqsKOVQXTZ1JSR0ufEQKULdE06eqPoZok&e=> [Image removed by sender. youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=Xk7Bbo5ZLK_RFppNOXZHBkxOi4ZjCbqxiCILhgyEcS4&e=> 2017-10-20 5:54 GMT+09:00 Xu, Nan <[email protected]<mailto:[email protected]>>: Hi, When load test our geode cluster, see those from our server log [warning 2017/10/19 16:00:47.219 EDT a1 <Handshaker 0.0.0.0/0.0.0.0:40411<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> Thread 20> tid=0x11e9] Rejected connection from /171.135.144.25<https://urldefense.proofpoint.com/v2/url?u=http-3A__171.135.144.25&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=0GmLAzVpw8ar3wV475-XLR93UTxMrEZzEyIDfcaT1qM&e=> because current connection count of 800 is greater than or equal to the configured max of 800 [warning 2017/10/19 16:00:47.220 EDT a1 <Handshaker 0.0.0.0/0.0.0.0:40411<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> Thread 20> tid=0x11e9] Rejected connection from Server connection from [client host address=171.135.144.71; client port=45894] because incoming request was rejected by pool possibly due to thread exhaustion We are client server setup. Is the first line means max-connection is a server side settting? The whole cluster only take 800 client connections? Or it’s a client side setting, so each client can use up to 800 connections, so if I have 5 clients, server side will be 4000 connections? Second line means no more thread on the server? I have about 4000 threads on server. any way to limit geode use certain number of thread? Thanks, Nan ________________________________ This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message. ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
