Re: ReceiveTimeoutTransportException in logs
Maybe you use a network filter / firewall which is misconfigured - no connection is possible, everything seems to time out. You must open TCP and UDP on port 9300 on all the hosts of the cluster nodes if you use TransportClient. Also check if your network can operate regarding other nodes, if you have set sniff=true. Jörg On Tue, Sep 16, 2014 at 10:49 AM, Abhishek Aggarwal boyobo...@gmail.com wrote: I am connecting to single instance of Elastic Search server remotely via Transport client. In my web application which makes use of Transport client, i am see following messages in the logs: I have checked, my network connection is proper and ES server is up. But still getting these messages in the logs. How to avoid these messages? Why it keeps getting disconnected/ connected? I would appreciate if you could provide fix for this issue. 12:39:24,658 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1115] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:24,799 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:39,260 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:44,283 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1116] timed out after [5023ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:44,283 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:50,663 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:08,447 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1119] timed out after [5007ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:40:08,447 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:18,306 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:23,330 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1120] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:40:23,330 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:28,618 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:42:46,792 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1146] timed out after [5008ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:42:46,792 DEBUG [netty] [Cassie Lang] disconnected from
Re: ReceiveTimeoutTransportException in logs
Thanks for the reply. I am facing this error intermittently. Transport Client works fine sometimes - so it rules out firewall or port related issues. I have only one ES node (version 1.1.1) - Firewall is not configured - TCP and UDP on port 9300 are open - sniff is disabled (I 'm using default transport configurations) Regards. abhishek On Tuesday, 16 September 2014 14:36:04 UTC+5:30, Jörg Prante wrote: Maybe you use a network filter / firewall which is misconfigured - no connection is possible, everything seems to time out. You must open TCP and UDP on port 9300 on all the hosts of the cluster nodes if you use TransportClient. Also check if your network can operate regarding other nodes, if you have set sniff=true. Jörg On Tue, Sep 16, 2014 at 10:49 AM, Abhishek Aggarwal boyo...@gmail.com javascript: wrote: I am connecting to single instance of Elastic Search server remotely via Transport client. In my web application which makes use of Transport client, i am see following messages in the logs: I have checked, my network connection is proper and ES server is up. But still getting these messages in the logs. How to avoid these messages? Why it keeps getting disconnected/ connected? I would appreciate if you could provide fix for this issue. 12:39:24,658 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1115] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:24,799 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:39,260 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:44,283 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1116] timed out after [5023ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:44,283 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:50,663 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:08,447 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1119] timed out after [5007ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:40:08,447 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:18,306 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:23,330 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1120] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:40:23,330 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:28,618 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:42:46,792 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException:
Re: ReceiveTimeoutTransportException in logs
Can you manually test all of that using telnet? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 16 September 2014 20:09, Abhishek Aggarwal boyobo...@gmail.com wrote: Thanks for the reply. I am facing this error intermittently. Transport Client works fine sometimes - so it rules out firewall or port related issues. I have only one ES node (version 1.1.1) - Firewall is not configured - TCP and UDP on port 9300 are open - sniff is disabled (I 'm using default transport configurations) Regards. abhishek On Tuesday, 16 September 2014 14:36:04 UTC+5:30, Jörg Prante wrote: Maybe you use a network filter / firewall which is misconfigured - no connection is possible, everything seems to time out. You must open TCP and UDP on port 9300 on all the hosts of the cluster nodes if you use TransportClient. Also check if your network can operate regarding other nodes, if you have set sniff=true. Jörg On Tue, Sep 16, 2014 at 10:49 AM, Abhishek Aggarwal boyo...@gmail.com wrote: I am connecting to single instance of Elastic Search server remotely via Transport client. In my web application which makes use of Transport client, i am see following messages in the logs: I have checked, my network connection is proper and ES server is up. But still getting these messages in the logs. How to avoid these messages? Why it keeps getting disconnected/ connected? I would appreciate if you could provide fix for this issue. 12:39:24,658 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1115] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:24,799 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:39,260 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:44,283 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1116] timed out after [5023ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:44,283 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:50,663 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:08,447 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1119] timed out after [5007ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:40:08,447 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:18,306 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:23,330 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1120] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:40:23,330 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:28,618 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]]
Re: ReceiveTimeoutTransportException in logs
We are also facing this kind of issue in es version 1.1.1. Some node gets disconnected and while analyzing the logs in that disconnected node we got a lot connection time out error. So sometime this issue gets solved by restarting the master node, but sometime we may need to restart the whole cluster We are planning to upgrade the cluster to version 1.3.2 Regards On Tue, Sep 16, 2014 at 3:55 PM, Mark Walkom ma...@campaignmonitor.com wrote: Can you manually test all of that using telnet? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 16 September 2014 20:09, Abhishek Aggarwal boyobo...@gmail.com wrote: Thanks for the reply. I am facing this error intermittently. Transport Client works fine sometimes - so it rules out firewall or port related issues. I have only one ES node (version 1.1.1) - Firewall is not configured - TCP and UDP on port 9300 are open - sniff is disabled (I 'm using default transport configurations) Regards. abhishek On Tuesday, 16 September 2014 14:36:04 UTC+5:30, Jörg Prante wrote: Maybe you use a network filter / firewall which is misconfigured - no connection is possible, everything seems to time out. You must open TCP and UDP on port 9300 on all the hosts of the cluster nodes if you use TransportClient. Also check if your network can operate regarding other nodes, if you have set sniff=true. Jörg On Tue, Sep 16, 2014 at 10:49 AM, Abhishek Aggarwal boyo...@gmail.com wrote: I am connecting to single instance of Elastic Search server remotely via Transport client. In my web application which makes use of Transport client, i am see following messages in the logs: I have checked, my network connection is proper and ES server is up. But still getting these messages in the logs. How to avoid these messages? Why it keeps getting disconnected/ connected? I would appreciate if you could provide fix for this issue. 12:39:24,658 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1115] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:24,799 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:39,260 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:44,283 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1116] timed out after [5023ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:39:44,283 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:39:50,663 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:08,447 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1119] timed out after [5007ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12:40:08,447 DEBUG [netty] [Cassie Lang] disconnected from [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:18,306 DEBUG [netty] [Cassie Lang] connected to node [[#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]]] 12:40:23,330 INFO [transport] [Cassie Lang] failed to get node info for [#transport#-1][ABAGGARW-WS01][inet[/173.36.55.182:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/173.36.55.182:9300]][cluster/nodes/info] request_id [1120] timed out after [5024ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run( TransportService.java:369) at
