From: Sheng Yang <sh...@yasker.org<mailto:sh...@yasker.org>> Date: Wednesday, April 3, 2013 10:44 AM To: "<dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>, Kelven Yang <kelven.y...@citrix.com<mailto:kelven.y...@citrix.com>> Subject: Re: [ACS41] Management server closes connection on port 8250
On Tue, Apr 2, 2013 at 1:12 PM, Chip Childers <chip.child...@sungard.com<mailto:chip.child...@sungard.com>> wrote: On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote: > Hi, > > On 04/02/2013 08:50 PM, Sheng Yang wrote: > >On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang > ><sh...@yasker.org<mailto:sh...@yasker.org>> wrote: > >>On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander > >><w...@widodh.nl<mailto:w...@widodh.nl>> wrote: > >>>Hi, > >>> > >>>Since I've upgraded my management server to 4.1 my agents refuse to > >>>connect, > >>>their log says: > >>> > >>>2013-04-02 20:26:11,207 INFO [utils.nio.NioClient] (Agent-Selector:null) > >>>Connecting to 31.25.X.X:8250 > >>>2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection] > >>>(Agent-Selector:null) Unable to initialize the threads. > >>>java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection > >>>closed with -1 on reading size. > >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:83) > >>> at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108) > >>> at java.lang.Thread.run(Thread.java:679) > >>> > >>>So I tried a telnet connection: > >>> > >>>Connection closed by foreign host. > >>>root@stack01:~# telnet 31.25.X.X 8250 > >>>Trying 31.25.X.X... > >>>Connected to 31.X.X.X. > >>>Escape character is '^]'. > >>>Connection closed by foreign host. > >>>root@stack01:~# > >>> > >>>So I didn't close the connection, but it was the management server. > >>> > >>>I cranked up the logging to DEBUG, but nothing shows in any of the logs, so > >>>I have no clue why this isn't working. > >>> > >>>On the mgmt server I see Java in LISTEN state on port 8250 > >>> > >>>There is no firewall on the management server (it's my lab!). > >>> > >>>Any clues to what this could be before I start filing a in Jira? Since I'm > >>>not sure if this is a bug. > >> > >>Have you upgrade your agent? > > > > Yes, the Agent is also 4.1 > > >Also you could try to enable TRACE for com.cloud.utils.nio to see more > >log of NIO. > > > > I think you mean at the agent? > > But what I think is weird is that the mgmt server directly closes > the telnet connection as well and nothing shows up in the logs. > > In the mgmt server log I found this: > > 2013-04-02 22:08:41,156 INFO [utils.nio.NioServer] > (AgentManager-Selector:null) NioConnection started and listening on > 0.0.0.0/0.0.0.0:8250<http://0.0.0.0/0.0.0.0:8250> > > What I did notice is that when I shutdown the mgmt server the Agent > connects for a second: > > 2013-04-02 22:07:53,689 INFO [utils.nio.NioClient] > (Agent-Selector:null) Connecting to 31.25.X.X:8250 > 2013-04-02 22:07:53,934 INFO [utils.nio.NioClient] > (Agent-Selector:null) SSL: Handshake done > > And then the mgmt server exists and the connection gets lost. > > ANY clues about this weird behavior? > > Wido > > >--Sheng > >> > >>--Sheng > >>> > >>>Wido > Wido - Can you please open a blocker bug to track this? It may end up being a non-issue or something... but I want to know that we have it outstanding when considering when to cut the RC. It remind me of one weird bug. The corrupted ssl keystore(due to some encryption/decryption of db which I don't understand well, I remember Kelven knows more about it) on mgmt server make it impossible to connect to mgmt server. Yes, I encountered an issue in upgrade case where password related fields seem to be encrypted/corrupted after the migration process. It it worth to dig it out for a better explanation -kelven Could you clear "ssl.keystore" entry in configuration table of db, as well as cloud.keystore file in the configuration path and restart the mgmt server and try again? --Sheng
