Re: Strange bug? "spam" in management log files...
I'm sorry for spaming, it seems that in my db.properties file on second MGMT srever, I had 127.0.0.1 as the Cluster IP. After this was changed to real IP address, it seems now that I dont have those "spam" log messages, seems all fine for some hours. On 5 June 2015 at 16:24, Andrija Panic wrote: > Hi, > > any hint on how to proceed ? > > on haproxy I see rougly 50%/50% sessions across 2 backend servers. > But inside DB, it all points to the one mgmt_server_ip... > > Thanks, > Andrija > > On 4 June 2015 at 19:27, Andrija Panic wrote: > >> And if of any help another hint: >> >> while Im having this lines sent to logs in high volume...if I stop second >> mgmt server, first one (that is making all these lines, doesnt stop to make >> them), so log is still heavily writen to - only when I also restart mgmt on >> 1st node (2nd node is down), then these log lines dissapear. >> >> Thx >> >> On 4 June 2015 at 19:19, Andrija Panic wrote: >> >>> And I could add - these lines (in this volume) only appears on first >>> mgmt server (Actually I have 2 separate, but identical ACS installations, >>> and same behaviour). >>> >>> On 4 June 2015 at 19:18, Andrija Panic wrote: >>> Just checked, in the HOSTS table, all agents are connected (via haproxy) to the first mgmt server...I just restarted haproxy, and still inside the DB, it says same mgmt_server_id for all agents - which is not really true. Actually, on the haproxy itslef (statistics page) I can see almoust 50%-50% distribution across 2 backends - which means by haproxy it should be fine. total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt server) This is our haproxy config, I think it's fine, but... DB says differently, althouh haproxy statistick say all fine ### ACS 8250 ### frontend front_ACS_8250 10.20.10.100:8250 option tcplog mode tcp default_backend back_8250 backend back_8250 mode tcp balance source server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise 3 fall 3 server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise 3 fall 3 ## Any info on how to proceed with this, since because of these lines, it makes mgmt logs almoust unreadable... :( Thanks, Andrija On 4 June 2015 at 19:00, Andrija Panic wrote: > Thanks Koushik, > > I will check and let you know - but 11GB log file for 10h ? I dont > expect this is expected :) > I understand that the message is there because of setup, just an awful > lot of lines > > Will check thx for the help ! > > Andrija > > On 4 June 2015 at 18:53, Koushik Das wrote: > >> This is expected in a clustered MS setup. What is the distribution of >> HV hosts across these MS (check host table in db for MS id)? MS owning >> the >> HV host processes all commands for that host. >> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in >> both MS logs to correlate. >> >> >> >> On 04-Jun-2015, at 8:30 PM, Andrija Panic >> wrote: >> >> > Hi, >> > >> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and >> sometimes it >> > happens that on the first node, we have extremem number of folowing >> line >> > entries in the log fie, which causes many GB log in just few hours >> or less: >> > (as you can see here they are not even that frequent, but >> sometimes, it >> > gets really crazy with the speed/numer logged per seconds: >> > >> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-27:null) Seq
Re: Strange bug? "spam" in management log files...
Hi, any hint on how to proceed ? on haproxy I see rougly 50%/50% sessions across 2 backend servers. But inside DB, it all points to the one mgmt_server_ip... Thanks, Andrija On 4 June 2015 at 19:27, Andrija Panic wrote: > And if of any help another hint: > > while Im having this lines sent to logs in high volume...if I stop second > mgmt server, first one (that is making all these lines, doesnt stop to make > them), so log is still heavily writen to - only when I also restart mgmt on > 1st node (2nd node is down), then these log lines dissapear. > > Thx > > On 4 June 2015 at 19:19, Andrija Panic wrote: > >> And I could add - these lines (in this volume) only appears on first mgmt >> server (Actually I have 2 separate, but identical ACS installations, and >> same behaviour). >> >> On 4 June 2015 at 19:18, Andrija Panic wrote: >> >>> Just checked, in the HOSTS table, all agents are connected (via haproxy) >>> to the first mgmt server...I just restarted haproxy, and still inside the >>> DB, it says same mgmt_server_id for all agents - which is not really true. >>> >>> Actually, on the haproxy itslef (statistics page) I can see almoust >>> 50%-50% distribution across 2 backends - which means by haproxy it should >>> be fine. >>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt >>> server) >>> >>> This is our haproxy config, I think it's fine, but... DB says >>> differently, althouh haproxy statistick say all fine >>> >>> ### ACS 8250 >>> ### >>> frontend front_ACS_8250 10.20.10.100:8250 >>> option tcplog >>> mode tcp >>> default_backend back_8250 >>> backend back_8250 >>> mode tcp >>> balance source >>> server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 >>> rise 3 fall 3 >>> server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 >>> rise 3 fall 3 >>> >>> ## >>> >>> Any info on how to proceed with this, since because of these lines, it >>> makes mgmt logs almoust unreadable... :( >>> >>> Thanks, >>> Andrija >>> >>> On 4 June 2015 at 19:00, Andrija Panic wrote: >>> Thanks Koushik, I will check and let you know - but 11GB log file for 10h ? I dont expect this is expected :) I understand that the message is there because of setup, just an awful lot of lines Will check thx for the help ! Andrija On 4 June 2015 at 18:53, Koushik Das wrote: > This is expected in a clustered MS setup. What is the distribution of > HV hosts across these MS (check host table in db for MS id)? MS owning the > HV host processes all commands for that host. > Grep for the sequence numbers (for e.g. 73-7374644389819187201) in > both MS logs to correlate. > > > > On 04-Jun-2015, at 8:30 PM, Andrija Panic > wrote: > > > Hi, > > > > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and > sometimes it > > happens that on the first node, we have extremem number of folowing > line > > entries in the log fie, which causes many GB log in just few hours > or less: > > (as you can see here they are not even that frequent, but sometimes, > it > > gets really crazy with the speed/numer logged per seconds: > > > > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Res
Re: Strange bug? "spam" in management log files...
And if of any help another hint: while Im having this lines sent to logs in high volume...if I stop second mgmt server, first one (that is making all these lines, doesnt stop to make them), so log is still heavily writen to - only when I also restart mgmt on 1st node (2nd node is down), then these log lines dissapear. Thx On 4 June 2015 at 19:19, Andrija Panic wrote: > And I could add - these lines (in this volume) only appears on first mgmt > server (Actually I have 2 separate, but identical ACS installations, and > same behaviour). > > On 4 June 2015 at 19:18, Andrija Panic wrote: > >> Just checked, in the HOSTS table, all agents are connected (via haproxy) >> to the first mgmt server...I just restarted haproxy, and still inside the >> DB, it says same mgmt_server_id for all agents - which is not really true. >> >> Actually, on the haproxy itslef (statistics page) I can see almoust >> 50%-50% distribution across 2 backends - which means by haproxy it should >> be fine. >> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt >> server) >> >> This is our haproxy config, I think it's fine, but... DB says >> differently, althouh haproxy statistick say all fine >> >> ### ACS 8250 >> ### >> frontend front_ACS_8250 10.20.10.100:8250 >> option tcplog >> mode tcp >> default_backend back_8250 >> backend back_8250 >> mode tcp >> balance source >> server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise >> 3 fall 3 >> server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise >> 3 fall 3 >> >> ## >> >> Any info on how to proceed with this, since because of these lines, it >> makes mgmt logs almoust unreadable... :( >> >> Thanks, >> Andrija >> >> On 4 June 2015 at 19:00, Andrija Panic wrote: >> >>> Thanks Koushik, >>> >>> I will check and let you know - but 11GB log file for 10h ? I dont >>> expect this is expected :) >>> I understand that the message is there because of setup, just an awful >>> lot of lines >>> >>> Will check thx for the help ! >>> >>> Andrija >>> >>> On 4 June 2015 at 18:53, Koushik Das wrote: >>> This is expected in a clustered MS setup. What is the distribution of HV hosts across these MS (check host table in db for MS id)? MS owning the HV host processes all commands for that host. Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both MS logs to correlate. On 04-Jun-2015, at 8:30 PM, Andrija Panic wrote: > Hi, > > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes it > happens that on the first node, we have extremem number of folowing line > entries in the log fie, which causes many GB log in just few hours or less: > (as you can see here they are not even that frequent, but sometimes, it > gets really crazy with the speed/numer logged per seconds: > > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: Mg
Re: Strange bug? "spam" in management log files...
And I could add - these lines (in this volume) only appears on first mgmt server (Actually I have 2 separate, but identical ACS installations, and same behaviour). On 4 June 2015 at 19:18, Andrija Panic wrote: > Just checked, in the HOSTS table, all agents are connected (via haproxy) > to the first mgmt server...I just restarted haproxy, and still inside the > DB, it says same mgmt_server_id for all agents - which is not really true. > > Actually, on the haproxy itslef (statistics page) I can see almoust > 50%-50% distribution across 2 backends - which means by haproxy it should > be fine. > total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt > server) > > This is our haproxy config, I think it's fine, but... DB says differently, > althouh haproxy statistick say all fine > > ### ACS 8250 > ### > frontend front_ACS_8250 10.20.10.100:8250 > option tcplog > mode tcp > default_backend back_8250 > backend back_8250 > mode tcp > balance source > server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise > 3 fall 3 > server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise > 3 fall 3 > > ## > > Any info on how to proceed with this, since because of these lines, it > makes mgmt logs almoust unreadable... :( > > Thanks, > Andrija > > On 4 June 2015 at 19:00, Andrija Panic wrote: > >> Thanks Koushik, >> >> I will check and let you know - but 11GB log file for 10h ? I dont >> expect this is expected :) >> I understand that the message is there because of setup, just an awful >> lot of lines >> >> Will check thx for the help ! >> >> Andrija >> >> On 4 June 2015 at 18:53, Koushik Das wrote: >> >>> This is expected in a clustered MS setup. What is the distribution of HV >>> hosts across these MS (check host table in db for MS id)? MS owning the HV >>> host processes all commands for that host. >>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both >>> MS logs to correlate. >>> >>> >>> >>> On 04-Jun-2015, at 8:30 PM, Andrija Panic >>> wrote: >>> >>> > Hi, >>> > >>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and >>> sometimes it >>> > happens that on the first node, we have extremem number of folowing >>> line >>> > entries in the log fie, which causes many GB log in just few hours or >>> less: >>> > (as you can see here they are not even that frequent, but sometimes, it >>> > gets really crazy with the speed/numer logged per seconds: >>> > >>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId >>> > 90520745449919: Resp: Routing to peer >>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImp
Re: Strange bug? "spam" in management log files...
Just checked, in the HOSTS table, all agents are connected (via haproxy) to the first mgmt server...I just restarted haproxy, and still inside the DB, it says same mgmt_server_id for all agents - which is not really true. Actually, on the haproxy itslef (statistics page) I can see almoust 50%-50% distribution across 2 backends - which means by haproxy it should be fine. total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt server) This is our haproxy config, I think it's fine, but... DB says differently, althouh haproxy statistick say all fine ### ACS 8250 ### frontend front_ACS_8250 10.20.10.100:8250 option tcplog mode tcp default_backend back_8250 backend back_8250 mode tcp balance source server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise 3 fall 3 server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise 3 fall 3 ## Any info on how to proceed with this, since because of these lines, it makes mgmt logs almoust unreadable... :( Thanks, Andrija On 4 June 2015 at 19:00, Andrija Panic wrote: > Thanks Koushik, > > I will check and let you know - but 11GB log file for 10h ? I dont expect > this is expected :) > I understand that the message is there because of setup, just an awful lot > of lines > > Will check thx for the help ! > > Andrija > > On 4 June 2015 at 18:53, Koushik Das wrote: > >> This is expected in a clustered MS setup. What is the distribution of HV >> hosts across these MS (check host table in db for MS id)? MS owning the HV >> host processes all commands for that host. >> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both >> MS logs to correlate. >> >> >> >> On 04-Jun-2015, at 8:30 PM, Andrija Panic >> wrote: >> >> > Hi, >> > >> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes >> it >> > happens that on the first node, we have extremem number of folowing line >> > entries in the log fie, which causes many GB log in just few hours or >> less: >> > (as you can see here they are not even that frequent, but sometimes, it >> > gets really crazy with the speed/numer logged per seconds: >> > >> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId >> > 90520745449919: Resp: Routing to peer >> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredA
Re: Strange bug? "spam" in management log files...
Thanks Koushik, I will check and let you know - but 11GB log file for 10h ? I dont expect this is expected :) I understand that the message is there because of setup, just an awful lot of lines Will check thx for the help ! Andrija On 4 June 2015 at 18:53, Koushik Das wrote: > This is expected in a clustered MS setup. What is the distribution of HV > hosts across these MS (check host table in db for MS id)? MS owning the HV > host processes all commands for that host. > Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both MS > logs to correlate. > > > > On 04-Jun-2015, at 8:30 PM, Andrija Panic wrote: > > > Hi, > > > > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes > it > > happens that on the first node, we have extremem number of folowing line > > entries in the log fie, which causes many GB log in just few hours or > less: > > (as you can see here they are not even that frequent, but sometimes, it > > gets really crazy with the speed/numer logged per seconds: > > > > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId > > 90520745449919: Resp: Routing to peer > > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId > > 90520745449919: Resp: Routing to peer > > > > > > We have haproxy VIP, to which SSVM connects, and all cloudstack agents > > (agent.properties file). > > > > Any suggestions, how to avoid this - I noticed when I turn off second ACS > > MGMT server, and then reboot first one (restart cloudstack-management) it > > stops and behaves nice :) > > > > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes. > > > > Thanks, > > -- > > > > Andrija Panić > > -- Andrija Panić
Re: Strange bug? "spam" in management log files...
This is expected in a clustered MS setup. What is the distribution of HV hosts across these MS (check host table in db for MS id)? MS owning the HV host processes all commands for that host. Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both MS logs to correlate. On 04-Jun-2015, at 8:30 PM, Andrija Panic wrote: > Hi, > > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes it > happens that on the first node, we have extremem number of folowing line > entries in the log fie, which causes many GB log in just few hours or less: > (as you can see here they are not even that frequent, but sometimes, it > gets really crazy with the speed/numer logged per seconds: > > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId > 90520745449919: Resp: Routing to peer > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId > 90520745449919: Resp: Routing to peer > > > We have haproxy VIP, to which SSVM connects, and all cloudstack agents > (agent.properties file). > > Any suggestions, how to avoid this - I noticed when I turn off second ACS > MGMT server, and then reboot first one (restart cloudstack-management) it > stops and behaves nice :) > > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes. > > Thanks, > -- > > Andrija Panić
Strange bug? "spam" in management log files...
Hi, I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes it happens that on the first node, we have extremem number of folowing line entries in the log fie, which causes many GB log in just few hours or less: (as you can see here they are not even that frequent, but sometimes, it gets really crazy with the speed/numer logged per seconds: 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId 90520745449919: Resp: Routing to peer 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId 90520745449919: Resp: Routing to peer We have haproxy VIP, to which SSVM connects, and all cloudstack agents (agent.properties file). Any suggestions, how to avoid this - I noticed when I turn off second ACS MGMT server, and then reboot first one (restart cloudstack-management) it stops and behaves nice :) This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes. Thanks, -- Andrija Panić