Just checked, in the HOSTS table, all agents are connected (via haproxy) to
the first mgmt server...I just restarted haproxy, and still inside the DB,
it says same mgmt_server_id for all agents - which is not really true.
Actually, on the haproxy itslef (statistics page) I can see almoust 50%-50%
distribution across 2 backends - which means by haproxy it should be fine.
total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
server)
This is our haproxy config, I think it's fine, but... DB says differently,
althouh haproxy statistick say all fine
### ACS 8250
#######################################################################################
frontend front_ACS_8250 10.20.10.100:8250
option tcplog
mode tcp
default_backend back_8250
backend back_8250
mode tcp
balance source
server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise 3
fall 3
server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise 3
fall 3
##################################################################################################
Any info on how to proceed with this, since because of these lines, it
makes mgmt logs almoust unreadable... :(
Thanks,
Andrija
On 4 June 2015 at 19:00, Andrija Panic <andrija.pa...@gmail.com> wrote:
> Thanks Koushik,
>
> I will check and let you know - but 11GB log file for 10h ? I dont expect
> this is expected :)
> I understand that the message is there because of setup, just an awful lot
> of lines....
>
> Will check thx for the help !
>
> Andrija
>
> On 4 June 2015 at 18:53, Koushik Das <koushik....@citrix.com> wrote:
>
>> This is expected in a clustered MS setup. What is the distribution of HV
>> hosts across these MS (check host table in db for MS id)? MS owning the HV
>> host processes all commands for that host.
>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both
>> MS logs to correlate.
>>
>>
>>
>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <andrija.pa...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes
>> it
>> > happens that on the first node, we have extremem number of folowing line
>> > entries in the log fie, which causes many GB log in just few hours or
>> less:
>> > (as you can see here they are not even that frequent, but sometimes, it
>> > gets really crazy with the speed/numer logged per seconds:
>> >
>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> >
>> >
>> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
>> > (agent.properties file).
>> >
>> > Any suggestions, how to avoid this - I noticed when I turn off second
>> ACS
>> > MGMT server, and then reboot first one (restart cloudstack-management)
>> it
>> > stops and behaves nice :)
>> >
>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>> >
>> > Thanks,
>> > --
>> >
>> > Andrija Panić
>>
>>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
