Re: [ossec-list] Re: problems registering agents
Thank you! This is a huge help. The upgrade to 2.0 locally was painless
*and* fixed my authd issues. Now to production.
On Mon, May 22, 2017 at 7:19 PM, Jesus Linares wrote:
> Hi,
>
> it is a known issue in that version (1.1.1). It is related with the
> algorithm that assigns an agent ID. This issue is fixed in Wazuh 2.0.
>
> Also, you can use the API to register agents remotely: 1.1.1
> <https://documentation.wazuh.com/1.1/ossec_api.html> and 2.0
> <https://documentation.wazuh.com/current/user-manual/api/index.html> API
> documentation.
>
> Regards.
>
> On Monday, May 22, 2017 at 6:56:10 PM UTC+2, Topper Bowers wrote:
>>
>> I deleted some of the lines starting with bang (!) but that didn't clear
>> up the problem. My client.keys is now smaller than 2048, but I still can't
>> add agents. I was able to duplicate this problem on a fresh install in
>> vagrant. Using the bin/manage_agents command I was able to add over 4k
>> clients (and clients.keys grew without problem). However, when I try to add
>> a new agent through authd... I get the same internal error problem.
>>
>> Results of commands:
>>
>> $ cat /var/ossec/etc/client.keys | wc -l
>>
>> 2032
>>
>> $ cat /var/ossec/etc/client.keys | grep -P "^\d+\s*\!" -v | wc -l
>>
>> 209
>>
>> $ cat /var/ossec/etc/client.keys | grep -P "^\d+\s*\!" | wc -l
>>
>> 1823
>>
>> On Mon, May 22, 2017 at 6:28 PM, Jesus Linares wrote:
>>
>>> Hi,
>>>
>>> as you mentioned, it seems that inactive agents are counting for the
>>> limit (2048 agents). Run the following commands in order to know the size
>>> of the *client.keys *file:
>>>
>>>- Total lines: cat /var/ossec/etc/client.keys | wc -l
>>>- Active agents: cat /var/ossec/etc/client.keys | grep -P
>>> "^\d+\s*\!" -v | wc -l
>>>- Inactive agents: cat /var/ossec/etc/client.keys | grep -P
>>>"^\d+\s*\!" | wc -l
>>>
>>> The solution could be clean the client.keys (lines with "!") after
>>> removing the agent.
>>>
>>> Regards.
>>>
>>>
>>> On Monday, May 22, 2017 at 11:05:38 AM UTC+2, Topper Bowers wrote:
>>>>
>>>> Hi,
>>>>
>>>> My client has a highly dynamic environment and we're using OSSEC (wazuh
>>>> 1.1.1 release, OSSEC v2.8). When a server spins up, it registers itself as
>>>> an agent to the servers authd and everything was going ok. However, my
>>>> client.keys file is now 2048 lines long and no new agents can register.
>>>> They get an "(internal error)" that we see in the /var/ossec/logs/ossec.log
>>>>
>>>> We have a process in place to remove inactive agents using the
>>>> `/var/ossec/bin/manage_agents -r ${ossec_id}` command. And if you use
>>>> /var/ossec/bin/manage_agents -l only about 100 agents show up.
>>>>
>>>> I've seen this https://groups.google.com/forum/#!topic/ossec-list/lgFD
>>>> OlR6zNg and it looks remarkably similar to what we're seeing. However,
>>>> we don't actually have thousands of active agents. It seems like inactive
>>>> agents are counting against the limit. Since we have a really dynamic
>>>> environment with servers going up and down all the time, increasing the
>>>> limit seems like it's just pushing out the inevitable.
>>>>
>>>> In summary... dynamic environment, can't add new agents, only 100 or so
>>>> active agents, 2048 lines in client.keys. No other error messages besides
>>>> "internal error"
>>>>
>>>> Any suggestions?
>>>>
>>>> Thanks!
>>>>
>>>> Topper
>>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "ossec-list" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>>> pic/ossec-list/k_MFr5aAjRU/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> ossec-list+...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>>
>> *Topper Bowers*
>>
>> *Engineering*
>> *Vitals* | 160 Chubb Ave, Suite 301, Lyndhurst, NJ 07071, USA
>>
>> M : 646.515.6630 <(646)%20515-6630>
>>
>> http://www.vitals.com
>>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "ossec-list" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/ossec-list/k_MFr5aAjRU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> ossec-list+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
*Topper Bowers*
*Engineering*
*Vitals* | 160 Chubb Ave, Suite 301, Lyndhurst, NJ 07071, USA
M : 646.515.6630
http://www.vitals.com
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [ossec-list] Re: problems registering agents
I deleted some of the lines starting with bang (!) but that didn't clear up
the problem. My client.keys is now smaller than 2048, but I still can't add
agents. I was able to duplicate this problem on a fresh install in vagrant.
Using the bin/manage_agents command I was able to add over 4k clients (and
clients.keys grew without problem). However, when I try to add a new agent
through authd... I get the same internal error problem.
Results of commands:
$ cat /var/ossec/etc/client.keys | wc -l
2032
$ cat /var/ossec/etc/client.keys | grep -P "^\d+\s*\!" -v | wc -l
209
$ cat /var/ossec/etc/client.keys | grep -P "^\d+\s*\!" | wc -l
1823
On Mon, May 22, 2017 at 6:28 PM, Jesus Linares wrote:
> Hi,
>
> as you mentioned, it seems that inactive agents are counting for the limit
> (2048 agents). Run the following commands in order to know the size of the
> *client.keys
> *file:
>
>- Total lines: cat /var/ossec/etc/client.keys | wc -l
>- Active agents: cat /var/ossec/etc/client.keys | grep -P "^\d+\s*\!"
>-v | wc -l
>- Inactive agents: cat /var/ossec/etc/client.keys | grep -P
>"^\d+\s*\!" | wc -l
>
> The solution could be clean the client.keys (lines with "!") after
> removing the agent.
>
> Regards.
>
>
> On Monday, May 22, 2017 at 11:05:38 AM UTC+2, Topper Bowers wrote:
>>
>> Hi,
>>
>> My client has a highly dynamic environment and we're using OSSEC (wazuh
>> 1.1.1 release, OSSEC v2.8). When a server spins up, it registers itself as
>> an agent to the servers authd and everything was going ok. However, my
>> client.keys file is now 2048 lines long and no new agents can register.
>> They get an "(internal error)" that we see in the /var/ossec/logs/ossec.log
>>
>> We have a process in place to remove inactive agents using the
>> `/var/ossec/bin/manage_agents -r ${ossec_id}` command. And if you use
>> /var/ossec/bin/manage_agents -l only about 100 agents show up.
>>
>> I've seen this https://groups.google.com/forum/#!topic/ossec-list/lgFD
>> OlR6zNg and it looks remarkably similar to what we're seeing. However,
>> we don't actually have thousands of active agents. It seems like inactive
>> agents are counting against the limit. Since we have a really dynamic
>> environment with servers going up and down all the time, increasing the
>> limit seems like it's just pushing out the inevitable.
>>
>> In summary... dynamic environment, can't add new agents, only 100 or so
>> active agents, 2048 lines in client.keys. No other error messages besides
>> "internal error"
>>
>> Any suggestions?
>>
>> Thanks!
>>
>> Topper
>>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "ossec-list" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/ossec-list/k_MFr5aAjRU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> ossec-list+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
*Topper Bowers*
*Engineering*
*Vitals* | 160 Chubb Ave, Suite 301, Lyndhurst, NJ 07071, USA
M : 646.515.6630
http://www.vitals.com
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[ossec-list] problems registering agents
Hi,
My client has a highly dynamic environment and we're using OSSEC (wazuh
1.1.1 release, OSSEC v2.8). When a server spins up, it registers itself as
an agent to the servers authd and everything was going ok. However, my
client.keys file is now 2048 lines long and no new agents can register.
They get an "(internal error)" that we see in the /var/ossec/logs/ossec.log
We have a process in place to remove inactive agents using the
`/var/ossec/bin/manage_agents -r ${ossec_id}` command. And if you use
/var/ossec/bin/manage_agents -l only about 100 agents show up.
I've seen
this https://groups.google.com/forum/#!topic/ossec-list/lgFDOlR6zNg and it
looks remarkably similar to what we're seeing. However, we don't actually
have thousands of active agents. It seems like inactive agents are counting
against the limit. Since we have a really dynamic environment with servers
going up and down all the time, increasing the limit seems like it's just
pushing out the inevitable.
In summary... dynamic environment, can't add new agents, only 100 or so
active agents, 2048 lines in client.keys. No other error messages besides
"internal error"
Any suggestions?
Thanks!
Topper
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[ossec-list] Re: Multi server ossec cluster with shared NFS
Old thread. Did it end up working out? We're having trouble with the sockets being on NFS even just restarting ossec on the same host (let alone on 5). On Tuesday, June 24, 2014 at 6:17:52 PM UTC+2, Roy Feintuch wrote: > > Just saw this thread and wish to add my 2 cents: > - Syscheck: there is a state that is in both memory and file system > regarding the agents that finished creating the initial baseline and are > ready. I suspect it might not trigger FIM alerts for new agents. > - Complex events (correlation). I'm not sure here but think there might be > some state in the servers' memory. Does anyone have idea on that? > - Rids - as Michael said, it would be best to get rid of the rids check in > this setup. > > Cheers, > Roy > > > Anyway, if you have the opportunity to use some stickiness / consistent > hashing so each client would be served by the same server, it would > probably solve all of that. > > > > On Thursday, November 14, 2013 7:55:11 AM UTC-8, Juan Berner wrote: >> >> Hi, I have 5 servers sharing the same NFS folder for /var/ossec, and it >> seems to be working. I've inherited this architecture. >> >> Right now, we have about 3000 clients that connect to an F5 vip, and then >> each client reports to this VIP. In the vip are 5 servers sharing the same >> /var/ossec nfs folder. >> >> My question is, does this architecture work? I mean, Im having issues >> with some clients not connecting and I'm not sure that the correlation >> would work properly, it depends if all the ossec correlation reads always >> from disk and does not save information to memory. >> >> This is a good setup to have HA. >> >> Thanks! >> > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ossec-list] /var/ossec on an NFS share
Hello! I'm trying to institute some automatic failover for OSSEC and we're using amazon's elastic files share as the persistence layer for ossec. Right now we have the whole /var/ossec directory symlinked to a directory that exists on the NFS mount. When we restart the ossec service on the same host, remoted fails to come up... with this in the logs: 2016/12/05 09:33:37 ossec-remoted(1210): ERROR: Queue '/queue/ossec/queue' not accessible: 'Connection refused'. 2016/12/05 09:33:37 ossec-remoted(1211): ERROR: Unable to access queue: '/queue/ossec/queue'. Giving up.. If we delete /var/ossec/queue/ossec/queue and queue/alerts/ar then ossec remoted comes up fine and can access the sockets. Is this a problem with sockets being on an NFS mount? Any suggestions on how to institute automatic failover for the ossec manager? -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ossec-list] trouble getting agent to connect to host
Thanks! Ok... so I turned off the counters and I get the same problem... agent doesn't see response from manager. This time on the agent, I turned on tcpdump: ``` tcpdump -n src host and dst portrange 4501-65000 ``` Then when I received a reply from the manager, I immediately did an lsof -i : . Through that I confirmed that the agent was actually listening on the port with an output like: ``` [root@host ~]# lsof -i :60884 COMMAND PID USER FD TYPEDEVICE SIZE/OFF NODE NAME ossec-age pi ossec7u IPv4 627449120 0t0 UDP ip-man-ip-addr-ad.ec2.internal:60884->ip-this-host-ip-address.ec2.internal:fujitsu-dtcns ``` When I did the strace the grep of IP_MANAGER produced no results... but looking through the strace output I see some... ``` recvfrom(7, 0x7fffecdd6850, 6144, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable) ``` There's a sendto right before that, but it's binary and I'm unsure of how sensitive that is to put on a mailing list. Topper On Wednesday, October 26, 2016 at 2:10:24 PM UTC+2, Pedro S wrote: > > For sure that ACK "HC_STARTUP" is not reaching the agent and that is why > it does not connect. > So the manager is sending the startup and it has connectivity with the > agent host but not with the agent software. > > Confirm if the agent is listening at the right port, also you can use > "strace" at agentd binary and inspect kernel calls to see what is > happening, it will be something like: > > $ strace -ff -o log -s 2 -p >> where is the pid from ossec-agentd >> and then >> $ tail -f log.XXXX | grep IP_MANAGER > > > > Regards, > > Pedro S. > > On Wed, Oct 26, 2016 at 1:45 PM, Topper Bowers > wrote: > >> Thanks for the reply! I will try the counter thing right now. The manager >> has this in the logs: >> >> DEBUG: Agent my-hostname sent HC_STARTUP from xx.xx.xx.xx >> >> So... I think that means it is receiving it on the right port. >> >> I've tried redoing agent-auth a few times on that host now. I've also >> reinstalled ossec-agent once. >> >> On Wednesday, October 26, 2016 at 1:06:11 PM UTC+2, Pedro S wrote: >>> >>> Seems like the agent is waiting for the ACK (HC_ACK) control message but >>> it is not receiving it (start_agent.c >>> <https://github.com/wazuh/ossec-wazuh/blob/cb5c736b1ea053b5ccff888286460c93f99003ab/src/client-agent/start_agent.c#L229>), >>> >>> few things you can try: >>> >>> - Disable counters on both sides, manager and agent (internal_options: r >>> emoted.verify_msg_id=0) >>> - I can see how you are using a non default port 4214, verify it is UDP >>> and Manager is listening to that port and agent is sending to that port. >>> - Verify/add/generate a new key for the Agent. >>> >>> >>> Regards, >>> >>> snaow. >>> >>> On Wed, Oct 26, 2016 at 11:59 AM, Topper Bowers >>> wrote: >>> >>>> Hello all, >>>> >>>> I'm using ossec 2.8.3 from wazzuh and I can't seem to get the agents to >>>> talk to the host. It is exactly as described here: >>>> https://botbot.me/freenode/ossec/2016-07-21/?msg=70001778&page=1. >>>> >>>> I've also put both the agent and the master into debug mode. I've also >>>> run tcpdump on both the agent and the master... I see traffic flowing from >>>> the agent to the master and I see the master responding to the agent... >>>> but >>>> the agent just consistently says: >>>> >>>> 2016/10/26 09:56:24 ossec-agentd: INFO: Trying to connect to server >>>> (X.X.X.X:4214). >>>> >>>> 2016/10/26 09:56:24 ossec-agentd: INFO: Using IPv4 for: X.X.X.X . >>>> >>>> 2016/10/26 09:56:45 ossec-agentd(1234): WARN: Waiting for server reply >>>> (not started). Tried: 'X.X.X.X'. >>>> >>>> >>>> (I removed IP address) >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "ossec-list" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to ossec-list+...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ossec-list" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to ossec-list+...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ossec-list] trouble getting agent to connect to host
Thanks for the reply! I will try the counter thing right now. The manager has this in the logs: DEBUG: Agent my-hostname sent HC_STARTUP from xx.xx.xx.xx So... I think that means it is receiving it on the right port. I've tried redoing agent-auth a few times on that host now. I've also reinstalled ossec-agent once. On Wednesday, October 26, 2016 at 1:06:11 PM UTC+2, Pedro S wrote: > > Seems like the agent is waiting for the ACK (HC_ACK) control message but > it is not receiving it (start_agent.c > <https://github.com/wazuh/ossec-wazuh/blob/cb5c736b1ea053b5ccff888286460c93f99003ab/src/client-agent/start_agent.c#L229>), > > few things you can try: > > - Disable counters on both sides, manager and agent (internal_options: r > emoted.verify_msg_id=0) > - I can see how you are using a non default port 4214, verify it is UDP > and Manager is listening to that port and agent is sending to that port. > - Verify/add/generate a new key for the Agent. > > > Regards, > > snaow. > > On Wed, Oct 26, 2016 at 11:59 AM, Topper Bowers > wrote: > >> Hello all, >> >> I'm using ossec 2.8.3 from wazzuh and I can't seem to get the agents to >> talk to the host. It is exactly as described here: >> https://botbot.me/freenode/ossec/2016-07-21/?msg=70001778&page=1. >> >> I've also put both the agent and the master into debug mode. I've also >> run tcpdump on both the agent and the master... I see traffic flowing from >> the agent to the master and I see the master responding to the agent... but >> the agent just consistently says: >> >> 2016/10/26 09:56:24 ossec-agentd: INFO: Trying to connect to server >> (X.X.X.X:4214). >> >> 2016/10/26 09:56:24 ossec-agentd: INFO: Using IPv4 for: X.X.X.X . >> >> 2016/10/26 09:56:45 ossec-agentd(1234): WARN: Waiting for server reply >> (not started). Tried: 'X.X.X.X'. >> >> >> (I removed IP address) >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ossec-list" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to ossec-list+...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ossec-list] trouble getting agent to connect to host
The agent (being the ossec software itself) does not... but a tcdump watching UDP sees the packets arriving at the agent. I also tried turning off iptables altogether to the same results. On Wednesday, October 26, 2016 at 1:27:05 PM UTC+2, dan (ddpbsd) wrote: > > On Wed, Oct 26, 2016 at 5:59 AM, Topper Bowers > > wrote: > > Hello all, > > > > I'm using ossec 2.8.3 from wazzuh and I can't seem to get the agents to > talk > > to the host. It is exactly as described here: > > https://botbot.me/freenode/ossec/2016-07-21/?msg=70001778&page=1. > > > > I've also put both the agent and the master into debug mode. I've also > run > > tcpdump on both the agent and the master... I see traffic flowing from > the > > agent to the master and I see the master responding to the agent... but > the > > agent just consistently says: > > > > Does the agent see the replies from the server? > > > 2016/10/26 09:56:24 ossec-agentd: INFO: Trying to connect to server > > (X.X.X.X:4214). > > > > 2016/10/26 09:56:24 ossec-agentd: INFO: Using IPv4 for: X.X.X.X . > > > > 2016/10/26 09:56:45 ossec-agentd(1234): WARN: Waiting for server reply > (not > > started). Tried: 'X.X.X.X'. > > > > > > (I removed IP address) > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups > > "ossec-list" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to ossec-list+...@googlegroups.com . > > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ossec-list] trouble getting agent to connect to host
Hello all, I'm using ossec 2.8.3 from wazzuh and I can't seem to get the agents to talk to the host. It is exactly as described here: https://botbot.me/freenode/ossec/2016-07-21/?msg=70001778&page=1. I've also put both the agent and the master into debug mode. I've also run tcpdump on both the agent and the master... I see traffic flowing from the agent to the master and I see the master responding to the agent... but the agent just consistently says: 2016/10/26 09:56:24 ossec-agentd: INFO: Trying to connect to server (X.X.X.X:4214). 2016/10/26 09:56:24 ossec-agentd: INFO: Using IPv4 for: X.X.X.X . 2016/10/26 09:56:45 ossec-agentd(1234): WARN: Waiting for server reply (not started). Tried: 'X.X.X.X'. (I removed IP address) -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
