[ossec-list] Re: ERROR: Unable to send file 'merged.mg' to agent.
Hi Kat! Yup, seen that happen too. On Wednesday, August 3, 2016 at 6:56:32 AM UTC-7, Kat wrote: > > One thing to also check is permissions and ownership on "merged.mg" - > many times I see it get mucked up and OSSEC can't read it. I have found > that if I delete it, then restart OSSEC it will be re-created and it no > longer has issues sending the file after that. (Not sure WHY it happens > though) > > Cheers > Kat > > (PS - Hi Graeme!) > > On Thursday, July 28, 2016 at 11:43:32 AM UTC-5, Graeme Stewart wrote: >> >> Seeing a lot of errors in the logfiles like this: >> >> 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file 'merged.mg' >> to agent. >> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted >> message. >> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg' >> to agent. >> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted >> message. >> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg' >> to agent. >> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted >> message. >> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg' >> to agent. >> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted >> message. >> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg' >> to agent. >> 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating encrypted >> message. >> 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file 'merged.mg' >> to agent. >> 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating encrypted >> message. >> >> Any guidance on troubleshooting? Search hasn't turned up much other than >> delete merged.mg and restart (which we've tried to no success)... >> > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ossec-list] Re: ERROR: Unable to send file 'merged.mg' to agent.
One thing to also check is permissions and ownership on "merged.mg" - many times I see it get mucked up and OSSEC can't read it. I have found that if I delete it, then restart OSSEC it will be re-created and it no longer has issues sending the file after that. (Not sure WHY it happens though) Cheers Kat (PS - Hi Graeme!) On Thursday, July 28, 2016 at 11:43:32 AM UTC-5, Graeme Stewart wrote: > > Seeing a lot of errors in the logfiles like this: > > 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file 'merged.mg' > to agent. > 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted > message. > 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg' > to agent. > 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted > message. > 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg' > to agent. > 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted > message. > 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg' > to agent. > 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted > message. > 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg' > to agent. > 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating encrypted > message. > 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file 'merged.mg' > to agent. > 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating encrypted > message. > > Any guidance on troubleshooting? Search hasn't turned up much other than > delete merged.mg and restart (which we've tried to no success)... > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ossec-list] Re: ERROR: Unable to send file 'merged.mg' to agent.
Awesome! Many thanks, this is exactly what I was looking for.
On Friday, July 29, 2016 at 12:16:35 PM UTC-7, Victor Fernandez wrote:
>
> Hi Graeme.
>
> I agree, it would be great to print on the log that the agent became
> disconnected. The SEC_ERROR definition is shared between manager and
> agents, but it's possible to extend some other messages. In fact, the line
> at sendmsg.c that tests if the agent is disconnected (more than 20
> minutes since the last keep-alive) is the only one that doesn't log an
> error.
>
> I did some modifications at the Wazuh repository, maybe it's useful to you:
>
>
> https://github.com/wazuh/ossec-wazuh/commit/efbd5c17cc3ea5109ea978208b11da6c98fa3083
>
> See below an example of the new log format for the error:
>
> 2016/07/29 11:43:57 ossec-remoted(1245): ERROR: Sending message to
> disconnected agent '001'.
> 2016/07/29 11:43:57 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
> 2016/07/29 11:43:57 ossec-remoted(1246): ERROR: Unable to send file '
> merged.mg' to agent '001' (centos).
>
> I hope this leads you to find the problem.
>
> Kind regards.
>
>
> On Friday, July 29, 2016 at 8:19:56 AM UTC-7, Graeme Stewart wrote:
>>
>> Hi Victor,
>>
>> Huge thanks for the detail, this would explain exactly why we're seeing
>> this; our OSSEC managers are likely overloaded.
>>
>> It would be very helpful to include the agentid in the logfile to
>> understand / track where this is occurring and the number of unique agents
>> that are impacted, perhaps something like:
>>
>> From: src/error_messages/error_messages.h
>> #define SEC_ERROR "%s(1217): ERROR: Error creating encrypted
>> message for: '%s')."
>>
>> Then inside: src/remoted/sendmsg.c
>> msg_size = CreateSecMSG(, msg, crypt_msg, agentid);
>> if (msg_size == 0) {
>> merror(SEC_ERROR, ARGV0, agentid);
>> return (-1);
>> }
>>
>> The clustered nature of this issue leads me to suspect it's repeating
>> this error in the logfiles multiple times for a connection attempt across
>> only one or two agents.
>>
>> Again, many thanks for the detailed response.
>>
>> Graeme
>>
>> On Thursday, July 28, 2016 at 5:33:29 PM UTC-7, Victor Fernandez wrote:
>>>
>>> Hi Graeme.
>>>
>>> According to the log, I think the problem occurs when the manager tries
>>> to send the merged.mg to an agent that has not sent the keep-alive in
>>> the last 20 minutes. This may happen if a lot of agents get connected, or
>>> send the keep-alive at the same time.
>>>
>>> So, if many agents send a keep-alive, the manager takes more than 20
>>> minutes to send the merged.mg to an agent, and that agent hasn't sent
>>> the keep-alive again, this problem occurs.
>>>
>>> I did some math: the manager sleeps one second every time it sends 27
>>> KB. With a 150 KB merged.mg, OSSEC takes 20 minutes to send the
>>> complete file to about 216 agents.
>>>
>>> The 20-minutes check appears on src/remoted/sendmsg.c:
>>>
>>> /* If we don't have the agent id, ignore it */
>>> if (keys.keyentries[agentid]->rcvd < (time(0) - (2 * NOTIFY_TIME))) {
>>> return (-1);
>>> }
>>>
>>> NOTIFY_TIME is 600 (10 minutes) by default. Nevertheless OSSEC labels an
>>> agent as disconnected when it hasn't send the keep-alive in the last 30:30
>>> minutes, as we can see at src/shared/read-agents.c:
>>>
>>> if (file_status.st_mtime > (time(0) - (3 * NOTIFY_TIME + 30))) {
>>> return (GA_STATUS_ACTIVE);
>>> }
>>>
>>> Because of this, I think that this may be an issue.
>>>
>>> I think that a good approach would be to check that there aren't alerts
>>> about disconnected agents that connected recently.
>>>
>>> Kind regards.
>>>
>>>
>>> On Thursday, July 28, 2016 at 9:43:32 AM UTC-7, Graeme Stewart wrote:
Seeing a lot of errors in the logfiles like this:
2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file '
merged.mg' to agent.
2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating
encrypted message.
2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file '
merged.mg' to agent.
2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating
encrypted message.
2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file '
merged.mg' to agent.
2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating
encrypted message.
2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file '
merged.mg' to agent.
2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating
encrypted message.
2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file '
merged.mg' to agent.
2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating
encrypted message.
2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file '
merged.mg' to agent.
2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating
encrypted message.
Any guidance on troubleshooting? Search hasn't turned up much
[ossec-list] Re: ERROR: Unable to send file 'merged.mg' to agent.
Hi Graeme.
I agree, it would be great to print on the log that the agent became
disconnected. The SEC_ERROR definition is shared between manager and
agents, but it's possible to extend some other messages. In fact, the line
at sendmsg.c that tests if the agent is disconnected (more than 20 minutes
since the last keep-alive) is the only one that doesn't log an error.
I did some modifications at the Wazuh repository, maybe it's useful to you:
https://github.com/wazuh/ossec-wazuh/commit/efbd5c17cc3ea5109ea978208b11da6c98fa3083
See below an example of the new log format for the error:
2016/07/29 11:43:57 ossec-remoted(1245): ERROR: Sending message to
disconnected agent '001'.
2016/07/29 11:43:57 ossec-remoted(1217): ERROR: Error creating encrypted
message.
2016/07/29 11:43:57 ossec-remoted(1246): ERROR: Unable to send file
'merged.mg' to agent '001' (centos).
I hope this leads you to find the problem.
Kind regards.
On Friday, July 29, 2016 at 8:19:56 AM UTC-7, Graeme Stewart wrote:
>
> Hi Victor,
>
> Huge thanks for the detail, this would explain exactly why we're seeing
> this; our OSSEC managers are likely overloaded.
>
> It would be very helpful to include the agentid in the logfile to
> understand / track where this is occurring and the number of unique agents
> that are impacted, perhaps something like:
>
> From: src/error_messages/error_messages.h
> #define SEC_ERROR "%s(1217): ERROR: Error creating encrypted message
> for: '%s')."
>
> Then inside: src/remoted/sendmsg.c
> msg_size = CreateSecMSG(, msg, crypt_msg, agentid);
> if (msg_size == 0) {
> merror(SEC_ERROR, ARGV0, agentid);
> return (-1);
> }
>
> The clustered nature of this issue leads me to suspect it's repeating this
> error in the logfiles multiple times for a connection attempt across only
> one or two agents.
>
> Again, many thanks for the detailed response.
>
> Graeme
>
> On Thursday, July 28, 2016 at 5:33:29 PM UTC-7, Victor Fernandez wrote:
>>
>> Hi Graeme.
>>
>> According to the log, I think the problem occurs when the manager tries
>> to send the merged.mg to an agent that has not sent the keep-alive in
>> the last 20 minutes. This may happen if a lot of agents get connected, or
>> send the keep-alive at the same time.
>>
>> So, if many agents send a keep-alive, the manager takes more than 20
>> minutes to send the merged.mg to an agent, and that agent hasn't sent
>> the keep-alive again, this problem occurs.
>>
>> I did some math: the manager sleeps one second every time it sends 27 KB.
>> With a 150 KB merged.mg, OSSEC takes 20 minutes to send the complete
>> file to about 216 agents.
>>
>> The 20-minutes check appears on src/remoted/sendmsg.c:
>>
>> /* If we don't have the agent id, ignore it */
>> if (keys.keyentries[agentid]->rcvd < (time(0) - (2 * NOTIFY_TIME))) {
>> return (-1);
>> }
>>
>> NOTIFY_TIME is 600 (10 minutes) by default. Nevertheless OSSEC labels an
>> agent as disconnected when it hasn't send the keep-alive in the last 30:30
>> minutes, as we can see at src/shared/read-agents.c:
>>
>> if (file_status.st_mtime > (time(0) - (3 * NOTIFY_TIME + 30))) {
>> return (GA_STATUS_ACTIVE);
>> }
>>
>> Because of this, I think that this may be an issue.
>>
>> I think that a good approach would be to check that there aren't alerts
>> about disconnected agents that connected recently.
>>
>> Kind regards.
>>
>>
>> On Thursday, July 28, 2016 at 9:43:32 AM UTC-7, Graeme Stewart wrote:
>>>
>>> Seeing a lot of errors in the logfiles like this:
>>>
>>> 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>>> to agent.
>>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted
>>> message.
>>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>>> to agent.
>>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted
>>> message.
>>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>>> to agent.
>>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted
>>> message.
>>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>>> to agent.
>>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted
>>> message.
>>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>>> to agent.
>>> 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating encrypted
>>> message.
>>> 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>>> to agent.
>>> 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating encrypted
>>> message.
>>>
>>> Any guidance on troubleshooting? Search hasn't turned up much other than
>>> delete merged.mg and restart (which we've tried to no success)...
>>>
>>
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[ossec-list] Re: ERROR: Unable to send file 'merged.mg' to agent.
Hi Victor,
Huge thanks for the detail, this would explain exactly why we're seeing
this; our OSSEC managers are likely overloaded.
It would be very helpful to include the agentid in the logfile to
understand / track where this is occurring and the number of unique agents
that are impacted, perhaps something like:
From: src/error_messages/error_messages.h
#define SEC_ERROR "%s(1217): ERROR: Error creating encrypted message
for: '%s')."
Then inside: src/remoted/sendmsg.c
msg_size = CreateSecMSG(, msg, crypt_msg, agentid);
if (msg_size == 0) {
merror(SEC_ERROR, ARGV0, agentid);
return (-1);
}
The clustered nature of this issue leads me to suspect it's repeating this
error in the logfiles multiple times for a connection attempt across only
one or two agents.
Again, many thanks for the detailed response.
Graeme
On Thursday, July 28, 2016 at 5:33:29 PM UTC-7, Victor Fernandez wrote:
>
> Hi Graeme.
>
> According to the log, I think the problem occurs when the manager tries to
> send the merged.mg to an agent that has not sent the keep-alive in the
> last 20 minutes. This may happen if a lot of agents get connected, or send
> the keep-alive at the same time.
>
> So, if many agents send a keep-alive, the manager takes more than 20
> minutes to send the merged.mg to an agent, and that agent hasn't sent the
> keep-alive again, this problem occurs.
>
> I did some math: the manager sleeps one second every time it sends 27 KB.
> With a 150 KB merged.mg, OSSEC takes 20 minutes to send the complete file
> to about 216 agents.
>
> The 20-minutes check appears on src/remoted/sendmsg.c:
>
> /* If we don't have the agent id, ignore it */
> if (keys.keyentries[agentid]->rcvd < (time(0) - (2 * NOTIFY_TIME))) {
> return (-1);
> }
>
> NOTIFY_TIME is 600 (10 minutes) by default. Nevertheless OSSEC labels an
> agent as disconnected when it hasn't send the keep-alive in the last 30:30
> minutes, as we can see at src/shared/read-agents.c:
>
> if (file_status.st_mtime > (time(0) - (3 * NOTIFY_TIME + 30))) {
> return (GA_STATUS_ACTIVE);
> }
>
> Because of this, I think that this may be an issue.
>
> I think that a good approach would be to check that there aren't alerts
> about disconnected agents that connected recently.
>
> Kind regards.
>
>
> On Thursday, July 28, 2016 at 9:43:32 AM UTC-7, Graeme Stewart wrote:
>>
>> Seeing a lot of errors in the logfiles like this:
>>
>> 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>> to agent.
>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted
>> message.
>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>> to agent.
>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted
>> message.
>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>> to agent.
>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted
>> message.
>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>> to agent.
>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted
>> message.
>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>> to agent.
>> 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating encrypted
>> message.
>> 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file 'merged.mg'
>> to agent.
>> 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating encrypted
>> message.
>>
>> Any guidance on troubleshooting? Search hasn't turned up much other than
>> delete merged.mg and restart (which we've tried to no success)...
>>
>
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[ossec-list] Re: ERROR: Unable to send file 'merged.mg' to agent.
Hi Graeme.
According to the log, I think the problem occurs when the manager tries to
send the merged.mg to an agent that has not sent the keep-alive in the last
20 minutes. This may happen if a lot of agents get connected, or send the
keep-alive at the same time.
So, if many agents send a keep-alive, the manager takes more than 20
minutes to send the merged.mg to an agent, and that agent hasn't sent the
keep-alive again, this problem occurs.
I did some math: the manager sleeps one second every time it sends 27 KB.
With a 150 KB merged.mg, OSSEC takes 20 minutes to send the complete file
to about 216 agents.
The 20-minutes check appears on src/remoted/sendmsg.c:
/* If we don't have the agent id, ignore it */
if (keys.keyentries[agentid]->rcvd < (time(0) - (2 * NOTIFY_TIME))) {
return (-1);
}
NOTIFY_TIME is 600 (10 minutes) by default. Nevertheless OSSEC labels an
agent as disconnected when it hasn't send the keep-alive in the last 30:30
minutes, as we can see at src/shared/read-agents.c:
if (file_status.st_mtime > (time(0) - (3 * NOTIFY_TIME + 30))) {
return (GA_STATUS_ACTIVE);
}
Because of this, I think that this may be an issue.
I think that a good approach would be to check that there aren't alerts
about disconnected agents that connected recently.
Kind regards.
On Thursday, July 28, 2016 at 9:43:32 AM UTC-7, Graeme Stewart wrote:
>
> Seeing a lot of errors in the logfiles like this:
>
> 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file 'merged.mg'
> to agent.
> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg'
> to agent.
> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg'
> to agent.
> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg'
> to agent.
> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg'
> to agent.
> 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
> 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file 'merged.mg'
> to agent.
> 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
>
> Any guidance on troubleshooting? Search hasn't turned up much other than
> delete merged.mg and restart (which we've tried to no success)...
>
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
