Re: kvm: /var/log/cloudstack/agent/agent.log is a binary file
> > Wido, makes sense that log4j and logrotate would conflict. Log4j > > has its > > own rotate functionality. > Note that a few CS log files are not generated by log4j. You still need external logrotatation if you don't want them to fill up your disk. I had this issue with access.log, for example. I haven't encountered your "binary" log file issue, however.
Re: kvm: /var/log/cloudstack/agent/agent.log is a binary file
On 9/5/19 11:13 AM, Daan Hoogland wrote: > Wido, makes sense that log4j and logrotate would conflict. Log4j has its > own rotate functionality. It does indeed :-) > I didn't understand before but the binary data is always the beginning of > the file? Is always nulls? I found this out yesterday, it's always the beginning of the file. Not sure what it is though. On two servers I removed /etc/logrotate.d/cloudstack-agent and I'll report back in a few days. Wido > > Op do 5 sep. 2019 09:18 schreef Wei ZHOU : > >> Hi Wido, >> >> I saw this issue in a 4.11.3 platform today. >> >> It seems to be caused by file >> >> https://github.com/apache/cloudstack/blob/master/agent/conf/cloudstack-agent.logrotate.in >> >> Maybe the file /etc/logrotate.d/cloudstack-agent is not needed ( in Ubuntu >> ?). >> >> -Wei >> >> >> >> On Thu, 5 Sep 2019 at 08:08, Wido den Hollander wrote: >> >>> >>> >>> On 9/3/19 2:55 PM, Daan Hoogland wrote: On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander >>> wrote: > > > On 9/3/19 9:57 AM, Daan Hoogland wrote: >> Can you find/look at the line before in the log. It is probably the >> one >> containing the hindering data. Or otherwise it *might* be a clue >> where >>> in >> the flow it happens. >> > > Do you have any idea what the easiest way might be? > In short, no, but.. in totally skewed order of alleged quickness 1: I'm afraid it is a line by line analysis. If you are lucky the line start with date and class name is not obscured and there is a newline entered in the log. That would give you the exact location. when in bad luck the binary data contains control chars that erase to beginning of >>> line or so and the best you can find is the previous and next lines. 3: code analysis; My first thought is that some binary data is read >> from >>> a file or a socket and logged as is. but starting this with code analysis >>> is not the quickest way to get to the culprit I imagine. 2: Other than line by line log analysis and if you can reproduce the >>> issue, you could try zooming in by playing around with log levels for >> different classes. >>> >>> I checked, but on all hypervisors I checked the "agent.log" contains >>> binary data, but all the rotated (.gz) files do not. >>> >>> The binary data is at the beginning of the file(s). So it seems like it >>> is logrotate and maybe a combination of log4j causing this? >>> >>> I'm not sure yet, but that seems to be the case. >>> >>> Wido >>> > > I checked agent.log.1.gz and after I decompress is that file is not > binary, text only. > so much for reproducibilty. Hope it was not a buffer overflow hack >>> attempt. > > The binary data only seems to be present in the current file. > > Wido > >> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander > wrote: >> >>> >>> >>> On 9/2/19 10:18 PM, Wei ZHOU wrote: Hi Wido, I had similar issue (not agent.log). It is probably caused by one >> or >>> few> lines with special characters. >>> >>> And I'm trying to figure out what causes it :-) >>> "grep -a" should work. >>> >>> I know, but other tools which analyze the logfile might also have >>> trouble reading the file due to binary characters. >>> >>> Wido >>> -Wei On Mon, 2 Sep 2019 at 19:35, Wido den Hollander > wrote: > Hi, > > I've seen this on multiple occasions with Ubuntu 18.04 (and maybe > 16.04?) hypervisors where according to 'grep' the agent.log is a > binary > file: > > root@n06:~# grep security_group >> /var/log/cloudstack/agent/agent.log > Binary file /var/log/cloudstack/agent/agent.log matches > root@n06:~# > > If I open the file with 'less' I indeed see binary data. > > Tailing the file works just fine. > > Does anybody know where this is coming from? What in the >> CloudStack > agent causes it to write binary data to the logfile? > > Wido > >>> >>> >>> >> > >>> >> >
Re: kvm: /var/log/cloudstack/agent/agent.log is a binary file
Wido, makes sense that log4j and logrotate would conflict. Log4j has its own rotate functionality. I didn't understand before but the binary data is always the beginning of the file? Is always nulls? Op do 5 sep. 2019 09:18 schreef Wei ZHOU : > Hi Wido, > > I saw this issue in a 4.11.3 platform today. > > It seems to be caused by file > > https://github.com/apache/cloudstack/blob/master/agent/conf/cloudstack-agent.logrotate.in > > Maybe the file /etc/logrotate.d/cloudstack-agent is not needed ( in Ubuntu > ?). > > -Wei > > > > On Thu, 5 Sep 2019 at 08:08, Wido den Hollander wrote: > > > > > > > On 9/3/19 2:55 PM, Daan Hoogland wrote: > > > On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander > > wrote: > > > > > >> > > >> > > >> On 9/3/19 9:57 AM, Daan Hoogland wrote: > > >>> Can you find/look at the line before in the log. It is probably the > one > > >>> containing the hindering data. Or otherwise it *might* be a clue > where > > in > > >>> the flow it happens. > > >>> > > >> > > >> Do you have any idea what the easiest way might be? > > >> > > > In short, no, but.. in totally skewed order of alleged quickness > > > 1: I'm afraid it is a line by line analysis. If you are lucky the line > > > start with date and class name is not obscured and there is a newline > > > entered in the log. That would give you the exact location. when in bad > > > luck the binary data contains control chars that erase to beginning of > > line > > > or so and the best you can find is the previous and next lines. > > > 3: code analysis; My first thought is that some binary data is read > from > > a > > > file or a socket and logged as is. but starting this with code analysis > > is > > > not the quickest way to get to the culprit I imagine. > > > 2: Other than line by line log analysis and if you can reproduce the > > issue, > > > you could try zooming in by playing around with log levels for > different > > > classes. > > > > > > > I checked, but on all hypervisors I checked the "agent.log" contains > > binary data, but all the rotated (.gz) files do not. > > > > The binary data is at the beginning of the file(s). So it seems like it > > is logrotate and maybe a combination of log4j causing this? > > > > I'm not sure yet, but that seems to be the case. > > > > Wido > > > > > > > >> > > >> I checked agent.log.1.gz and after I decompress is that file is not > > >> binary, text only. > > >> > > > so much for reproducibilty. Hope it was not a buffer overflow hack > > attempt. > > > > > > > > >> > > >> The binary data only seems to be present in the current file. > > >> > > >> Wido > > >> > > >>> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander > > >> wrote: > > >>> > > > > > > On 9/2/19 10:18 PM, Wei ZHOU wrote: > > > Hi Wido, > > > > > > I had similar issue (not agent.log). It is probably caused by one > or > > few> lines with special characters. > > > > And I'm trying to figure out what causes it :-) > > > > > "grep -a" should work. > > > > I know, but other tools which analyze the logfile might also have > > trouble reading the file due to binary characters. > > > > Wido > > > > > > > > -Wei > > > > > > On Mon, 2 Sep 2019 at 19:35, Wido den Hollander > > >> wrote: > > > > > >> Hi, > > >> > > >> I've seen this on multiple occasions with Ubuntu 18.04 (and maybe > > >> 16.04?) hypervisors where according to 'grep' the agent.log is a > > >> binary > > >> file: > > >> > > >> root@n06:~# grep security_group > /var/log/cloudstack/agent/agent.log > > >> Binary file /var/log/cloudstack/agent/agent.log matches > > >> root@n06:~# > > >> > > >> If I open the file with 'less' I indeed see binary data. > > >> > > >> Tailing the file works just fine. > > >> > > >> Does anybody know where this is coming from? What in the > CloudStack > > >> agent causes it to write binary data to the logfile? > > >> > > >> Wido > > >> > > > > > > > > > > > >>> > > >> > > > > > > > > >
Re: kvm: /var/log/cloudstack/agent/agent.log is a binary file
Hi Wido, I saw this issue in a 4.11.3 platform today. It seems to be caused by file https://github.com/apache/cloudstack/blob/master/agent/conf/cloudstack-agent.logrotate.in Maybe the file /etc/logrotate.d/cloudstack-agent is not needed ( in Ubuntu ?). -Wei On Thu, 5 Sep 2019 at 08:08, Wido den Hollander wrote: > > > On 9/3/19 2:55 PM, Daan Hoogland wrote: > > On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander > wrote: > > > >> > >> > >> On 9/3/19 9:57 AM, Daan Hoogland wrote: > >>> Can you find/look at the line before in the log. It is probably the one > >>> containing the hindering data. Or otherwise it *might* be a clue where > in > >>> the flow it happens. > >>> > >> > >> Do you have any idea what the easiest way might be? > >> > > In short, no, but.. in totally skewed order of alleged quickness > > 1: I'm afraid it is a line by line analysis. If you are lucky the line > > start with date and class name is not obscured and there is a newline > > entered in the log. That would give you the exact location. when in bad > > luck the binary data contains control chars that erase to beginning of > line > > or so and the best you can find is the previous and next lines. > > 3: code analysis; My first thought is that some binary data is read from > a > > file or a socket and logged as is. but starting this with code analysis > is > > not the quickest way to get to the culprit I imagine. > > 2: Other than line by line log analysis and if you can reproduce the > issue, > > you could try zooming in by playing around with log levels for different > > classes. > > > > I checked, but on all hypervisors I checked the "agent.log" contains > binary data, but all the rotated (.gz) files do not. > > The binary data is at the beginning of the file(s). So it seems like it > is logrotate and maybe a combination of log4j causing this? > > I'm not sure yet, but that seems to be the case. > > Wido > > > > >> > >> I checked agent.log.1.gz and after I decompress is that file is not > >> binary, text only. > >> > > so much for reproducibilty. Hope it was not a buffer overflow hack > attempt. > > > > > >> > >> The binary data only seems to be present in the current file. > >> > >> Wido > >> > >>> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander > >> wrote: > >>> > > > On 9/2/19 10:18 PM, Wei ZHOU wrote: > > Hi Wido, > > > > I had similar issue (not agent.log). It is probably caused by one or > few> lines with special characters. > > And I'm trying to figure out what causes it :-) > > > "grep -a" should work. > > I know, but other tools which analyze the logfile might also have > trouble reading the file due to binary characters. > > Wido > > > > > -Wei > > > > On Mon, 2 Sep 2019 at 19:35, Wido den Hollander > >> wrote: > > > >> Hi, > >> > >> I've seen this on multiple occasions with Ubuntu 18.04 (and maybe > >> 16.04?) hypervisors where according to 'grep' the agent.log is a > >> binary > >> file: > >> > >> root@n06:~# grep security_group /var/log/cloudstack/agent/agent.log > >> Binary file /var/log/cloudstack/agent/agent.log matches > >> root@n06:~# > >> > >> If I open the file with 'less' I indeed see binary data. > >> > >> Tailing the file works just fine. > >> > >> Does anybody know where this is coming from? What in the CloudStack > >> agent causes it to write binary data to the logfile? > >> > >> Wido > >> > > > > > > >>> > >> > > > > >
Re: kvm: /var/log/cloudstack/agent/agent.log is a binary file
On 9/3/19 2:55 PM, Daan Hoogland wrote: > On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander wrote: > >> >> >> On 9/3/19 9:57 AM, Daan Hoogland wrote: >>> Can you find/look at the line before in the log. It is probably the one >>> containing the hindering data. Or otherwise it *might* be a clue where in >>> the flow it happens. >>> >> >> Do you have any idea what the easiest way might be? >> > In short, no, but.. in totally skewed order of alleged quickness > 1: I'm afraid it is a line by line analysis. If you are lucky the line > start with date and class name is not obscured and there is a newline > entered in the log. That would give you the exact location. when in bad > luck the binary data contains control chars that erase to beginning of line > or so and the best you can find is the previous and next lines. > 3: code analysis; My first thought is that some binary data is read from a > file or a socket and logged as is. but starting this with code analysis is > not the quickest way to get to the culprit I imagine. > 2: Other than line by line log analysis and if you can reproduce the issue, > you could try zooming in by playing around with log levels for different > classes. > I checked, but on all hypervisors I checked the "agent.log" contains binary data, but all the rotated (.gz) files do not. The binary data is at the beginning of the file(s). So it seems like it is logrotate and maybe a combination of log4j causing this? I'm not sure yet, but that seems to be the case. Wido > >> >> I checked agent.log.1.gz and after I decompress is that file is not >> binary, text only. >> > so much for reproducibilty. Hope it was not a buffer overflow hack attempt. > > >> >> The binary data only seems to be present in the current file. >> >> Wido >> >>> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander >> wrote: >>> On 9/2/19 10:18 PM, Wei ZHOU wrote: > Hi Wido, > > I had similar issue (not agent.log). It is probably caused by one or few> lines with special characters. And I'm trying to figure out what causes it :-) > "grep -a" should work. I know, but other tools which analyze the logfile might also have trouble reading the file due to binary characters. Wido > > -Wei > > On Mon, 2 Sep 2019 at 19:35, Wido den Hollander >> wrote: > >> Hi, >> >> I've seen this on multiple occasions with Ubuntu 18.04 (and maybe >> 16.04?) hypervisors where according to 'grep' the agent.log is a >> binary >> file: >> >> root@n06:~# grep security_group /var/log/cloudstack/agent/agent.log >> Binary file /var/log/cloudstack/agent/agent.log matches >> root@n06:~# >> >> If I open the file with 'less' I indeed see binary data. >> >> Tailing the file works just fine. >> >> Does anybody know where this is coming from? What in the CloudStack >> agent causes it to write binary data to the logfile? >> >> Wido >> > >>> >> > >
