Re: [graylog2] Re: hostname missing in logs received from syslog-ng
Thanks a lot Richard! Tried as you suggested and hostname is now back in
the message!
On Wed, Jan 18, 2017 at 12:32 AM, Richard S. Westmoreland <
richar...@gmail.com> wrote:
> Instead of syslog("graylog.lab.xxx.xxx" port(5514)); try:
>
> network("graylog.lab.xxx.xxx" port(5514));
>
> This used to be udp() or tcp() depending on the transport you wanted.
>
> Newer versions of syslog-ng documentation have emphasized IETF/RFC5424
> examples, but what you are looking for is the older BSD/RFC3164 formatted
> message.
>
>
> > On Jan 18, 2017, at 12:30 PM, Li Li <lita...@gmail.com> wrote:
> >
> > syslog("graylog.lab.xxx.xxx" port(5514));
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/graylog2/I2VNhU3sFcc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/F7E70BCB-C248-4DA8-AB1B-B0C8A61533BB%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAN-mCbMQT83G8v3abbeN4GShy%2BE%3D%3D%3DB%2B-5Ym-5Yk%3DDhbqtAfwA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: hostname missing in logs received from syslog-ng
Sure.
==below is my syslog-ng.conf file=
# cat syslog-ng.conf
@version:3.7
@include "scl.conf"
# syslog-ng configuration file.
#
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
#
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
#
# Note: it also sources additional configuration files (*.conf)
# located in /etc/syslog-ng/conf.d/
options {
flush_lines (0);
time_reopen (10);
log_fifo_size (1000);
chain_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
};
source s_sys {
system();
internal();
# udp(ip(0.0.0.0) port(514));
};
destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog" flush_lines(10)); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_kern { file("/var/log/kern"); };
destination d_mlal { usertty("*"); };
filter f_kernel { facility(kern); };
filter f_default{ level(info..emerg) and
not (facility(mail)
or facility(authpriv)
or facility(cron)); };
filter f_auth { facility(authpriv); };
filter f_mail { facility(mail); };
filter f_emergency { level(emerg); };
filter f_news { facility(uucp) or
(facility(news)
and level(crit..emerg)); };
filter f_boot { facility(local7); };
filter f_cron { facility(cron); };
#log { source(s_sys); filter(f_kernel); destination(d_cons); };
log { source(s_sys); filter(f_kernel); destination(d_kern); };
log { source(s_sys); filter(f_default); destination(d_mesg); };
log { source(s_sys); filter(f_auth); destination(d_auth); };
log { source(s_sys); filter(f_mail); destination(d_mail); };
log { source(s_sys); filter(f_emergency); destination(d_mlal); };
log { source(s_sys); filter(f_news); destination(d_spol); };
log { source(s_sys); filter(f_boot); destination(d_boot); };
log { source(s_sys); filter(f_cron); destination(d_cron); };
# Source additional configuration files (.conf extension only)
@include "/etc/syslog-ng/conf.d/*.conf"
# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
[root@syslogng syslog-ng]#
==below is my graylog.conf =
cat graylog.conf
# Define TCP syslog destination.
source s_net {
#udp(ip(0.0.0.0) port(514));
tcp(ip(0.0.0.0) port(5514));
};
destination d_lab_graylog {
syslog("graylog.lab.xxx.xxx" port(5514));
};
# Tell syslog-ng to send data from source s_src to the newly defined syslog
destination.
log {
source(s_net); # Defined in the default syslog-ng configuration.
destination(d_lab_graylog);
};
On Tuesday, January 17, 2017 at 9:19:59 PM UTC-6, Richard S. Westmoreland
wrote:
>
> Can you paste your filter, destination and log statement for sending to
> graylog?
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/2b3e1196-5f77-4366-8896-c74a9ae945be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: hostname missing in logs received from syslog-ng
Sure.
==below is my syslog-ng.conf file=
# cat syslog-ng.conf
@version:3.7
@include "scl.conf"
# syslog-ng configuration file.
#
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
#
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
#
# Note: it also sources additional configuration files (*.conf)
# located in /etc/syslog-ng/conf.d/
options {
flush_lines (0);
time_reopen (10);
log_fifo_size (1000);
chain_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
};
source s_sys {
system();
internal();
# udp(ip(0.0.0.0) port(514));
};
destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog" flush_lines(10)); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_kern { file("/var/log/kern"); };
destination d_mlal { usertty("*"); };
filter f_kernel { facility(kern); };
filter f_default{ level(info..emerg) and
not (facility(mail)
or facility(authpriv)
or facility(cron)); };
filter f_auth { facility(authpriv); };
filter f_mail { facility(mail); };
filter f_emergency { level(emerg); };
filter f_news { facility(uucp) or
(facility(news)
and level(crit..emerg)); };
filter f_boot { facility(local7); };
filter f_cron { facility(cron); };
#log { source(s_sys); filter(f_kernel); destination(d_cons); };
log { source(s_sys); filter(f_kernel); destination(d_kern); };
log { source(s_sys); filter(f_default); destination(d_mesg); };
log { source(s_sys); filter(f_auth); destination(d_auth); };
log { source(s_sys); filter(f_mail); destination(d_mail); };
log { source(s_sys); filter(f_emergency); destination(d_mlal); };
log { source(s_sys); filter(f_news); destination(d_spol); };
log { source(s_sys); filter(f_boot); destination(d_boot); };
log { source(s_sys); filter(f_cron); destination(d_cron); };
# Source additional configuration files (.conf extension only)
@include "/etc/syslog-ng/conf.d/*.conf"
# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
[root@syslogng syslog-ng]#
==below is my graylog.conf =
cat graylog.conf
# Define TCP syslog destination.
source s_net {
#udp(ip(0.0.0.0) port(514));
tcp(ip(0.0.0.0) port(5514));
};
destination d_lab_graylog {
syslog("graylog.lab.xxx.xxx" port(5514));
};
# Tell syslog-ng to send data from source s_src to the newly defined syslog
destination.
log {
source(s_net); # Defined in the default syslog-ng configuration.
destination(d_lab_graylog);
};
On Tue, Jan 17, 2017 at 9:19 PM, Richard S. Westmoreland <
richar...@gmail.com> wrote:
> Can you paste your filter, destination and log statement for sending to
> graylog?
>
>
> On Jan 18, 2017, at 11:52 AM, Li Li <lita...@gmail.com> wrote:
>
> Hi, Richard,
>
> Thanks for taking your time to look at it! In graylog, below is how my
> message looks like:
>
> message
> 1,2017/01/13 16:58:30,0011C102743,TRAFFIC,end,1,2017/01/13
> 16:58:30,157.142.11.40,10.100.30.205,0.0.0.0,0.0.0.0,Allow all to
> Control,,,ping,vsys1,Untrust,Control,ethernet1/1,ethernet1/2.120,Flume,2017/01/13
> 16:58:30,34899,2,0,0,0,0,0x100019,icmp,allow,120,120,0,2,2017/01/13
> 16:58:20,0,any,0,1478205121,0x0,US,10.0.0.0-10.255.255.
> 255,0,2,0,aged-out,0,0,0,0,,Lab-PA5020,from-policy
>
> As you can see, the date and hostname are missing. When I have syslog-ng
> write to a local file, the date and hostname existed.
>
> Thanks,
> -Li
>
> On Tuesday, January 17, 2017 at 7:51:02 PM UTC-6, Richard S. Westmoreland
> wrote:
>>
>> It shouldn't be missing, just parsed. When you search, do you see the
>> hostname in a separate field? In your Syslog Input there should be an
>> option to keep original message, so then there will be a separate intact
>> copy included. If you don't want the syslog to parse at all you could
>> change to a RAW Input, but then you'd lose the indexing performance
>> advantage for searching on syslog datetime and hostname.
>>
>>
>> On Jan 18, 2017, at 2:53 AM, Li Li <lit...@gmail.com> wrote:
>>
>> Hi Jochen,
>>
>> Thanks for your reply! We were sending our firewall logs directly to
>> graylog through syslog protocol and the messages received do contain the
>> date and the hostname, I wonder why? we are now trying to redesign our log
>&
Re: [graylog2] Re: hostname missing in logs received from syslog-ng
Hi, Richard, Thanks for taking your time to look at it! In graylog, below is how my message looks like: message 1,2017/01/13 16:58:30,0011C102743,TRAFFIC,end,1,2017/01/13 16:58:30,157.142.11.40,10.100.30.205,0.0.0.0,0.0.0.0,Allow all to Control,,,ping,vsys1,Untrust,Control,ethernet1/1,ethernet1/2.120,Flume,2017/01/13 16:58:30,34899,2,0,0,0,0,0x100019,icmp,allow,120,120,0,2,2017/01/13 16:58:20,0,any,0,1478205121,0x0,US,10.0.0.0-10.255.255.255,0,2,0,aged-out,0,0,0,0,,Lab-PA5020,from-policy As you can see, the date and hostname are missing. When I have syslog-ng write to a local file, the date and hostname existed. Thanks, -Li On Tuesday, January 17, 2017 at 7:51:02 PM UTC-6, Richard S. Westmoreland wrote: > > It shouldn't be missing, just parsed. When you search, do you see the > hostname in a separate field? In your Syslog Input there should be an > option to keep original message, so then there will be a separate intact > copy included. If you don't want the syslog to parse at all you could > change to a RAW Input, but then you'd lose the indexing performance > advantage for searching on syslog datetime and hostname. > > > On Jan 18, 2017, at 2:53 AM, Li Li <lit...@gmail.com > wrote: > > Hi Jochen, > > Thanks for your reply! We were sending our firewall logs directly to > graylog through syslog protocol and the messages received do contain the > date and the hostname, I wonder why? we are now trying to redesign our log > solution, we decided to use syslog-ng as a centralized hub to receive logs > from different devices, then relay to graylog and flume, etc. It was at > this point that we discovered the messages in graylog was no longer > containing the date and hostname > > Thanks, > -Li > > On Saturday, January 14, 2017 at 4:39:23 AM UTC-6, Jochen Schalanda wrote: >> >> Hi Li, >> >> Graylog is parsing syslog messages according to the syslog protocol >> standard(s), so it will not repeat the date and the hostname on the start >> of each syslog message but fill the "timestamp" and "source" message fields >> accordingly. >> >> Also see >> https://github.com/Graylog2/graylog-guide-syslog-linux/blob/master/README.md#syslog-ng >> >> for configuration instructions for syslog-ng. >> >> Cheers, >> Jochen >> >> On Friday, 13 January 2017 18:15:40 UTC+1, Li Li wrote: >>> >>> Hi, all, >>> >>> A portion of logs received from syslog-ng is missing, for example, logs >>> entries expected are: >>> >>> Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu 1,2017/01/12 >>> 17:04:21,0011C102743,TRAFFIC,start,1 >>> >>> But in graylog, "Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu >>> 1,2017/01/12" is missing, logs seen in graylog start with >>> "17:04:21,0011C102743,TRAFFIC,start,1" >>> >>> when I have graylog writing to a file, the logs appear to be correct, >>> ie, nothing is missing. >>> >>> My syslog-ng version is 3.7.3, graylog version is 2.0.3. >>> >>> Can anyone give some suggestions? Your help would be greatly appreciated! >>> >>> Thanks, >>> -Li >>> >> -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+u...@googlegroups.com . > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/94b9926b-4ed2-4fbc-a18e-0a5e2918f403%40googlegroups.com > > <https://groups.google.com/d/msgid/graylog2/94b9926b-4ed2-4fbc-a18e-0a5e2918f403%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/641a8713-2022-4eb1-97bf-a3123f3e8ffc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: hostname missing in logs received from syslog-ng
Hi Jochen, Thanks for your reply! We were sending our firewall logs directly to graylog through syslog protocol and the messages received do contain the date and the hostname, I wonder why? we are now trying to redesign our log solution, we decided to use syslog-ng as a centralized hub to receive logs from different devices, then relay to graylog and flume, etc. It was at this point that we discovered the messages in graylog was no longer containing the date and hostname Thanks, -Li On Saturday, January 14, 2017 at 4:39:23 AM UTC-6, Jochen Schalanda wrote: > > Hi Li, > > Graylog is parsing syslog messages according to the syslog protocol > standard(s), so it will not repeat the date and the hostname on the start > of each syslog message but fill the "timestamp" and "source" message fields > accordingly. > > Also see > https://github.com/Graylog2/graylog-guide-syslog-linux/blob/master/README.md#syslog-ng > > for configuration instructions for syslog-ng. > > Cheers, > Jochen > > On Friday, 13 January 2017 18:15:40 UTC+1, Li Li wrote: >> >> Hi, all, >> >> A portion of logs received from syslog-ng is missing, for example, logs >> entries expected are: >> >> Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu 1,2017/01/12 >> 17:04:21,0011C102743,TRAFFIC,start,1 >> >> But in graylog, "Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu >> 1,2017/01/12" is missing, logs seen in graylog start with >> "17:04:21,0011C102743,TRAFFIC,start,1" >> >> when I have graylog writing to a file, the logs appear to be correct, ie, >> nothing is missing. >> >> My syslog-ng version is 3.7.3, graylog version is 2.0.3. >> >> Can anyone give some suggestions? Your help would be greatly appreciated! >> >> Thanks, >> -Li >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/94b9926b-4ed2-4fbc-a18e-0a5e2918f403%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] hostname missing in logs received from syslog-ng
Hi, all, A portion of logs received from syslog-ng is missing, for example, logs entries expected are: Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu 1,2017/01/12 17:04:21,0011C102743,TRAFFIC,start,1 But in graylog, "Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu 1,2017/01/12" is missing, logs seen in graylog start with "17:04:21,0011C102743,TRAFFIC,start,1" when I have graylog writing to a file, the logs appear to be correct, ie, nothing is missing. My syslog-ng version is 3.7.3, graylog version is 2.0.3. Can anyone give some suggestions? Your help would be greatly appreciated! Thanks, -Li -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/baaa281f-5376-4168-8e2c-7a771ca38ce6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
