On Tue, 29 Mar 2016, Giulio Vaccari wrote:
Hi everybody
I have just start using Rsyslog because I need to send the access_log of
apache to a collector.
By the way, I want to send them in Json format so, in this way, the
database can see any field without any sort of post processing (or at
least I hope!).
First, I'm using centos7 with Rsyslog v 7.7.4 and centos6 with Rsyslog
v 5.8.10.
I know that those version are outdated but I can't install or update
nothing on the server, I must use them and that's all! :\
I have alredy convert the standard log file in Json format and they are
correct (tested using a Json validator)
(as example)
{ "host": "192.168.122.1", "user": "-", "timestamp":
"25-03-201609:44:47+0100", "protocol": "HTTP/1.1", "method": "GET",
"alive": "4", "urlpath": "/noindex/css/fonts/Bold/OpenSans-Bold.ttf",
"urlquery": "", "status":" 404", "bytes": "238", "header":
"http://192.168.122.20/noindex/css/open-sans.css";, "useragent":
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
Firefox/45.0", "duration": "331", "connection": "+", "bReceived": "393",
"bSent": "453", "firstLine": "GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1", "vhost":
"centos7_test.example.com" }
As a test I have try to copy the custom log and copy in the same machine
where I have installed the collector and I have added them using the
file input option... Simply perfect...
Now i want send them to the remote collector (Logstash) using rsyslog
and the imfile protocol:
input(type="imfile"
File="/var/log/httpd/logstash_access_log" <-- (custom log folder)
Tag="apache"
StateFile="statefile1")
if $programname == 'apache' then {
action(
type="omfwd"
Target="192.168.122.32"
Port="5514"
Protocol="tcp"
)
stop
}
This syntax doesn't work with rsyslog v5.x
Logstash can receive the message but it give me a jsonparsefailure (so
for the program that isn't a json)
(as example)
"message" => "<133>Mar 25 17:10:49 centos7_test apache
192.168.122.1 - - [25/Mar/2016:17:10:48 +0100] \"GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238
\"http://192.168.122.20/noindex/css/open-sans.css\"; \"Mozilla/5.0 (X11;
Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0\"",
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"@timestamp" => "2016-03-25T16:10:50.544Z",
"host" => "192.168.122.20",
"port" => 52698,
"type" => "apache"
}
Looks like the message is not formatted in json anymore! Why? Rsyslog
can't import formatted file?
there are a lot of things that current versions of rsyslog can do that ancient
(5+ year old) versions of rsyslog can't do.
but in this case, you are telling it to write out in a standard format that is
not json, but telling logstash to read the file as if it were all json, the
writer and reader need to agree on the format of a file.
But why are you having Apache write the log to a file, then having rsyslog read
the file and write to a different file for logstash to read and then deliver to
ElasticSearch? It would seem that you should cut out one of the items in this
chain. Either have logstash read the file from Apache, or have rsyslog deliver
the log into ElasticSearch.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
