rgoers commented on a change in pull request #335: Import of LogstashLayout as
JsonTemplateLayout
URL: https://github.com/apache/logging-log4j2/pull/335#discussion_r403825698
##########
File path: src/site/markdown/manual/cloud.md
##########
@@ -118,76 +118,69 @@ being configured with a list of hosts and ports so high
availability is not an i
-## <a name="ELK"></a>Logging using ElasticSearch, Logstash, and Kibana
+## <a name="ELK"></a>Logging using Elasticsearch, Logstash, and Kibana
-The following configurations have been tested with an ELK stack and are known
to work.
+There are various approaches with different trade-offs for ingesting logs into
+an ELK stack. Here we will briefly cover how one can forward Log4j generated
+events first to Logstash and then to Elasticsearch.
### Log4j Configuration
-Use a socket appender with the GELF layout. Note that if the host name used by
the socket appender has more than
-one ip address associated with its DNS entry the socket appender will fail
through them all if needed.
-
- <Socket name="Elastic" host="${sys:elastic.search.host}" port="12222"
protocol="tcp" bufferedIo="true">
- <GelfLayout includeStackTrace="true" host="${hostName}"
includeThreadContext="true" includeNullDelimiter="true"
- compressionType="OFF">
-
<ThreadContextIncludes>requestId,sessionId,loginId,userId,ipAddress,callingHost</ThreadContextIncludes>
- <MessagePattern>%d [%t] %-5p %X{requestId, sessionId, loginId, userId,
ipAddress} %C{1.}.%M:%L - %m%n</MessagePattern>
- <KeyValuePair key="containerId" value="${docker:containerId:-}"/>
- <KeyValuePair key="application"
value="$${lower:${spring:spring.application.name:-spring}}"/>
- <KeyValuePair key="kubernetes.serviceAccountName"
value="${k8s:accountName:-}"/>
- <KeyValuePair key="kubernetes.containerId"
value="${k8s:containerId:-}"/>
- <KeyValuePair key="kubernetes.containerName"
value="${k8s:containerName:-}"/>
- <KeyValuePair key="kubernetes.host" value="${k8s:host:-}"/>
- <KeyValuePair key="kubernetes.labels.app" value="${k8s:labels.app:-}"/>
- <KeyValuePair key="kubernetes.labels.pod-template-hash"
value="${k8s:labels.podTemplateHash:-}"/>
- <KeyValuePair key="kubernetes.master_url" value="${k8s:masterUrl:-}"/>
- <KeyValuePair key="kubernetes.namespaceId"
value="${k8s:namespaceId:-}"/>
- <KeyValuePair key="kubernetes.namespaceName"
value="${k8s:namespaceName:-}"/>
- <KeyValuePair key="kubernetes.podID" value="${k8s:podId:-}"/>
- <KeyValuePair key="kubernetes.podIP" value="${k8s:podIp:-}"/>
- <KeyValuePair key="kubernetes.podName" value="${k8s:podName:-}"/>
- <KeyValuePair key="kubernetes.imageId" value="${k8s:imageId:-}"/>
- <KeyValuePair key="kubernetes.imageName" value="${k8s:imageName:-}"/>
- </GelfLayout>
+
+Log4j provides a multitude of JSON generating layouts. In particular, [JSON
+Template Layout](layouts.html#JSONTemplateLayout) allows full schema
+customization and bundles ELK-specific layouts by default, which makes it a
+great fit for the bill.
+
+ <Socket name="Logstash"
+ host="${sys:logstash.host}"
+ port="12345"
+ protocol="tcp"
+ bufferedIo="true">
+ <JsonTemplateLayout eventTemplateUri="classpath:EcsLayout.json">
+ <EventTemplateAdditionalFields>
+ <KeyValuePair key="containerId"
value="${docker:containerId:-}"/>
+ <KeyValuePair key="application"
value="$${lower:${spring:spring.application.name:-spring}}"/>
+ <KeyValuePair key="kubernetes.serviceAccountName"
value="${k8s:accountName:-}"/>
+ <KeyValuePair key="kubernetes.containerId"
value="${k8s:containerId:-}"/>
+ <KeyValuePair key="kubernetes.containerName"
value="${k8s:containerName:-}"/>
+ <KeyValuePair key="kubernetes.host" value="${k8s:host:-}"/>
+ <KeyValuePair key="kubernetes.labels.app"
value="${k8s:labels.app:-}"/>
+ <KeyValuePair key="kubernetes.labels.pod-template-hash"
value="${k8s:labels.podTemplateHash:-}"/>
+ <KeyValuePair key="kubernetes.master_url"
value="${k8s:masterUrl:-}"/>
+ <KeyValuePair key="kubernetes.namespaceId"
value="${k8s:namespaceId:-}"/>
+ <KeyValuePair key="kubernetes.namespaceName"
value="${k8s:namespaceName:-}"/>
+ <KeyValuePair key="kubernetes.podID" value="${k8s:podId:-}"/>
+ <KeyValuePair key="kubernetes.podIP" value="${k8s:podIp:-}"/>
+ <KeyValuePair key="kubernetes.podName"
value="${k8s:podName:-}"/>
+ <KeyValuePair key="kubernetes.imageId"
value="${k8s:imageId:-}"/>
+ <KeyValuePair key="kubernetes.imageName"
value="${k8s:imageName:-}"/>
+ </EventTemplateAdditionalFields>
+ </JsonTemplateLayout>
</Socket>
### Logstash Configuration
- input {
- gelf {
- host => "localhost"
- use_tcp => true
- use_udp => false
- port => 12222
- type => "gelf"
- }
- }
+We will configure Logstash to listen on TCP port 12345 for payloads of type
JSON
+and then forward these to (either console and/or) an Elasticsearch server.
Review comment:
In my testing gelf has to be configured so that it uses null to delimit log
events, otherwise events that contain exceptions get split into two log events.
Did you do something magical to make the tcp input with JSON suddenly behave
properly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
