[graylog2] Re: Shortened Xml in message field
Hello Jochen,
I have nospecial extractors running. Maybe I can send you the real (not
anonymized) XML on a private way? To me it seems like it is a special
sequence in that XML that is causing this. I will try to reproduce it first
by myself.
The message that shows the shortened XML is still there.
I found this out because I am now tracking sizes of all messages sent to
Graylog to see big sized messages that should be shortened before sending
them. I investigated further and this one was a very large message but in
my framework internally shortened to 64kb (the full_message field) and ".."
appended. So the XML got cut. Nevertheless the message in graylog is
showing closing tags that were never sent and a full_message field of
~200byte. Very weired because I do no XML processing in the logging
framework as well just plain strings.
Thanks
Robby
Am Mittwoch, 19. Oktober 2016 12:41:16 UTC+2 schrieb Jochen Schalanda:
>
> Hi Robby,
>
> I cannot reproduce your problem.
>
> I've created a GELF UDP input on an installation of Graylog 2.1.1 and sent
> a GELF message to it with the XML example from your first email in the
> full_text message field.
>
> Everything has been ingested and is being shown as intended (i. e.
> unparsed and the verbatim XML). Maybe some extractor is interfering with
> your payload or even the client already sends the modified GELF message.
>
> Graylog and Elasticsearch itself do not parse XML from message fields by
> default.
>
> For reproduction, here's what I did:
>
> Create GELF UDP input with the following configuration and no extractors:
>
> bind_address: 127.0.0.1
> decompress_size_limit: 8388608
> override_source:
> port: 12201
> recv_buffer_size: 16384
>
>
> Send message to Graylog:
>
>
> $ echo '{"version":"1.1", "message":"XML TEST", "full_message":" version=\"1.0\" encoding=\"utf-8\"?>\n \n
> Success\n DoSomething\n
> 8\n \n 0\n \n
>
>
> Message in Graylog:
>
> $ curl -i -u admin:admin
> http://127.0.0.1:9000/api/messages/graylog2_27/43c3aca0-95e7-11e6-8fc7-6c4008b8fc28?pretty=true
> HTTP/1.1 200 OK
> X-Graylog-Node-ID: cd03ee44-b2a7-4824-be16-bb7456149dbd
> X-Runtime-Microseconds: 1887
> Content-Type: application/json
> Date: Wed, 19 Oct 2016 10:38:40 GMT
> Content-Length: 1975
>
> {
> "message" : {
> "fields" : {
> "full_message" : "\n
> \n Success\n DoSomething\n
>8\n \n 0\n \n
> "streams" : [ "552b92b2e4b0c055e41ffb8e", "0001"
> ],
> "gl2_source_node" : "cd03ee44-b2a7-4824-be16-bb7456149dbd",
> "_id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28",
> "source" : "unknown",
> "message" : "XML TEST",
> "gl2_source_input" : "5592913ec026f261b8d5ce9a",
> "timestamp" : "2016-10-19T10:31:58.824Z"
> },
> "streams" : [ ],
> "source_input_id" : null,
> "filter_out" : false,
> "journal_offset" : -9223372036854775808,
> "message" : "XML TEST",
> "id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28",
> "source" : "unknown",
> "field_names" : [ "full_message", "streams", "gl2_source_node", "_id",
> "source", "message", "gl2_source_input", "timestamp" ],
> "complete" : true,
> "timestamp" : "2016-10-19T10:31:58.824Z",
> "validation_errors" : "",
> "fields_entries" : [ {
> "full_message" : "\n
> \n Success\n DoSomething\n
>8\n \n 0\n \n
> }, {
> "streams" : [ "552b92b2e4b0c055e41ffb8e", "0001"
> ]
> }, {
> "gl2_source_node" : "cd03ee44-b2a7-4824-be16-bb7456149dbd"
> }, {
> "_id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28"
> }, {
> "source" : "unknown"
> }, {
> "message" : "XML TEST"
> }, {
> "gl2_source_input" : "5592913ec026f261b8d5ce9a"
> }, {
> "timestamp" : "2016-10-19T10:31:58.824Z"
> } ],
> "field_count" : 8,
> "stream_ids" : [ "552b92b2e4b0c055e41ffb8e",
> "0001" ],
> "is_source_inet_address" : false,
> "inet_address" : null
> },
> "index" : "graylog2_27"
> }
>
>
> Screenshot:
>
>
>
>
>
> Cheers,
> Jochen
>
> On Tuesday, 18 October 2016 14:10:11 UTC+2, 4BRobby wrote:
>>
>> Hell Jochen,
>>
>> Enclosed my configuration if this helps. It is the OVA appliance, latest
>> graylog version.
>>
>> A new log worked b.t.w. so it may be the problem that the .. at one time
>> breaks the xml tag itself.
>> Nevertheless it is an unexpected behaviour that the xml even gets parsed
>> out of the message. XML is of course base64 coded like all other log
>> message bodies (full_message) that are send to Graylog.
>>
>> I have found no information about this behaviour so far.
>>
>> I had to anonymize
[graylog2] Re: Shortened Xml in message field
Hi Robby,
I cannot reproduce your problem.
I've created a GELF UDP input on an installation of Graylog 2.1.1 and sent
a GELF message to it with the XML example from your first email in the
full_text message field.
Everything has been ingested and is being shown as intended (i. e. unparsed
and the verbatim XML). Maybe some extractor is interfering with your
payload or even the client already sends the modified GELF message.
Graylog and Elasticsearch itself do not parse XML from message fields by
default.
For reproduction, here's what I did:
Create GELF UDP input with the following configuration and no extractors:
bind_address: 127.0.0.1
decompress_size_limit: 8388608
override_source:
port: 12201
recv_buffer_size: 16384
Send message to Graylog:
$ echo '{"version":"1.1", "message":"XML TEST", "full_message":"\n \n
Success\n DoSomething\n
8\n \n 0\n \n
Cheers,
Jochen
On Tuesday, 18 October 2016 14:10:11 UTC+2, 4BRobby wrote:
>
> Hell Jochen,
>
> Enclosed my configuration if this helps. It is the OVA appliance, latest
> graylog version.
>
> A new log worked b.t.w. so it may be the problem that the .. at one time
> breaks the xml tag itself.
> Nevertheless it is an unexpected behaviour that the xml even gets parsed
> out of the message. XML is of course base64 coded like all other log
> message bodies (full_message) that are send to Graylog.
>
> I have found no information about this behaviour so far.
>
> I had to anonymize the message (That is why it reads nested). One tag is
> called result - but only numbers and usual text in the tag values, Nothing
> special really.
>
> Thanks
> Robby
>
>
> Am Dienstag, 18. Oktober 2016 12:45:50 UTC+2 schrieb Jochen Schalanda:
>>
>> Hi,
>>
>> On Tuesday, 18 October 2016 12:27:45 UTC+2, 4BRobby wrote:
>>>
>>> Configuraiton: There is no transformation for the message processing,
>>> all to default.
>>>
>>
>> That's not what I've asked for. Please provide the requested information,
>> otherwise we might not be able to help you.
>>
>> Cheers,
>> Jochen
>>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/cda4020d-0595-4c34-bd1e-d597366ba6c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Shortened Xml in message field
Hi, On Tuesday, 18 October 2016 12:27:45 UTC+2, 4BRobby wrote: > > Configuraiton: There is no transformation for the message processing, all > to default. > That's not what I've asked for. Please provide the requested information, otherwise we might not be able to help you. Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/1ac2784a-979a-4e2b-b201-b1ad53bdb1f0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Shortened Xml in message field
This is the complete message above (first one)... It is shortened by my code and ".." appended and then the XML is malformed / autocompleted by elasticsearch (my guess). Configuraiton: There is no transformation for the message processing, all to default. Am Dienstag, 18. Oktober 2016 12:20:04 UTC+2 schrieb Jochen Schalanda: > > Hi, > > please post the complete message and the configuration of your Graylog > node and the input you're using to ingest these messages. > > Cheers, > Jochen > > On Tuesday, 18 October 2016 11:17:00 UTC+2, 4BRobby wrote: >> >> Hey guys, >> >> Maybe someone has noticed this before. It seems like Elasticsearch (out >> of the box) is applying an XML filter to the fields. >> >> Input-Example for full_message field: >> >> >> Success >> DoSomething >> 8 >> >> 0 >> >> > >> Result full_message field: >> >> >>Success >>DoSomething >>8 >> >> >> >> >> It seems like Elasticsearch is automatically completing the XML >> discarding all tags that cannot be completed. >> In this case an 400kb XML is shortened to 65kb by code and the gets >> completely truncated to the result above. >> >> Is there any way to prevent this by making a special configuration to >> elasticsearch? >> >> Thanks. >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/46a182f0-827c-43de-95af-f12f9e445206%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Shortened Xml in message field
Hi, please post the complete message and the configuration of your Graylog node and the input you're using to ingest these messages. Cheers, Jochen On Tuesday, 18 October 2016 11:17:00 UTC+2, 4BRobby wrote: > > Hey guys, > > Maybe someone has noticed this before. It seems like Elasticsearch (out of > the box) is applying an XML filter to the fields. > > Input-Example for full_message field: > > > Success > DoSomething > 8 > > 0 > > > Result full_message field: > > >Success >DoSomething >8 > > > > > It seems like Elasticsearch is automatically completing the XML discarding > all tags that cannot be completed. > In this case an 400kb XML is shortened to 65kb by code and the gets > completely truncated to the result above. > > Is there any way to prevent this by making a special configuration to > elasticsearch? > > Thanks. > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/9b7b127c-1a06-444f-8670-d9c78407154e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
