have a look here: https://github.com/rsyslog/rsyslog/pull/306
2015-05-26 7:57 GMT+02:00 David Lang <da...@lang.hm>:
> Ok, so what you are asking for is a JSON minify option to the property
> replacer.
>
> Is there an option in ES to have it not store teh _source field? that would
> save FAR more space.
>
> although, when you are putting things in ES, you are not going for the most
> space efficient storage in the first place. ES deliberatly uses a LOT of
> space to optimize it's searches.
>
> with 1 character values and fieldnames the extra whitespace could amount to
> a significant percentage of the raw data, but once you add the indexing data
> that ES also stores, the overall percentage gets knocked down drastically
>
> '{ "a" : "b" }, ' is about the worst possible case. total length 15
> characters, 5 of them 'wasted', so it seems like 30%, but then when you
> realize that everything gets stored at least twice, that cuts it down to
> 15%, then you add in the index data, and the fact that the field names and
> data values are going to be longer....
>
> I have trouble believing that this would make even a 1% difference on
> anything resembling real-world data. And if it did, the best thing would be
> for ES to store the data in a compressed form that would make the whitespace
> effectively free. (assuming it doesn't already)
>
> could you run a test, do two ES instances, one that you populate with
> 'normal' data and one that you populate with the identical data 'minified'
> and see if you can measure the difference?
>
> David Lang
>
>
> On Tue, 26 May 2015, chenlin rao wrote:
>
>> No, ES store the raw JSON in _source field. We can set `"_size": {
>> "enabled": true }` to check the record size.
>>
>> $ curl
>>
>> 10.19.0.97:9200/testindex/testtype/AU2OSfj0ZRvQT5qcC_l3?fields=_size,_source
>>
>> {"_index":"testindex","_type":"testtype","_id":"AU2OSfj0ZRvQT5qcC_l3","_version":1,"found":true,"_source":{"@timestamp":"2015-05-25T07:29:35+08:00","host":"
>> web032.mweibo.yf.sinanode.com", "content": "Traceroute Result of
>> api.weibo.cn:\nDNS is 202.103.224.68\nIP is
>>
>> 180.149.153.216\n1|192:168:1:1|34.482ms\n2|219:159:136:1|27.005ms\n3|218:65:201:21|206.549ms\n4|202:103:236:53|116.733ms\n5|*\n6|*\n7|*\n8|180:149:128:54|339.948ms\n9|*\n10|180:149:129:178|302.180ms\n11|180:149:153:216|1.840s\n\n",
>> "__date": 1432477827.545300 },"fields":{"_size":420}}
>>
>> $ curl
>>
>> 10.19.0.97:9200/testindex/testtype/AU2OSflQZRvQT5qcC_l4?fields=_size,_source
>>
>> {"_index":"testindex","_type":"testtype","_id":"AU2OSflQZRvQT5qcC_l4","_version":1,"found":true,"_source":{"@timestamp":"2015-05-25T07:29:35+08:00","host":"
>> web032.mweibo.yf.sinanode.com", "content":"Traceroute Result of
>> api.weibo.cn:\nDNS
>> is 202.103.224.68\nIP is
>>
>> 180.149.153.216\n1|192:168:1:1|34.482ms\n2|219:159:136:1|27.005ms\n3|218:65:201:21|206.549ms\n4|202:103:236:53|116.733ms\n5|*\n6|*\n7|*\n8|180:149:128:54|339.948ms\n9|*\n10|180:149:129:178|302.180ms\n11|180:149:153:216|1.840s\n\n","__date":1432477827.545300},"fields":{"_size":416}}
>>
>> Well, I know this is not a good example. but I have some other loglines
>> that has hundred of fields, hundreds of blank space...
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
