Hi Klaus, MongoDB is only used for configuration data (e. g. inputs configuration, dashboards, streams, users, etc.) and to store index failures (i. e. messages that couldn't be properly indexed into Elasticsearch because of a index mapping conflict) and all processed log messages are being stored in Elasticsearch, so you should be fine with a rather small MongoDB setup (usually only a few megabytes) and should size the Elasticsearch cluster to accommodate your projected log message throughput.
Cheers, Jochen On Thursday, 5 November 2015 07:36:07 UTC+1, kl...@tachtler.net wrote: > > Hi Jochen, > hi list, > > first of all, thank you! > > of course your're right, that the disk volume is not only depending on the > count of messages, the averge message size is important to. I will reme,ber > this! > > BUT, *another Question*: How is the *relationship between* the *used > space from MongoDB (/var/lib/mongodb)* and the *space, used by > Elasticsearch (/var/lib/elasticsearch*)? > > Is it possible to sa 1/3 to 2/3 or something similar? > > > Thank you! > > Klaus. > > Am Montag, 2. November 2015 10:15:29 UTC+1 schrieb Jochen Schalanda: >> >> Hi Klaus, >> >> unfortunately it's not that easy to calculate the exact disk space >> requirement for the given numbers. For example your log messages could be >> as small as a few bytes and as big as several kilobytes or even megabytes. >> Additionally it's important how heterogenous the log messages are. If they >> are identical for the most part, they take less disk space than if they >> have large entropy. And last but not least the inverted index being used >> for full text searches by Elasticsearch/Lucene might grow or shrink in size >> over the lifetime of the indices (due to cleanup operations, segment >> merges, etc.). >> >> If we guess that the log messages are 1 kilobyte in size on average, you >> might need between 16 and 32 gigabytes of disk space (1,024 bytes * >> 1,200,000 messages * 7 days * 2 replicas), but that's totally rule of thumb >> and you'll have to try it yourself and monitor the setup closely. >> >> >> Cheers, >> Jochen >> >> On Monday, 2 November 2015 09:57:19 UTC+1, kl...@tachtler.net wrote: >>> >>> Hi Group, >>> >>> I'm new to graylog2 and I would like to know, how much disk space is >>> required for a small system with up to *15 server* and a log volume >>> about >>> >>> * * 1,200,000 messages* >>> ** in 7 days* >>> >>> Is there a formula to calculate the disk space? >>> >>> >>> Thank you! >>> Klaus. >>> >> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/7d3ce149-13b9-476a-8960-f4a357184703%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
