A few examples of index recalc performance differences: In 1.1.6 ... 2015-09-04_03:56:38.19431 INFO [RebuildIndexRangesJob] Calculated range of [graylog_2075] in [9ms]. 2015-09-04_03:56:38.20487 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1800] in [10ms]. 2015-09-04_03:56:38.21176 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1802] in [6ms]. 2015-09-04_03:56:38.22028 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1801] in [8ms]. 2015-09-04_03:56:38.23003 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1808] in [9ms]. 2015-09-04_03:56:38.23855 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1807] in [8ms]. 2015-09-04_03:56:38.24877 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1809] in [10ms]. 2015-09-04_03:56:38.26011 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1804] in [11ms]. 2015-09-04_03:56:38.26902 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1803] in [8ms]. 2015-09-04_03:56:38.28049 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1806] in [11ms]. 2015-09-04_03:56:38.29289 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1805] in [12ms]. 2015-09-04_03:56:38.41557 INFO [RebuildIndexRangesJob] Done calculating index ranges for 800 indices. Took 10608ms.
In 1.2.1: ... 2015-10-03_17:00:59.70210 INFO [EsIndexRangeService] Calculated range of [graylog_3358] in [1609ms]. 2015-10-03_17:01:05.63202 INFO [EsIndexRangeService] Calculated range of [graylog_3357] in [50ms]. 2015-10-03_17:01:08.85062 INFO [EsIndexRangeService] Calculated range of [graylog_3359] in [47ms]. 2015-10-03_17:01:12.39922 INFO [EsIndexRangeService] Calculated range of [graylog_3365] in [119ms]. 2015-10-03_17:01:15.80573 INFO [EsIndexRangeService] Calculated range of [graylog_3364] in [40ms]. 2015-10-03_17:01:19.29222 INFO [EsIndexRangeService] Calculated range of [graylog_3367] in [43ms]. 2015-10-03_17:01:22.98674 INFO [EsIndexRangeService] Calculated range of [graylog_3366] in [142ms]. 2015-10-03_17:01:29.74936 INFO [EsIndexRangeService] Calculated range of [graylog_3361] in [120ms]. 2015-10-03_17:01:34.11754 INFO [EsIndexRangeService] Calculated range of [graylog_3360] in [128ms]. 2015-10-03_17:01:37.91544 INFO [EsIndexRangeService] Calculated range of [graylog_3363] in [134ms]. 2015-10-03_17:01:41.41116 INFO [EsIndexRangeService] Calculated range of [graylog_3362] in [237ms]. 2015-10-03_17:01:45.31576 INFO [RebuildIndexRangesJob] Done calculating index ranges for 1855 indices. Took 9551409ms. For good measure, I tried scaling the ES cluster up and down. Performance is exactly the same whether I run on a single large server vs 4 smaller servers. On Friday, October 2, 2015 at 9:29:37 AM UTC-6, Jesse Skrivseth wrote: > > Note that for the 1800 indices we have in this instance, the ~5 second > delay between each index during range calculation adds up to ~150 minutes > of additional delay when calculating ranges. My "> 20 minutes" comment > should be more like "> 2.5 hours", plus actual time spent calculating > ranges. > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/073650ab-3892-4657-a341-a83d8ab78dec%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
