And a bit more... it's almost certainly to do with the commits happening after
every batch.
When i set rows rather high (for example; 25k) I hold a sustained throughput
for a longer duration and on the smaller index, things finish faster.
Downside is obviously more memory being used (but we have quite a bit).
Which really leads me down a route of "how to minimise the ever increasing
costs of the commits.
FYI the index is about 30gb
This is my merge policy:
<indexConfig>
<ramBufferSizeMB>128</ramBufferSizeMB>
<mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
<int name="maxMergeAtOnce">5</int>
<int name="segmentsPerTier">5</int>
<int name="maxMergedSegmentMB">5000</int>
<int name="deletesPctAllowed">25</int>
</mergePolicyFactory>
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
<int name="maxThreadCount">10</int>
<int name="maxMergeCount">15</int>
</mergeScheduler>
<mergedSegmentWarmer
class="org.apache.lucene.index.SimpleMergedSegmentWarmer" />
</indexConfig>
I'm thinking I could disable via configAPI cache autoWarming on the index for
the reindex duration?
________________________________
From: Karl Stoney <[email protected]>
Sent: 18 March 2021 10:21
To: [email protected] <[email protected]>;
[email protected] <[email protected]>
Subject: Re: REINDEXCOLLECTION gradually slows
A bit more data on this to help... really welcome any ideas as the slow down
over time just means we're never really able to finish a reindex via
REINDEXCOLLECTION
We have disabled auto commits on the destination collection during the reindex,
however we're still seeing commits happen as the reindexing happens. Does the
REINDEXCOLLECTION perform a commit after every batch or every N seconds? They
seem to be bang on about 30 seconds?
```
/karl-test/config/updateHandler
{
*
responseHeader:
{
*
status: 0,
*
QTime: 0
},
*
config:
{
*
updateHandler:
{
*
indexWriter:
{
*
closeWaitsForMerges: true
},
*
commitWithin:
{
*
softCommit: true
},
*
autoCommit:
{
*
maxDocs: -1,
*
maxTime: -1,
*
openSearcher: false
},
*
autoSoftCommit:
{
*
maxDocs: -1,
*
maxTime: 900000
}
}
}
}
```
This in turn seems to be causing cache warming, which obviously gets
increasingly slower and increasingly adds load.
________________________________
From: Karl Stoney <[email protected]>
Sent: 17 March 2021 19:16
To: [email protected] <[email protected]>
Subject: REINDEXCOLLECTION gradually slows
Hey,
So we're trying to use REINDEXCOLLECTION again (solr 8.8) and similar to the
last time we tried it (8.1) we see it gradually slow down; as you can see from
the logs below.
I terminated it early but you can see the trend in the processingRatePerSecond.
I'm wondering if anyone has any idea's why this might happen? 🤷♂️. The machine
isn't CPU or Memory constrained, so it makes reindexing larger collections
really hard (we've had to code a separate app to do it).
Thanks
```
{"timestamp":"2021-03-17T19:04:44.703Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":125,"processingRatePerSecond":1248}
{"timestamp":"2021-03-17T19:04:49.719Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":130,"processingRatePerSecond":1200}
{"timestamp":"2021-03-17T19:04:54.736Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":135,"processingRatePerSecond":1274}
{"timestamp":"2021-03-17T19:04:59.751Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":140,"processingRatePerSecond":1229}
{"timestamp":"2021-03-17T19:05:04.765Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":145,"processingRatePerSecond":1297}
{"timestamp":"2021-03-17T19:05:09.782Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":150,"processingRatePerSecond":1253}
{"timestamp":"2021-03-17T19:05:14.798Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":155,"processingRatePerSecond":1303}
{"timestamp":"2021-03-17T19:05:19.816Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":161,"processingRatePerSecond":1255}
{"timestamp":"2021-03-17T19:05:24.832Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":166,"processingRatePerSecond":1277}
{"timestamp":"2021-03-17T19:05:29.850Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":171,"processingRatePerSecond":1240}
{"timestamp":"2021-03-17T19:05:34.866Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":176,"processingRatePerSecond":1250}
{"timestamp":"2021-03-17T19:05:39.884Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":181,"processingRatePerSecond":1215}
{"timestamp":"2021-03-17T19:05:44.899Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":186,"processingRatePerSecond":1226}
{"timestamp":"2021-03-17T19:05:49.916Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":191,"processingRatePerSecond":1194}
{"timestamp":"2021-03-17T19:05:54.931Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":196,"processingRatePerSecond":1204}
{"timestamp":"2021-03-17T19:05:59.947Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":201,"processingRatePerSecond":1174}
{"timestamp":"2021-03-17T19:06:04.963Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":206,"processingRatePerSecond":1184}
{"timestamp":"2021-03-17T19:06:09.977Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":211,"processingRatePerSecond":1156}
{"timestamp":"2021-03-17T19:06:14.993Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":216,"processingRatePerSecond":1176}
{"timestamp":"2021-03-17T19:06:20.007Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":221,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:06:25.024Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":226,"processingRatePerSecond":1168}
{"timestamp":"2021-03-17T19:06:30.037Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":231,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:06:35.053Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":236,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:06:40.067Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":241,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:06:45.083Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":246,"processingRatePerSecond":1146}
{"timestamp":"2021-03-17T19:06:50.099Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":251,"processingRatePerSecond":1124}
{"timestamp":"2021-03-17T19:06:55.114Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":256,"processingRatePerSecond":1141}
{"timestamp":"2021-03-17T19:07:00.140Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":261,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:07:05.155Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":266,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:07:10.174Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":271,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:07:15.189Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":316000,"targetDocs":787176,"percentComplete":40,"durationSeconds":276,"processingRatePerSecond":1145}
{"timestamp":"2021-03-17T19:07:20.203Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":281,"processingRatePerSecond":1167}
{"timestamp":"2021-03-17T19:07:25.220Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":286,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:07:30.237Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":291,"processingRatePerSecond":1182}
{"timestamp":"2021-03-17T19:07:35.253Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":296,"processingRatePerSecond":1162}
{"timestamp":"2021-03-17T19:07:40.268Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":301,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:45.283Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":306,"processingRatePerSecond":1163}
{"timestamp":"2021-03-17T19:07:50.297Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":311,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:55.313Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":316,"processingRatePerSecond":1165}
{"timestamp":"2021-03-17T19:08:00.329Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":321,"processingRatePerSecond":1171}
{"timestamp":"2021-03-17T19:08:05.345Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":326,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:08:10.360Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":331,"processingRatePerSecond":1166}
{"timestamp":"2021-03-17T19:08:15.377Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":336,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:08:20.391Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":341,"processingRatePerSecond":1155}
{"timestamp":"2021-03-17T19:08:25.408Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":346,"processingRatePerSecond":1139}
{"timestamp":"2021-03-17T19:08:30.423Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":351,"processingRatePerSecond":1151}
{"timestamp":"2021-03-17T19:08:35.440Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":356,"processingRatePerSecond":1135}
{"timestamp":"2021-03-17T19:08:40.457Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":361,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:08:45.471Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":366,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:08:50.490Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":371,"processingRatePerSecond":1137}
{"timestamp":"2021-03-17T19:08:55.506Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":376,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:09:00.523Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":381,"processingRatePerSecond":1134}
{"timestamp":"2021-03-17T19:09:05.538Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":386,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:09:10.554Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":391,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:09:15.571Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":396,"processingRatePerSecond":1111}
{"timestamp":"2021-03-17T19:09:20.587Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":401,"processingRatePerSecond":1127}
{"timestamp":"2021-03-17T19:09:25.603Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":406,"processingRatePerSecond":1113}
{"timestamp":"2021-03-17T19:09:30.618Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":411,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:09:35.635Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":416,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:09:40.651Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":421,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:09:45.668Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":426,"processingRatePerSecond":1117}
{"timestamp":"2021-03-17T19:09:50.683Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":431,"processingRatePerSecond":1128}
{"timestamp":"2021-03-17T19:09:55.698Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":436,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:10:00.714Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":441,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:10:05.731Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":446,"processingRatePerSecond":1112}
{"timestamp":"2021-03-17T19:10:10.746Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":451,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:10:15.763Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":456,"processingRatePerSecond":1110}
{"timestamp":"2021-03-17T19:10:20.778Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":461,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:25.792Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":466,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:30.808Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":472,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:35.821Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":477,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:40.840Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":482,"processingRatePerSecond":1116}
{"timestamp":"2021-03-17T19:10:45.854Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":487,"processingRatePerSecond":1105}
{"timestamp":"2021-03-17T19:10:50.870Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":492,"processingRatePerSecond":1106}
{"timestamp":"2021-03-17T19:10:55.887Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":497,"processingRatePerSecond":1095}
{"timestamp":"2021-03-17T19:11:00.901Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":502,"processingRatePerSecond":1092}
{"timestamp":"2021-03-17T19:11:05.916Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":507,"processingRatePerSecond":1081}
{"timestamp":"2021-03-17T19:11:10.930Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":512,"processingRatePerSecond":1078}
{"timestamp":"2021-03-17T19:11:15.946Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":517,"processingRatePerSecond":1068}
{"timestamp":"2021-03-17T19:11:20.962Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":522,"processingRatePerSecond":1061}
{"timestamp":"2021-03-17T19:11:25.978Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":527,"processingRatePerSecond":1051}
{"timestamp":"2021-03-17T19:11:30.993Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":532,"processingRatePerSecond":1049}
{"timestamp":"2021-03-17T19:11:36.012Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":537,"processingRatePerSecond":1039}
{"timestamp":"2021-03-17T19:11:41.038Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":542,"processingRatePerSecond":1037}
{"timestamp":"2021-03-17T19:11:46.055Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":547,"processingRatePerSecond":1027}
{"timestamp":"2021-03-17T19:11:51.071Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":552,"processingRatePerSecond":1025}
{"timestamp":"2021-03-17T19:11:56.087Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":557,"processingRatePerSecond":1016}
{"timestamp":"2021-03-17T19:12:01.104Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":562,"processingRatePerSecond":1014}
{"timestamp":"2021-03-17T19:12:06.123Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":567,"processingRatePerSecond":1005}
{"timestamp":"2021-03-17T19:12:11.143Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":572,"processingRatePerSecond":1007}
{"timestamp":"2021-03-17T19:12:16.158Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":577,"processingRatePerSecond":998}
{"timestamp":"2021-03-17T19:12:21.174Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":582,"processingRatePerSecond":997}
{"timestamp":"2021-03-17T19:12:26.189Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":587,"processingRatePerSecond":988}
{"timestamp":"2021-03-17T19:12:31.205Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":592,"processingRatePerSecond":990}
{"timestamp":"2021-03-17T19:12:36.220Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":597,"processingRatePerSecond":982}
{"timestamp":"2021-03-17T19:12:41.236Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":602,"processingRatePerSecond":980}
{"timestamp":"2021-03-17T19:12:46.252Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":607,"processingRatePerSecond":972}
{"timestamp":"2021-03-17T19:12:51.266Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":612,"processingRatePerSecond":971}
{"timestamp":"2021-03-17T19:12:56.282Z","level":"info","module":"solr","message":"reindex
still
running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":617,"processingRatePerSecond":963}
```
Unless expressly stated otherwise in this email, this e-mail is sent on behalf
of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester,
Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited
is part of the Auto Trader Group Plc group. This email and any files
transmitted with it are confidential and may be legally privileged, and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the sender.
This email message has been swept for the presence of computer viruses.
Unless expressly stated otherwise in this email, this e-mail is sent on behalf
of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester,
Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited
is part of the Auto Trader Group Plc group. This email and any files
transmitted with it are confidential and may be legally privileged, and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the sender.
This email message has been swept for the presence of computer viruses.
