Roy, One thing that might help is upgrading to 1.6.6 to get the fix for OAK-6642 /OAK-6110. With that fix you should be able to run within a 2GB heap and compaction times should go down.
Re. the bak files: from an operational POV it is safe to remove them. Those files are kept in the case an instances in shut down in an non clean way and needs to recover the most recent tar file that hasn't been properly closed. There is up to 5 seconds worth of data that might get lost in that process. The bak files *might* contain parts of this data and it *might* be possible to recover *some* of it manually. If this is not needed (I haven't seen a case where it was so far), you can safely remove them. Michael . On 5 March 2018 at 14:44, Roy Teeuwen <[email protected]> wrote: > Hey guys, > > We have upgraded to Oak 1.6.5 (AEM 6.3) and we have noticed that the offline > compaction is a lot slower than it was initially in AEM 6.1. While we had > compaction of 5-15 mins for 15GB at first, it now takes half an hour. Is > there any performance tweaking / parameters we can use to speed it up? Below > you can find the script we use to do daily offline compaction and a log file > of one of those times. Usually it takes away one GB per day. Seeing as we > have the possibility to take down the system once a night, we don't have to > solely rely on online compaction. > > Another question I have with this, would it be ok that we remove the .bak > files automatically after the compaction has ran? Or why do they stay in the > folder? > > Script: > ``` > #!/bin/bash > > # AEM config > AEM_PATH="/opt/aem/author" > AEM_USER="aem-user" > CRX_QUICKSTART_PATH="${AEM_PATH}/crx-quickstart" > REPOSITORY_PATH="${CRX_QUICKSTART_PATH}/repository" > SEGMENTSTORE_PATH="${REPOSITORY_PATH}/segmentstore" > AEM_STATUS="${CRX_QUICKSTART_PATH}/bin/status" > AEM_START="${CRX_QUICKSTART_PATH}/bin/start" > AEM_STOP="${CRX_QUICKSTART_PATH}/bin/stop" > > # OAK Run config > OAK_RUN_JAR="/opt/aem_dependencies/aem/oak-run-1.6.5.jar" > HEAPSIZE="-Xmx12228M" > UPDATE_LIMIT="-Dupdate.limit=10000" > COMPRESS_INTERVAL="-Dcompress-interval=100000" > COMPACTION_PROGRESS_LOG="-Dcompaction-progress-log=150000" > MEMORY_MAPPED="-Dtar.memoryMapped=true" > OAK_RUN_PARAMETERS="${HEAPSIZE} ${UPDATE_LIMIT} ${COMPRESS_INTERVAL} > ${COMPACTION_PROGRESS_LOG} ${MEMORY_MAPPED}" > OAK_RUN="java ${OAK_RUN_PARAMETERS} -jar ${OAK_RUN_JAR}" > > if ! [ "$(whoami)" = "${AEM_USER}" ] > then > echo "Error: Maintenance task needs to be run as AEM user (${AEM_USER})." > exit 1 > fi > > echo "==> Starting maintenance ($(date))" > > echo "==> Make sure AEM is not running" > if ${AEM_STATUS} > then > echo "==> Stopping AEM.." > ${AEM_STOP} > fi > > echo "==> Cleanup unrefrenced checkpoints" > ${OAK_RUN} checkpoints ${SEGMENTSTORE_PATH} || exit 1 > ${OAK_RUN} checkpoints ${SEGMENTSTORE_PATH} rm-unreferenced || exit 1 > > echo "==> Compact TAR segmentstore" > ${OAK_RUN} compact ${SEGMENTSTORE_PATH} || exit 1 > > echo "==> Starting AEM" > ${AEM_START} > > echo "==> Maintenance completed ($(date))" > > exit 0 > > ``` > > Logs: > > ``` > ==> Stopping AEM.. > 05.03.2018 04:00:01.716 *INFO * [main] Setting sling.home=. (command line) > 05.03.2018 04:00:01.743 *INFO * [main] Sent 'stop' to /127.0.0.1:45726: OK > Application not running > ==> Cleanup unrefrenced checkpoints > Apache Jackrabbit Oak 1.6.5 > Checkpoints /opt/aem/author/crx-quickstart/repository/segmentstore > - 3f22ad0f-3b78-4764-a697-c323f6d95180 created 2018-03-05 04:26:30.67 expires > 2020-11-29 04:26:30.67 > - 5caca09a-9d6b-426d-8166-e967ca1fbf7e created 2018-03-05 04:26:30.567 > expires 2020-11-29 04:26:30.567 > Found 2 checkpoints > Apache Jackrabbit Oak 1.6.5 > Checkpoints /opt/aem/author/crx-quickstart/repository/segmentstore > Referenced checkpoint from /:async@async is > 3f22ad0f-3b78-4764-a697-c323f6d95180 > Referenced checkpoint from /:async@fulltext-async is > 5caca09a-9d6b-426d-8166-e967ca1fbf7e > Removed 0 checkpoints in 44ms. > ==> Compact TAR segmentstore > Apache Jackrabbit Oak 1.6.5 > Compacting /opt/aem/author/crx-quickstart/repository/segmentstore > With default access mode > before > Sun Mar 04 04:48:27 CET 2018, data05195a.tar > Sun Mar 04 04:51:46 CET 2018, data05213a.tar > Sun Mar 04 04:37:16 CET 2018, data05173a.tar > Sun Mar 04 04:51:22 CET 2018, data05211a.tar > Sun Mar 04 04:43:13 CET 2018, data05182a.tar > Sun Mar 04 04:55:57 CET 2018, data05226a.tar > Mon Mar 05 03:00:57 CET 2018, data05229a.tar > Sun Mar 04 04:36:03 CET 2018, data05171a.tar > Sun Mar 04 04:52:42 CET 2018, data05218a.tar > Sun Mar 04 04:34:42 CET 2018, data05169a.tar > Tue Jan 16 06:25:35 CET 2018, data01538a.tar.bak > Sun Mar 04 04:34:07 CET 2018, data05168a.tar > Sun Mar 04 04:49:31 CET 2018, data05201a.tar > Sun Mar 04 04:53:31 CET 2018, data05222a.tar > Sun Mar 04 06:11:04 CET 2018, data05228a.tar > Sun Mar 04 04:46:31 CET 2018, data05188a.tar > Sun Mar 04 04:50:08 CET 2018, data05204a.tar > Sun Mar 04 04:48:36 CET 2018, data05196a.tar > Thu Dec 14 07:14:54 CET 2017, repo.lock > Sun Mar 04 04:43:54 CET 2018, data05183a.tar > Sun Mar 04 04:49:08 CET 2018, data05199a.tar > Sun Mar 04 04:49:43 CET 2018, data05202a.tar > Fri Mar 02 04:57:25 CET 2018, data05022a.tar.2.bak > Sun Mar 04 04:41:19 CET 2018, data05179a.tar > Mon Mar 05 04:26:34 CET 2018, journal.log > Fri Mar 02 04:57:24 CET 2018, data05022a.tar.bak > Sun Mar 04 04:53:55 CET 2018, data05224a.tar > Sun Mar 04 04:49:55 CET 2018, data05203a.tar > Sun Mar 04 04:53:03 CET 2018, data05220a.tar > Mon Mar 05 04:26:45 CET 2018, manifest > Sun Mar 04 04:49:20 CET 2018, data05200a.tar > Sun Mar 04 04:42:30 CET 2018, data05181a.tar > Sun Mar 04 04:45:37 CET 2018, data05186a.tar > Sun Mar 04 04:55:57 CET 2018, data05227a.tar > Sun Mar 04 04:48:59 CET 2018, data05198a.tar > Sun Mar 04 04:37:39 CET 2018, data05174a.tar > Sun Mar 04 04:52:19 CET 2018, data05216a.tar > Sun Mar 04 04:47:43 CET 2018, data05191a.tar > Sun Mar 04 04:48:47 CET 2018, data05197a.tar > Sun Mar 04 04:45:01 CET 2018, data05185a.tar > Sun Mar 04 04:38:20 CET 2018, data05175a.tar > Sun Mar 04 04:50:42 CET 2018, data05207a.tar > Sun Mar 04 04:40:31 CET 2018, data05178a.tar > Sun Mar 04 04:52:30 CET 2018, data05217a.tar > Sun Mar 04 04:46:56 CET 2018, data05189a.tar > Sun Mar 04 04:39:05 CET 2018, data05176a.tar > Sun Mar 04 04:35:12 CET 2018, data05170a.tar > Sun Mar 04 04:50:33 CET 2018, data05206a.tar > Sun Mar 04 04:47:22 CET 2018, data05190a.tar > Sun Mar 04 04:51:34 CET 2018, data05212a.tar > Sun Mar 04 04:54:25 CET 2018, data05225a.tar > Sun Mar 04 04:36:55 CET 2018, data05172a.tar > Sun Mar 04 04:50:20 CET 2018, data05205a.tar > Sun Mar 04 04:51:58 CET 2018, data05214a.tar > Mon Mar 05 04:26:41 CET 2018, data05231a.tar > Mon Mar 05 04:19:14 CET 2018, data05230a.tar > Sun Mar 04 04:53:43 CET 2018, data05223a.tar > Sun Mar 04 04:44:31 CET 2018, data05184a.tar > Sun Mar 04 04:47:53 CET 2018, data05192a.tar > Sun Mar 04 04:39:47 CET 2018, data05177a.tar > Sun Mar 04 04:48:17 CET 2018, data05194a.tar > Sun Mar 04 04:52:09 CET 2018, data05215a.tar > Sun Mar 04 04:48:05 CET 2018, data05193a.tar > Sun Mar 04 04:52:52 CET 2018, data05219a.tar > Sun Mar 04 04:53:20 CET 2018, data05221a.tar > Sun Mar 04 04:50:52 CET 2018, data05208a.tar > Sun Mar 04 04:46:07 CET 2018, data05187a.tar > Sun Mar 04 04:51:12 CET 2018, data05210a.tar > Sun Mar 04 04:56:04 CET 2018, gc.log > Sun Mar 04 04:33:34 CET 2018, data05167a.tar > Sun Mar 04 04:41:47 CET 2018, data05180a.tar > Sun Mar 04 04:51:02 CET 2018, data05209a.tar > size 17.1 GB (17094875037 bytes) > -> compacting > 04:26:46.520 [main] WARN o.a.jackrabbit.oak.segment.Compactor - Deprecated > argument 'compress-interval', please use 'oak.segment.compaction.cacheSize' > instead. > -> cleaning up > -> writing new journal.log: bf675b66-e8e6-4bf4-a982-dc7316323e9c:859 root > 1520222043417 > > after > Mon Mar 05 04:51:23 CET 2018, data05285a.tar > Mon Mar 05 04:36:53 CET 2018, data05244a.tar > Mon Mar 05 04:48:54 CET 2018, data05276a.tar > Mon Mar 05 04:43:10 CET 2018, data05254a.tar > Mon Mar 05 04:53:55 CET 2018, data05291a.tar > Mon Mar 05 04:48:31 CET 2018, data05274a.tar > Mon Mar 05 04:50:47 CET 2018, data05283a.tar > Mon Mar 05 04:40:27 CET 2018, data05249a.tar > Mon Mar 05 04:52:22 CET 2018, data05288a.tar > Mon Mar 05 04:49:30 CET 2018, data05278a.tar > Mon Mar 05 04:28:56 CET 2018, data05233a.tar > Mon Mar 05 04:31:18 CET 2018, data05236a.tar > Mon Mar 05 04:52:02 CET 2018, data05287a.tar > Tue Jan 16 06:25:35 CET 2018, data01538a.tar.bak > Mon Mar 05 04:47:51 CET 2018, data05272a.tar > Mon Mar 05 04:36:02 CET 2018, data05243a.tar > Mon Mar 05 04:35:14 CET 2018, data05242a.tar > Mon Mar 05 04:42:46 CET 2018, data05253a.tar > Mon Mar 05 04:44:55 CET 2018, data05260a.tar > Mon Mar 05 04:39:07 CET 2018, data05247a.tar > Mon Mar 05 04:46:50 CET 2018, data05268a.tar > Mon Mar 05 04:44:06 CET 2018, data05256a.tar > Mon Mar 05 04:45:33 CET 2018, data05263a.tar > Mon Mar 05 04:53:22 CET 2018, data05290a.tar > Thu Dec 14 07:14:54 CET 2017, repo.lock > Mon Mar 05 04:50:19 CET 2018, data05281a.tar > Mon Mar 05 04:49:13 CET 2018, data05277a.tar > Mon Mar 05 04:45:51 CET 2018, data05264a.tar > Mon Mar 05 04:33:40 CET 2018, data05240a.tar > Mon Mar 05 04:46:08 CET 2018, data05265a.tar > Mon Mar 05 04:32:55 CET 2018, data05239a.tar > Mon Mar 05 04:41:04 CET 2018, data05250a.tar > Fri Mar 02 04:57:25 CET 2018, data05022a.tar.2.bak > Mon Mar 05 04:45:18 CET 2018, data05262a.tar > Mon Mar 05 04:30:20 CET 2018, data05235a.tar > Mon Mar 05 04:44:44 CET 2018, data05259a.tar > Mon Mar 05 04:51:43 CET 2018, data05286a.tar > Mon Mar 05 04:54:03 CET 2018, journal.log > Fri Mar 02 04:57:24 CET 2018, data05022a.tar.bak > Mon Mar 05 04:32:32 CET 2018, data05238a.tar > Mon Mar 05 04:28:18 CET 2018, data05232a.tar > Mon Mar 05 04:45:07 CET 2018, data05261a.tar > Mon Mar 05 04:47:38 CET 2018, data05271a.tar > Mon Mar 05 04:49:48 CET 2018, data05279a.tar > Mon Mar 05 04:53:56 CET 2018, manifest > Mon Mar 05 04:34:30 CET 2018, data05241a.tar > Mon Mar 05 04:46:20 CET 2018, data05266a.tar > Mon Mar 05 04:43:41 CET 2018, data05255a.tar > Mon Mar 05 04:38:21 CET 2018, data05246a.tar > Mon Mar 05 04:48:14 CET 2018, data05273a.tar > Mon Mar 05 04:52:50 CET 2018, data05289a.tar > Mon Mar 05 04:42:19 CET 2018, data05252a.tar > Mon Mar 05 04:41:47 CET 2018, data05251a.tar > Mon Mar 05 04:47:07 CET 2018, data05269a.tar > Mon Mar 05 04:50:34 CET 2018, data05282a.tar > Mon Mar 05 04:47:21 CET 2018, data05270a.tar > Mon Mar 05 04:39:50 CET 2018, data05248a.tar > Mon Mar 05 04:29:40 CET 2018, data05234a.tar > Mon Mar 05 04:44:19 CET 2018, data05257a.tar > Mon Mar 05 04:44:30 CET 2018, data05258a.tar > Mon Mar 05 04:32:13 CET 2018, data05237a.tar > Mon Mar 05 04:48:43 CET 2018, data05275a.tar > Mon Mar 05 04:50:05 CET 2018, data05280a.tar > Mon Mar 05 04:37:30 CET 2018, data05245a.tar > Mon Mar 05 04:51:02 CET 2018, data05284a.tar > Mon Mar 05 04:46:33 CET 2018, data05267a.tar > Mon Mar 05 04:54:03 CET 2018, gc.log > size 16.1 GB (16081145586 bytes) > removed files [data05228a.tar, data05222a.tar, data05167a.tar, > data05184a.tar, data05214a.tar, data05192a.tar, data05200a.tar, > data05198a.tar, data05183a.tar, data05189a.tar, data05205a.tar, > data05175a.tar, data05230a.tar, data05206a.tar, data05191a.tar, > data05197a.tar, data05174a.tar, data05177a.tar, data05188a.tar, > data05171a.tar, data05182a.tar, data05169a.tar, data05224a.tar, > data05227a.tar, data05221a.tar, data05213a.tar, data05193a.tar, > data05185a.tar, data05190a.tar, data05216a.tar, data05219a.tar, > data05196a.tar, data05199a.tar, data05195a.tar, data05211a.tar, > data05187a.tar, data05203a.tar, data05208a.tar, data05186a.tar, > data05172a.tar, data05180a.tar, data05202a.tar, data05210a.tar, > data05178a.tar, data05209a.tar, data05217a.tar, data05225a.tar, > data05194a.tar, data05207a.tar, data05218a.tar, data05229a.tar, > data05220a.tar, data05168a.tar, data05223a.tar, data05226a.tar, > data05179a.tar, data05176a.tar, data05231a.tar, data05170a.tar, > data05215a.tar, data05201a.tar, data05181a.tar, data05173a.tar, > data05204a.tar, data05212a.tar] > added files [data05259a.tar, data05267a.tar, data05245a.tar, > data05237a.tar, data05253a.tar, data05261a.tar, data05283a.tar, > data05244a.tar, data05258a.tar, data05291a.tar, data05236a.tar, > data05275a.tar, data05260a.tar, data05252a.tar, data05289a.tar, > data05232a.tar, data05246a.tar, data05243a.tar, data05238a.tar, > data05235a.tar, data05240a.tar, data05279a.tar, data05262a.tar, > data05268a.tar, data05276a.tar, data05251a.tar, data05284a.tar, > data05270a.tar, data05254a.tar, data05273a.tar, data05281a.tar, > data05265a.tar, data05249a.tar, data05257a.tar, data05250a.tar, > data05256a.tar, data05264a.tar, data05239a.tar, data05287a.tar, > data05272a.tar, data05242a.tar, data05248a.tar, data05241a.tar, > data05280a.tar, data05247a.tar, data05255a.tar, data05278a.tar, > data05233a.tar, data05286a.tar, data05271a.tar, data05274a.tar, > data05288a.tar, data05263a.tar, data05266a.tar, data05285a.tar, > data05290a.tar, data05277a.tar, data05282a.tar, data05269a.tar, > data05234a.tar] > Compaction succeeded in 27.29 min (1637s). > > ``` > > Greets, > Roy
