Correct, when an instance is shutdown in an unclean way (i.e. kill -9) a backup of the most recent tar file will most likely be created on the next start.
Michael On 5 March 2018 at 17:07, Roy Teeuwen <[email protected]> wrote: > Hey Michael, > > Good to hear because we are updating to 1.6.6. in 3 weeks! Let's hope it > helps. > > About the .bak files, do I understand it correctly then that the .bak files > are only created when something went wrong on shutdown? > > Greets, > Roy > >> On 5 Mar 2018, at 16:59, Michael Dürig <[email protected]> wrote: >> >> Roy, >> >> One thing that might help is upgrading to 1.6.6 to get the fix for >> OAK-6642 /OAK-6110. With that fix you should be able to run within a >> 2GB heap and compaction times should go down. >> >> Re. the bak files: from an operational POV it is safe to remove them. >> Those files are kept in the case an instances in shut down in an non >> clean way and needs to recover the most recent tar file that hasn't >> been properly closed. There is up to 5 seconds worth of data that >> might get lost in that process. The bak files *might* contain parts of >> this data and it *might* be possible to recover *some* of it manually. >> If this is not needed (I haven't seen a case where it was so far), you >> can safely remove them. >> >> Michael >> >> . >> >> On 5 March 2018 at 14:44, Roy Teeuwen <[email protected]> wrote: >>> Hey guys, >>> >>> We have upgraded to Oak 1.6.5 (AEM 6.3) and we have noticed that the >>> offline compaction is a lot slower than it was initially in AEM 6.1. While >>> we had compaction of 5-15 mins for 15GB at first, it now takes half an >>> hour. Is there any performance tweaking / parameters we can use to speed it >>> up? Below you can find the script we use to do daily offline compaction and >>> a log file of one of those times. Usually it takes away one GB per day. >>> Seeing as we have the possibility to take down the system once a night, we >>> don't have to solely rely on online compaction. >>> >>> Another question I have with this, would it be ok that we remove the .bak >>> files automatically after the compaction has ran? Or why do they stay in >>> the folder? >>> >>> Script: >>> ``` >>> #!/bin/bash >>> >>> # AEM config >>> AEM_PATH="/opt/aem/author" >>> AEM_USER="aem-user" >>> CRX_QUICKSTART_PATH="${AEM_PATH}/crx-quickstart" >>> REPOSITORY_PATH="${CRX_QUICKSTART_PATH}/repository" >>> SEGMENTSTORE_PATH="${REPOSITORY_PATH}/segmentstore" >>> AEM_STATUS="${CRX_QUICKSTART_PATH}/bin/status" >>> AEM_START="${CRX_QUICKSTART_PATH}/bin/start" >>> AEM_STOP="${CRX_QUICKSTART_PATH}/bin/stop" >>> >>> # OAK Run config >>> OAK_RUN_JAR="/opt/aem_dependencies/aem/oak-run-1.6.5.jar" >>> HEAPSIZE="-Xmx12228M" >>> UPDATE_LIMIT="-Dupdate.limit=10000" >>> COMPRESS_INTERVAL="-Dcompress-interval=100000" >>> COMPACTION_PROGRESS_LOG="-Dcompaction-progress-log=150000" >>> MEMORY_MAPPED="-Dtar.memoryMapped=true" >>> OAK_RUN_PARAMETERS="${HEAPSIZE} ${UPDATE_LIMIT} ${COMPRESS_INTERVAL} >>> ${COMPACTION_PROGRESS_LOG} ${MEMORY_MAPPED}" >>> OAK_RUN="java ${OAK_RUN_PARAMETERS} -jar ${OAK_RUN_JAR}" >>> >>> if ! [ "$(whoami)" = "${AEM_USER}" ] >>> then >>> echo "Error: Maintenance task needs to be run as AEM user (${AEM_USER})." >>> exit 1 >>> fi >>> >>> echo "==> Starting maintenance ($(date))" >>> >>> echo "==> Make sure AEM is not running" >>> if ${AEM_STATUS} >>> then >>> echo "==> Stopping AEM.." >>> ${AEM_STOP} >>> fi >>> >>> echo "==> Cleanup unrefrenced checkpoints" >>> ${OAK_RUN} checkpoints ${SEGMENTSTORE_PATH} || exit 1 >>> ${OAK_RUN} checkpoints ${SEGMENTSTORE_PATH} rm-unreferenced || exit 1 >>> >>> echo "==> Compact TAR segmentstore" >>> ${OAK_RUN} compact ${SEGMENTSTORE_PATH} || exit 1 >>> >>> echo "==> Starting AEM" >>> ${AEM_START} >>> >>> echo "==> Maintenance completed ($(date))" >>> >>> exit 0 >>> >>> ``` >>> >>> Logs: >>> >>> ``` >>> ==> Stopping AEM.. >>> 05.03.2018 04:00:01.716 *INFO * [main] Setting sling.home=. (command line) >>> 05.03.2018 04:00:01.743 *INFO * [main] Sent 'stop' to /127.0.0.1:45726: OK >>> Application not running >>> ==> Cleanup unrefrenced checkpoints >>> Apache Jackrabbit Oak 1.6.5 >>> Checkpoints /opt/aem/author/crx-quickstart/repository/segmentstore >>> - 3f22ad0f-3b78-4764-a697-c323f6d95180 created 2018-03-05 04:26:30.67 >>> expires 2020-11-29 04:26:30.67 >>> - 5caca09a-9d6b-426d-8166-e967ca1fbf7e created 2018-03-05 04:26:30.567 >>> expires 2020-11-29 04:26:30.567 >>> Found 2 checkpoints >>> Apache Jackrabbit Oak 1.6.5 >>> Checkpoints /opt/aem/author/crx-quickstart/repository/segmentstore >>> Referenced checkpoint from /:async@async is >>> 3f22ad0f-3b78-4764-a697-c323f6d95180 >>> Referenced checkpoint from /:async@fulltext-async is >>> 5caca09a-9d6b-426d-8166-e967ca1fbf7e >>> Removed 0 checkpoints in 44ms. >>> ==> Compact TAR segmentstore >>> Apache Jackrabbit Oak 1.6.5 >>> Compacting /opt/aem/author/crx-quickstart/repository/segmentstore >>> With default access mode >>> before >>> Sun Mar 04 04:48:27 CET 2018, data05195a.tar >>> Sun Mar 04 04:51:46 CET 2018, data05213a.tar >>> Sun Mar 04 04:37:16 CET 2018, data05173a.tar >>> Sun Mar 04 04:51:22 CET 2018, data05211a.tar >>> Sun Mar 04 04:43:13 CET 2018, data05182a.tar >>> Sun Mar 04 04:55:57 CET 2018, data05226a.tar >>> Mon Mar 05 03:00:57 CET 2018, data05229a.tar >>> Sun Mar 04 04:36:03 CET 2018, data05171a.tar >>> Sun Mar 04 04:52:42 CET 2018, data05218a.tar >>> Sun Mar 04 04:34:42 CET 2018, data05169a.tar >>> Tue Jan 16 06:25:35 CET 2018, data01538a.tar.bak >>> Sun Mar 04 04:34:07 CET 2018, data05168a.tar >>> Sun Mar 04 04:49:31 CET 2018, data05201a.tar >>> Sun Mar 04 04:53:31 CET 2018, data05222a.tar >>> Sun Mar 04 06:11:04 CET 2018, data05228a.tar >>> Sun Mar 04 04:46:31 CET 2018, data05188a.tar >>> Sun Mar 04 04:50:08 CET 2018, data05204a.tar >>> Sun Mar 04 04:48:36 CET 2018, data05196a.tar >>> Thu Dec 14 07:14:54 CET 2017, repo.lock >>> Sun Mar 04 04:43:54 CET 2018, data05183a.tar >>> Sun Mar 04 04:49:08 CET 2018, data05199a.tar >>> Sun Mar 04 04:49:43 CET 2018, data05202a.tar >>> Fri Mar 02 04:57:25 CET 2018, data05022a.tar.2.bak >>> Sun Mar 04 04:41:19 CET 2018, data05179a.tar >>> Mon Mar 05 04:26:34 CET 2018, journal.log >>> Fri Mar 02 04:57:24 CET 2018, data05022a.tar.bak >>> Sun Mar 04 04:53:55 CET 2018, data05224a.tar >>> Sun Mar 04 04:49:55 CET 2018, data05203a.tar >>> Sun Mar 04 04:53:03 CET 2018, data05220a.tar >>> Mon Mar 05 04:26:45 CET 2018, manifest >>> Sun Mar 04 04:49:20 CET 2018, data05200a.tar >>> Sun Mar 04 04:42:30 CET 2018, data05181a.tar >>> Sun Mar 04 04:45:37 CET 2018, data05186a.tar >>> Sun Mar 04 04:55:57 CET 2018, data05227a.tar >>> Sun Mar 04 04:48:59 CET 2018, data05198a.tar >>> Sun Mar 04 04:37:39 CET 2018, data05174a.tar >>> Sun Mar 04 04:52:19 CET 2018, data05216a.tar >>> Sun Mar 04 04:47:43 CET 2018, data05191a.tar >>> Sun Mar 04 04:48:47 CET 2018, data05197a.tar >>> Sun Mar 04 04:45:01 CET 2018, data05185a.tar >>> Sun Mar 04 04:38:20 CET 2018, data05175a.tar >>> Sun Mar 04 04:50:42 CET 2018, data05207a.tar >>> Sun Mar 04 04:40:31 CET 2018, data05178a.tar >>> Sun Mar 04 04:52:30 CET 2018, data05217a.tar >>> Sun Mar 04 04:46:56 CET 2018, data05189a.tar >>> Sun Mar 04 04:39:05 CET 2018, data05176a.tar >>> Sun Mar 04 04:35:12 CET 2018, data05170a.tar >>> Sun Mar 04 04:50:33 CET 2018, data05206a.tar >>> Sun Mar 04 04:47:22 CET 2018, data05190a.tar >>> Sun Mar 04 04:51:34 CET 2018, data05212a.tar >>> Sun Mar 04 04:54:25 CET 2018, data05225a.tar >>> Sun Mar 04 04:36:55 CET 2018, data05172a.tar >>> Sun Mar 04 04:50:20 CET 2018, data05205a.tar >>> Sun Mar 04 04:51:58 CET 2018, data05214a.tar >>> Mon Mar 05 04:26:41 CET 2018, data05231a.tar >>> Mon Mar 05 04:19:14 CET 2018, data05230a.tar >>> Sun Mar 04 04:53:43 CET 2018, data05223a.tar >>> Sun Mar 04 04:44:31 CET 2018, data05184a.tar >>> Sun Mar 04 04:47:53 CET 2018, data05192a.tar >>> Sun Mar 04 04:39:47 CET 2018, data05177a.tar >>> Sun Mar 04 04:48:17 CET 2018, data05194a.tar >>> Sun Mar 04 04:52:09 CET 2018, data05215a.tar >>> Sun Mar 04 04:48:05 CET 2018, data05193a.tar >>> Sun Mar 04 04:52:52 CET 2018, data05219a.tar >>> Sun Mar 04 04:53:20 CET 2018, data05221a.tar >>> Sun Mar 04 04:50:52 CET 2018, data05208a.tar >>> Sun Mar 04 04:46:07 CET 2018, data05187a.tar >>> Sun Mar 04 04:51:12 CET 2018, data05210a.tar >>> Sun Mar 04 04:56:04 CET 2018, gc.log >>> Sun Mar 04 04:33:34 CET 2018, data05167a.tar >>> Sun Mar 04 04:41:47 CET 2018, data05180a.tar >>> Sun Mar 04 04:51:02 CET 2018, data05209a.tar >>> size 17.1 GB (17094875037 bytes) >>> -> compacting >>> 04:26:46.520 [main] WARN o.a.jackrabbit.oak.segment.Compactor - Deprecated >>> argument 'compress-interval', please use 'oak.segment.compaction.cacheSize' >>> instead. >>> -> cleaning up >>> -> writing new journal.log: bf675b66-e8e6-4bf4-a982-dc7316323e9c:859 >>> root 1520222043417 >>> >>> after >>> Mon Mar 05 04:51:23 CET 2018, data05285a.tar >>> Mon Mar 05 04:36:53 CET 2018, data05244a.tar >>> Mon Mar 05 04:48:54 CET 2018, data05276a.tar >>> Mon Mar 05 04:43:10 CET 2018, data05254a.tar >>> Mon Mar 05 04:53:55 CET 2018, data05291a.tar >>> Mon Mar 05 04:48:31 CET 2018, data05274a.tar >>> Mon Mar 05 04:50:47 CET 2018, data05283a.tar >>> Mon Mar 05 04:40:27 CET 2018, data05249a.tar >>> Mon Mar 05 04:52:22 CET 2018, data05288a.tar >>> Mon Mar 05 04:49:30 CET 2018, data05278a.tar >>> Mon Mar 05 04:28:56 CET 2018, data05233a.tar >>> Mon Mar 05 04:31:18 CET 2018, data05236a.tar >>> Mon Mar 05 04:52:02 CET 2018, data05287a.tar >>> Tue Jan 16 06:25:35 CET 2018, data01538a.tar.bak >>> Mon Mar 05 04:47:51 CET 2018, data05272a.tar >>> Mon Mar 05 04:36:02 CET 2018, data05243a.tar >>> Mon Mar 05 04:35:14 CET 2018, data05242a.tar >>> Mon Mar 05 04:42:46 CET 2018, data05253a.tar >>> Mon Mar 05 04:44:55 CET 2018, data05260a.tar >>> Mon Mar 05 04:39:07 CET 2018, data05247a.tar >>> Mon Mar 05 04:46:50 CET 2018, data05268a.tar >>> Mon Mar 05 04:44:06 CET 2018, data05256a.tar >>> Mon Mar 05 04:45:33 CET 2018, data05263a.tar >>> Mon Mar 05 04:53:22 CET 2018, data05290a.tar >>> Thu Dec 14 07:14:54 CET 2017, repo.lock >>> Mon Mar 05 04:50:19 CET 2018, data05281a.tar >>> Mon Mar 05 04:49:13 CET 2018, data05277a.tar >>> Mon Mar 05 04:45:51 CET 2018, data05264a.tar >>> Mon Mar 05 04:33:40 CET 2018, data05240a.tar >>> Mon Mar 05 04:46:08 CET 2018, data05265a.tar >>> Mon Mar 05 04:32:55 CET 2018, data05239a.tar >>> Mon Mar 05 04:41:04 CET 2018, data05250a.tar >>> Fri Mar 02 04:57:25 CET 2018, data05022a.tar.2.bak >>> Mon Mar 05 04:45:18 CET 2018, data05262a.tar >>> Mon Mar 05 04:30:20 CET 2018, data05235a.tar >>> Mon Mar 05 04:44:44 CET 2018, data05259a.tar >>> Mon Mar 05 04:51:43 CET 2018, data05286a.tar >>> Mon Mar 05 04:54:03 CET 2018, journal.log >>> Fri Mar 02 04:57:24 CET 2018, data05022a.tar.bak >>> Mon Mar 05 04:32:32 CET 2018, data05238a.tar >>> Mon Mar 05 04:28:18 CET 2018, data05232a.tar >>> Mon Mar 05 04:45:07 CET 2018, data05261a.tar >>> Mon Mar 05 04:47:38 CET 2018, data05271a.tar >>> Mon Mar 05 04:49:48 CET 2018, data05279a.tar >>> Mon Mar 05 04:53:56 CET 2018, manifest >>> Mon Mar 05 04:34:30 CET 2018, data05241a.tar >>> Mon Mar 05 04:46:20 CET 2018, data05266a.tar >>> Mon Mar 05 04:43:41 CET 2018, data05255a.tar >>> Mon Mar 05 04:38:21 CET 2018, data05246a.tar >>> Mon Mar 05 04:48:14 CET 2018, data05273a.tar >>> Mon Mar 05 04:52:50 CET 2018, data05289a.tar >>> Mon Mar 05 04:42:19 CET 2018, data05252a.tar >>> Mon Mar 05 04:41:47 CET 2018, data05251a.tar >>> Mon Mar 05 04:47:07 CET 2018, data05269a.tar >>> Mon Mar 05 04:50:34 CET 2018, data05282a.tar >>> Mon Mar 05 04:47:21 CET 2018, data05270a.tar >>> Mon Mar 05 04:39:50 CET 2018, data05248a.tar >>> Mon Mar 05 04:29:40 CET 2018, data05234a.tar >>> Mon Mar 05 04:44:19 CET 2018, data05257a.tar >>> Mon Mar 05 04:44:30 CET 2018, data05258a.tar >>> Mon Mar 05 04:32:13 CET 2018, data05237a.tar >>> Mon Mar 05 04:48:43 CET 2018, data05275a.tar >>> Mon Mar 05 04:50:05 CET 2018, data05280a.tar >>> Mon Mar 05 04:37:30 CET 2018, data05245a.tar >>> Mon Mar 05 04:51:02 CET 2018, data05284a.tar >>> Mon Mar 05 04:46:33 CET 2018, data05267a.tar >>> Mon Mar 05 04:54:03 CET 2018, gc.log >>> size 16.1 GB (16081145586 bytes) >>> removed files [data05228a.tar, data05222a.tar, data05167a.tar, >>> data05184a.tar, data05214a.tar, data05192a.tar, data05200a.tar, >>> data05198a.tar, data05183a.tar, data05189a.tar, data05205a.tar, >>> data05175a.tar, data05230a.tar, data05206a.tar, data05191a.tar, >>> data05197a.tar, data05174a.tar, data05177a.tar, data05188a.tar, >>> data05171a.tar, data05182a.tar, data05169a.tar, data05224a.tar, >>> data05227a.tar, data05221a.tar, data05213a.tar, data05193a.tar, >>> data05185a.tar, data05190a.tar, data05216a.tar, data05219a.tar, >>> data05196a.tar, data05199a.tar, data05195a.tar, data05211a.tar, >>> data05187a.tar, data05203a.tar, data05208a.tar, data05186a.tar, >>> data05172a.tar, data05180a.tar, data05202a.tar, data05210a.tar, >>> data05178a.tar, data05209a.tar, data05217a.tar, data05225a.tar, >>> data05194a.tar, data05207a.tar, data05218a.tar, data05229a.tar, >>> data05220a.tar, data05168a.tar, data05223a.tar, data05226a.tar, >>> data05179a.tar, data05176a.tar, data05231a.tar, data05170a.tar, >>> data05215a.tar, data05201a.tar, data05181a.tar, data05173a.tar, >>> data05204a.tar, data05212a.tar] >>> added files [data05259a.tar, data05267a.tar, data05245a.tar, >>> data05237a.tar, data05253a.tar, data05261a.tar, data05283a.tar, >>> data05244a.tar, data05258a.tar, data05291a.tar, data05236a.tar, >>> data05275a.tar, data05260a.tar, data05252a.tar, data05289a.tar, >>> data05232a.tar, data05246a.tar, data05243a.tar, data05238a.tar, >>> data05235a.tar, data05240a.tar, data05279a.tar, data05262a.tar, >>> data05268a.tar, data05276a.tar, data05251a.tar, data05284a.tar, >>> data05270a.tar, data05254a.tar, data05273a.tar, data05281a.tar, >>> data05265a.tar, data05249a.tar, data05257a.tar, data05250a.tar, >>> data05256a.tar, data05264a.tar, data05239a.tar, data05287a.tar, >>> data05272a.tar, data05242a.tar, data05248a.tar, data05241a.tar, >>> data05280a.tar, data05247a.tar, data05255a.tar, data05278a.tar, >>> data05233a.tar, data05286a.tar, data05271a.tar, data05274a.tar, >>> data05288a.tar, data05263a.tar, data05266a.tar, data05285a.tar, >>> data05290a.tar, data05277a.tar, data05282a.tar, data05269a.tar, >>> data05234a.tar] >>> Compaction succeeded in 27.29 min (1637s). >>> >>> ``` >>> >>> Greets, >>> Roy >
