Hi ,
We have started Se Garbage Collection and S3 size is more than 2 TB.
We are facing memory issues while executing GC even when we have given 11 GB of
memory to the application.
Code is:
final MarkSweepGarbageCollector gc =
documentNodeStore.createBlobGarbageCollector(seconds, repository.toString(),
wb, new DefaultStatisticsProvider(Executors.newScheduledThreadPool(1)));
gc.collectGarbage(markOnly);
final OperationsStatsMBean stats = gc.getOperationStats();
log.info(
"number deleted : " + stats.numDeleted() + " Size deleted : " +
stats.sizeDeleted());
The exceptions are :
java.lang.RuntimeException: org.postgresql.util.PSQLException: Ran out of
memory retrieving query results.
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$3.iterator(RDBDocumentStore.java:1876)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBBlobReferenceIterator.getIteratorOverDocsWithBinaries(RDBBlobReferenceIterator.java:48)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.loadBatch(BlobReferenceIterator.java:64)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:52)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:36)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.jackrabbit.oak.plugins.document.DocumentBlobReferenceRetriever.collectReferences(DocumentBlobReferenceRetriever.java:49)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.iterateNodeTree(MarkSweepGarbageCollector.java:600)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.mark(MarkSweepGarbageCollector.java:397)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.markAndSweep(MarkSweepGarbageCollector.java:347)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.collectGarbage(MarkSweepGarbageCollector.java:235)
at
com.springernature.mango.cms.common.helper.RepositoryUtil.s3GarbageCollection(RepositoryUtil.java:324)
at
com.springernature.mango.cms.services.rest.service.ServicesController.lambda$runS3GC$1(ServicesController.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2145)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at
org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
at
org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
at sun.reflect.GeneratedMethodAccessor49.invoke
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tomcat.jdbc.pool.interceptor.StatementDecoratorInterceptor$StatementProxy.invoke(StatementDecoratorInterceptor.java:237)
at com.sun.proxy.$Proxy93.executeQuery
at sun.reflect.GeneratedMethodAccessor49.invoke
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210)
at com.sun.proxy.$Proxy93.executeQuery
at sun.reflect.GeneratedMethodAccessor49.invoke
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy93.executeQuery
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC$ResultSetIterator.<init>(RDBDocumentStoreJDBC.java:591)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC.queryAsIterator(RDBDocumentStoreJDBC.java:558)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$3.iterator(RDBDocumentStore.java:1865)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBBlobReferenceIterator.getIteratorOverDocsWithBinaries(RDBBlobReferenceIterator.java:48)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.loadBatch(BlobReferenceIterator.java:64)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:52)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:36)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.jackrabbit.oak.plugins.document.DocumentBlobReferenceRetriever.collectReferences(DocumentBlobReferenceRetriever.java:49)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.iterateNodeTree(MarkSweepGarbageCollector.java:600)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.mark(MarkSweepGarbageCollector.java:397)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.markAndSweep(MarkSweepGarbageCollector.java:347)
at
org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.collectGarbage(MarkSweepGarbageCollector.java:235)
at
com.springernature.mango.cms.common.helper.RepositoryUtil.s3GarbageCollection(RepositoryUtil.java:324)
at
com.springernature.mango.cms.services.rest.service.ServicesController.lambda$runS3GC$1(ServicesController.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.postgresql.core.PGStream.receiveTupleV3(PGStream.java:401)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2141)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at
org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
at
org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
at sun.reflect.GeneratedMethodAccessor49.invoke
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tomcat.jdbc.pool.interceptor.StatementDecoratorInterceptor$StatementProxy.invoke(StatementDecoratorInterceptor.java:237)
at com.sun.proxy.$Proxy93.executeQuery
at sun.reflect.GeneratedMethodAccessor49.invoke
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210)
at com.sun.proxy.$Proxy93.executeQuery
at sun.reflect.GeneratedMethodAccessor49.invoke
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy93.executeQuery
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC$ResultSetIterator.<init>(RDBDocumentStoreJDBC.java:591)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC.queryAsIterator(RDBDocumentStoreJDBC.java:558)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$3.iterator(RDBDocumentStore.java:1865)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBBlobReferenceIterator.getIteratorOverDocsWithBinaries(RDBBlobReferenceIterator.java:48)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.loadBatch(BlobReferenceIterator.java:64)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:52)
at
org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:36)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.jackrabbit.oak.plugins.document.DocumentBlobReferenceRetriever.collectReferences(DocumentBlobReferenceRetriever.java:49)
So I wanted to ask is there any other way how can we restrict this and
successfully run S3 Garbage Collection.
Thanks.
**********************************************************************
Disclaimer: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Springer Nature Technology and Publishing Solutions Private Limited
does not accept liability for any statements made which are clearly the
sender's own and not expressly made on behalf of Springer Nature Technology and
Publishing Solutions Private Limited or one of their agents.
Please note that Springer Nature Technology and Publishing Solutions Private
Limited and their agents and affiliates do not accept any responsibility for
viruses or malware that may be contained in this e-mail or its attachments and
it is your responsibility to scan the e-mail and attachments (if any).
Springer Nature Technology and Publishing Solutions Private Limited. Registered
office: Upper Ground Floor, Wing B, Tower 8, Magarpatta City SEZ, Hadapsar Pune
MH 411013 IN
Registered number: U72200PN2006FTC128967
