[
https://issues.apache.org/jira/browse/DERBY-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904620#comment-16904620
]
Marco edited comment on DERBY-7049 at 8/11/19 11:58 AM:
--------------------------------------------------------
Thanks a lot!
But I won't extend DataNucleus to use such a special solution. Instead, I
refactored my code to organise the IDs in ranges and then create a query like
{code:sql}
SELECT ... FROM ...
WHERE
"a0"."discriminator" = 'SsNormalFile' AND
(
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?)
)
{code}
The number of ranges in my queries is hard-coded to 10 and I simply repeat the
query multiple times (in the Java code), if there are more than 10 ranges.
This reduces the quantity of unique queries (analysed via derby.log and
StatementLogReadingVTI.java as explained by you above) from multiple thousand
(and potentially unlimited) to less than 200. GREAT! Thanks again for this
valuable hint.
However, I thought that this would solve the problem, but my friend still
encountered the same exception again :-( His derby.log had a size of over 50
GB, when it happened, so that he decided to throw it away and try it again for
a shorter time (only about 2 hours -- the queries should be the same).
Unfortunately, though, in this smaller log (only about 5 GB), there were still
only less than 200 unique queries. I'm thus not sure, whether the problem is a
different one, now, or whether derby compiles new classes even though the very
same query-text was already used before (and there are only roughly 200).
Is the query-class-cache maybe somehow limited? Maybe derby evicts entries from
this cache, but the classes are still in the ClassLoader? Or is there maybe
some other constraint on the query-class-reusing-code, e.g. a maximum
query-text-length? And the query-text-to-class-cache is global, i.e. shared
across transactions, right? What about multiple databases? Are the classes
shared across different DBs, too?
If there are constraints like cache-size or query-text-length, can this be
configured using some system-properties?
P.S.: Sorry for my late reaction. I'm usually very busy with paid work and
don't have much time to help my friend fixing his troubles with my hobby
project ;-)
was (Author: nlmarco):
Thanks a lot!
But I won't extend DataNucleus to use such a special solution. Instead, I
refactored my code to organise the IDs in ranges and then create a query like
{code:sql}
SELECT ... FROM ...
WHERE
"a0"."discriminator" = 'SsNormalFile' AND
(
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?) OR
(? <= "a0"."id" AND "a0"."id" <= ?)
)
{code}
The number of ranges in my queries is hard-coded to 10 and I simply repeat the
query multiple times (in the Java code), if there are more than 10 ranges.
This reduces the quantity of unique queries (analysed via derby.log and
StatementLogReadingVTI.java as explained by you above) from multiple thousand
(and potentially unlimited) to less than 200. GREAT! Thanks again for this
valuable hint.
However, I thought that this would solve the problem, but my friend still
encountered the same exception again :-( His derby.log had a size of over 50
GB, when it happened, so that he decided to throw it away and try it again for
a shorter time (only about 2 hours -- the queries should be the same).
Unfortunately, though, in this smaller log (only about 5 GB), there were still
only less than 200 unique queries. I'm thus not sure, whether the problem is a
different one, now, or whether derby compiles new classes even though the very
same query-text was already used before (and there are only roughly 200).
Is the query-class-cache maybe somehow limited? Maybe derby evicts entries from
this cache, but the classes are still in the ClassLoader? Or is there maybe
some other constraint on the query-class-reusing-code, e.g. a maximum
query-text-length?
If there are constraints like cache-size or query-text-length, can this be
configured using some system-properties?
P.S.: Sorry for my late reaction. I'm usually very busy with paid work and
don't have much time to help my friend fixing his troubles with my hobby
project ;-)
> OutOfMemoryError: Compressed class space
> ----------------------------------------
>
> Key: DERBY-7049
> URL: https://issues.apache.org/jira/browse/DERBY-7049
> Project: Derby
> Issue Type: Bug
> Components: SQL
> Affects Versions: 10.13.1.1
> Reporter: Marco
> Priority: Major
> Attachments: StatementLogReadingVTI.java
>
>
> After a few days of working with an embedded Derby database (currently
> version 10.13.1.1 on Oracle Java 1.8.0_201-b09), the following error occurs:
> *java.lang.OutOfMemoryError: Compressed class space*
> {code:java}
> java.lang.OutOfMemoryError: Compressed class space
> at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_201]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[na:1.8.0_201]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:642) ~[na:1.8.0_201]
> at
> org.apache.derby.impl.services.reflect.ReflectLoaderJava2.loadGeneratedClass(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at
> org.apache.derby.impl.services.reflect.ReflectClassesJava2.loadGeneratedClassFromData(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at
> org.apache.derby.impl.services.reflect.DatabaseClasses.loadGeneratedClass(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at
> org.apache.derby.impl.services.bytecode.GClass.getGeneratedClass(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at
> org.apache.derby.impl.sql.compile.ExpressionClassBuilder.getGeneratedClass(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.compile.StatementNode.generate(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
> ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
> ~[derby-10.13.1.1.jar:na]
> at
> org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.<init>(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown
> Source) ~[derby-10.13.1.1.jar:na]
> at
> org.datanucleus.store.rdbms.datasource.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:259)
> ~[datanucleus-rdbms-4.0.12.jar:na]{code}
> I tried to solve the problem by periodically shutting down the database,
> because I read that the generated classes as well as all other allocated
> resources should be released when the DB is shut-down.
> I thus perform the following code once per roughly 20 minutes:
> {code:java}
> String shutdownConnectionURL = connectionURL + ";shutdown=true";
> try {
> DriverManager.getConnection(shutdownConnectionURL);
> } catch (SQLException e) {
> int errorCode = e.getErrorCode();
> if (DERBY_ERROR_CODE_SHUTDOWN_DATABASE_SUCCESSFULLY != errorCode &&
> DERBY_ERROR_CODE_SHUTDOWN_DATABASE_WAS_NOT_RUNNING != errorCode) {
> throw new RuntimeException(e);
> }
> }
> {code}
> Unfortunately, this has no effect :( The OutOfMemoryError still occurs after
> about 2 days. Do I assume correctly that the above code _should_ properly
> shut-down the database? And do I assume correctly that this shut-down should
> release the generated classes?
> IMHO, it is already a bug in Derby that I need to shut-down the database at
> all in order to prevent it from piling up generated classes. Shouldn't it
> already release the generated classes at the end of each transaction? But
> even if I really have to shut-down the DB, it is certainly a bug that the
> classes are still kept in "compressed class space" even after the shut-down.
> I searched the release notes and the existing bugs (here in JIRA) and did not
> find anything related to this {{OutOfMemoryError}}. Hence, I open this
> bug-report, now.
> This issue was originally reported in
> [subshare#74|https://github.com/subshare/subshare/issues/74], but it is IMHO
> clearly a Derby bug.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
