[
https://issues.apache.org/jira/browse/DAFFODIL-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851658#comment-17851658
]
Steve Lawrence commented on DAFFODIL-2901:
------------------------------------------
I believe I have tracked it down. Although there is always a small memory leak,
we leak significantly more if validationMode="on" in a TDML test suite. Here's
what I believe is going on:
In the TDML Runner, each TestCase instance has a "var processor" variable,
which is used to keep track of the DataProcessor used for that test case. The
main reason this is because so many functions need access to the processor that
it's easiest to just set it once as a member and then access the member when
it's needed instead of changing a bunch of function definitions and calls to
accept a processor.
However, each TestCase instance lasts the entire lifetime of the TestSuite,
which means the DataProcessor that is stored in each TestCase also lasts the
lifetime of a TestSuite. I.e. DataProcessor are never garbage collected until
the end of a TestSuite.
And if a test case calls one of the withXYZ functions, it will create a copy of
the DataProcessor and store that in the test case. So essentially, it's very
easy to have a bunch of copies of DataProcessors store in a TestSuite.
Fortulatey, copies of DataProcessors tend to be pretty small, because they tend
to share most of the state (e.g. compiled parser/unparse).
The one place where this isn't the case is with validators. Validators are not
copied when withXYZ is called, so each DataProcessor has its own copy of a
validator, even if it is the same as a previous validator. For large schemas,
these validators can be very large. And since DataProcessor are not garbage
collected during a TestSuite, neither are these large and dupliate validators.
So if validation is on with lots of test cases, we could easily run into memory
issues.
So we probably need two fixes:
# When a TestCase finishes running a test, it should set the "processor"
variable to null so it can be garbage collected. Or alternatively, we just
start passing the DataProcessor around to all the functions that need it.
# Modify the DataProcessor so the withXYZ functions also copy the validators.
They validators are supposed to be thread safe, so this should be fine. In
addition to avoiding memory leaks, this should also make TDML tests much faster
since they will not need to compile a validator for every new DataProcessor if
the validators are the same.
> Unable to run many tests in single tdml suite without large amounts of memory
> -----------------------------------------------------------------------------
>
> Key: DAFFODIL-2901
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2901
> Project: Daffodil
> Issue Type: Bug
> Components: TDML Runner
> Affects Versions: 3.7.0
> Reporter: Olabusayo Kilo
> Priority: Major
>
> The test suite I'm working with contains 600+ tests, and run just fine in
> IntelliJ (which has a heap size of 4G), but fails to run with `sbt test`
> failing with an out of memory error. This error doesn't go away until we
> increase the sbt heap size to 8G. Then the tests run successfully.
> Obvious workaround is just to do `sbt -mem 8192 test`, but it would be ideal
> if this wasn't necessary, especially since the tests are run sequentially and
> not in parallel. A memory leak in the runner seems like a likely suspect.
> We would obviously need to profile an `sbt test` run using either JProfiler
> or Intellij's built in profiler. Example schemas with lots of tests are the
> JREAP schema and the P8 schema.
> According to [~jadams_tresys] , this issue doesn't occur if the tests are
> split out into multiple test suites
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
