[
https://issues.apache.org/jira/browse/DAFFODIL-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851648#comment-17851648
]
Steve Lawrence commented on DAFFODIL-2901:
------------------------------------------
I believe I have tracked it down. Although there is always a small memory leak,
we leak significantly more if validationMode="on" in a TDML test suite. Here's
what I believe is going on:
# In the TDML Runner, each TestCase instance has a "var processor" variable,
which is used to keep track of the DataProcessor used for that test case. The
main reason this is because so many functions need access to the processor that
it's easiest to just set it once as a member and then access the member when
it's needed instead of changing a bunch of function definitions and calls to
accept a processor. However, each TestCase instance lasts the entire lifetime
of the TestSuite, which means the DataProcessor that is stored in each TestCase
also lasts the lifetime of a TestSuite. I.e. DataProcessor are never garbage
collected until the end of a TestSuite.
This isn't great, but in general not too big of a deal. DataProcessors are
often fairly small or a DataProcess is copied with something like withXYZ and
most of the contents (especially the large parsers) are shared among the
copies. So still a memory leak, but a relatively small one.
# The one place where this really causes a problem is with validators. When we
copy a DataProcessor with one of the withXYZ functions, we do not copy the
validator. So when a DataProcessor needs to validate it will create a new
instance of that validator if it hasn't already been created.
> Unable to run many tests in single tdml suite without large amounts of memory
> -----------------------------------------------------------------------------
>
> Key: DAFFODIL-2901
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2901
> Project: Daffodil
> Issue Type: Bug
> Components: TDML Runner
> Affects Versions: 3.7.0
> Reporter: Olabusayo Kilo
> Priority: Major
>
> The test suite I'm working with contains 600+ tests, and run just fine in
> IntelliJ (which has a heap size of 4G), but fails to run with `sbt test`
> failing with an out of memory error. This error doesn't go away until we
> increase the sbt heap size to 8G. Then the tests run successfully.
> Obvious workaround is just to do `sbt -mem 8192 test`, but it would be ideal
> if this wasn't necessary, especially since the tests are run sequentially and
> not in parallel. A memory leak in the runner seems like a likely suspect.
> We would obviously need to profile an `sbt test` run using either JProfiler
> or Intellij's built in profiler. Example schemas with lots of tests are the
> JREAP schema and the P8 schema.
> According to [~jadams_tresys] , this issue doesn't occur if the tests are
> split out into multiple test suites
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
