[
https://issues.apache.org/jira/browse/DAFFODIL-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18013506#comment-18013506
]
Steve Lawrence commented on DAFFODIL-2901:
------------------------------------------
I've confirmed that the description in issue #2 was correct when it was
written. Validators were not copied. Instead only the validation *mode* was
copied. This meant that if multiple TDML tests had validationMode="full", then
each test would copy the fact that the validation mode was full, but the
DataProcessor would still need to create a unique validator.
This is partially solved with the changes to the API in 4.0.0 and the new
withValidation API. This new API creates a validator when withValidation is
called and it is stored in the DataProcessor so that it is copied if any other
withXYZ functions are called. So that issue is resolved.
However, how the TDML runner uses this API causes issues. The problem is that
the TDML runner calls withValidation twice, once when compiling the schema
(with whatever the default validation mode is) and again when the test is run
(with the tests specific validation mode). This second withValidation will
cause the Validator to be re-created even if it's exactly the same as what was
used to compile the schema, which means every test will still end up recreating
the validator. So #2 is still an issue, it's just a problem int he TDMLRunner
instead of in the DataProcessor.
For any TMDL files with defaultValidation="full" or "on", that could cause a
lot of extra work, especially if the schemas are large.
> Unable to run many tests in single tdml suite without large amounts of memory
> -----------------------------------------------------------------------------
>
> Key: DAFFODIL-2901
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2901
> Project: Daffodil
> Issue Type: Bug
> Components: TDML Runner
> Affects Versions: 3.7.0
> Reporter: Olabusayo Kilo
> Assignee: Steve Lawrence
> Priority: Major
>
> The test suite I'm working with contains 600+ tests, and run just fine in
> IntelliJ (which has a heap size of 4G), but fails to run with `sbt test`
> failing with an out of memory error. This error doesn't go away until we
> increase the sbt heap size to 8G. Then the tests run successfully.
> Obvious workaround is just to do `sbt -mem 8192 test`, but it would be ideal
> if this wasn't necessary, especially since the tests are run sequentially and
> not in parallel. A memory leak in the runner seems like a likely suspect.
> We would obviously need to profile an `sbt test` run using either JProfiler
> or Intellij's built in profiler. Example schemas with lots of tests are the
> JREAP schema and the P8 schema.
> According to [~jadams_tresys] , this issue doesn't occur if the tests are
> split out into multiple test suites
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
