This fix has been merged to master. We'll have to watch carefully about runtime performance.
I tested against all schemas that can be easily tested and they all work. Compilation times are not "blink of an eye", but they're under 10 seconds even for larger schemas. That still means the tests for a big schema can take a bit, as often the schema gets compiled several times in variations for such tests. I am still planning a fix that cleans up the various factories which insert backpointers, and removes all the unique-backpointer cruft which in principle isn't used anymore. That 's about cleanup and maintainability though, and I don't think waiting for it is critical. The namespace minimization issue (which is a bug that we're not hitting in any tests nor schemas currently) and the isHidden issue (ditto on why we're not hitting it) still exist and need to be addressed. ...mikeb ________________________________ From: Beckerle, Mike <mbecke...@tresys.com> Sent: Sunday, February 16, 2020 12:09 AM To: dev@daffodil.apache.org <dev@daffodil.apache.org> Cc: Sood, Harinder <hs...@tresys.com> Subject: Re: progress on schema compiler space/speed issue Compiler space/speed issue is *solved* to my satisfaction. All tests pass. I also tested all DFDLSchemas from github, and all our FOUO schemas and they all work also. The PR is ready for "real" review now. I removed the "WIP" status on it. I have not yet updated the design notes to match the implementation. The pending fixes to isHidden, and to namespace binding minimization - basically those fixes are needed, but we have no tests nor schemas that run into the problems. E.g, for isHidden: we have no tests nor schemas where a single group is both hidden and non-hidden. Similarly we have no tests nor schemas where namespace binding minimization must be different on two different inclusions of a common element. We could *not fix* those issues and create JIRA tickets for lifting them. But we need tests to at least exercise the problems, so we can characterize what happens if we do these unimplemented things. ________________________________ From: Beckerle, Mike Sent: Thursday, February 13, 2020 10:29 PM To: dev@daffodil.apache.org <dev@daffodil.apache.org> Subject: progress on schema compiler space/speed issue See https://github.com/apache/incubator-daffodil/pull/324 I still have debugging to do, and I'm not happy with the non-type-safety, and maintainability headaches this creates, but I was able to get rid of the combinatorial explosion with a modest number of lines of code changed. But most schemas work now, and schema compilation is faster, and is seemingly linear in schema size with these changes.
