Ok, well if you want to undertake this, here's a few things I thought of when scoping the complexity of this:
1) built-in-formats.xsd is designed to be included, not imported. Its replacement wants to be the same otherwise you'll have to make many more edits to every TDML file or schema that includes it beyond just changing the name to something we like. The reason for the include is just convenience - by including a format definition into the current schema's target namespace you avoid the need to define yet another namespace prefix for the namespace being imported. Most likely this prefix is used exactly once, so it's a lot of clutter for very little. Basically, namespace clutter adds little value for most people, and while you need it just in case of name conflicts, it's nice to just avoid it. A No-namespace schema can avoid namespace prefix definitions entirely this way, for example. There are big block comments at the top of the IBM generalpurposeformat, and DFDLGeneralFormatDefinition.xsd that are very "import" specific that need to change for an "include-oriented" schema. 2) DFDLGeneralFormatDefinition.xsd exists today, but to my knowledge it is entirely unused. It clearly looks to be based on the one IBM defined, just renamed and relocated. We should remove this and replace it with what we want, which I believe should have the exact same property definitions as daffodilTest1 has in built-in-formats.xsd I think the defaults in built-in-formats.xsd are good and useful, and changes to that could lead to subtle bugs so we should just capture that set of property definitions, name the file say, DFDLGeneralFormat.dfdl.xsd, name the actual named format "GeneralFormat". This file should be an included (i.e., no-namespace) schema. That way you'll change built-in-formats.xsd to DFDLGeneralFormat.dfdl.xsd, and tns:daffodilTest1 to tns:GeneralFormat, and that's all that would need to change. 3) SchemaUtils.scala has a method that is used to create test schemas, and that's where the implicit import of built-in-format is. So clearly that has to change also. 4) There's a jira ticket for changing all these schemas to follow the ".dfdl.xsd" convention. We should do that too if we're changing these things. ________________________________ From: Steve Lawrence <slawre...@apache.org> Sent: Thursday, February 15, 2018 7:41:59 AM To: dev@daffodil.apache.org; Mike Beckerle Subject: Re: Need to revert change to DAFFODIL-1897 (built-in-formats.xsd move to test) Although our testing infrastructure appears to heavily rely on built-in-formats.xsd, I still think it needs be to deprecated and eventually removed. Having real world schemas that reference "tns:daffodilTest1" is clearly wrong. And we probably don't want TDML files using that since those are obvious starting points as a reference when building a new schema. So I feel this must be deprecated, and if we are going to deprecated it, I'd rather go through the headache now and be done with it. To deprecate this without too much pain, it seems reasonable to me to create our own version of IBM's GeneralPurposeFormat (which actually has odd choices for defaults IMO) and modify our TDMLRunner to use that. I think modifying all our tests to reference that should be pretty easily scriptable. I'm fine putting built-in-formats back in and adding some backwards compatibility that isn't quite so aggressive as what let to this change. If done right, I'm a little more optimistic about this not causing subtle bugs since it's would essentially just be a rename. I'll spend a little time this morning investigating this change. - Steve On 02/14/2018 04:46 PM, Mike Beckerle wrote: > This change is causing no end of headaches. I have wasted a day already > fixing things that are now broken on account of this. > > > Turns out built-in-formats.xsd is *not* only used by our internal tests. > > > It is part of the TDML runner's tdml:defineSchema infrastructure - included > into every such embedded schema implicitly. > > TDML runner is callable/usable from our CLI, which means it is part of the > published non-test daffodil jars. > > > built-in-formats.xsd and daffodilTest1 are also is "out there" in that they > appear on slides that are full of small examples that have been published on > the web for a while now. > > > I think this horse has left the barn, and we're committed to supporting > built-in-formats.xsd as is, in the original location as part of daffodil-lib. > > > If we want to deprecate it in the future in favor of some better named thing > I'm fine with that. But changing this is going to require a deprecate, and > slow-cut over, otherwise it requires a big comprehensive sweep hundreds of > lines of code change, subtle bugs come up. And the changes have to be done in > lots of things that aren't even part of the apache daffodil code base. > > > Also unless DAFFODIL-900 is fixed (missing diagnostic on dfdl:ref not found), > changing any of this stuff is very dangerous. > > >
