Vincent, I think we disagree on two major aspects which are kind of related:
- writing document structure and metadata should be doable by hand editing as opposed to writing into the wiki alone: - I believe this is a fundamental aspect of source editing and should be supported (hence the wish to a validatable widespread syntax) - you seem to indicate that the current process you've agreed upon has discouraged it (but, at the same time, you indicate you wish to do it for YAML) - writing XML can be done by hand and is easy: - I am a strong defender of this: all editors I've worked with (IntelliJ IDEA, jEdit, BBEdit, XCode, maybe not vim) support XML very well so that it's just careless to commit a && (not to mention that mvn would not validate it, and would not build it if with my patch, simply because it parses). Clearly, editing in a textbox can get you there, same with MS Word! - you seem to say that bad things happen with such commits as not even valid XML documents and seem to motivate that to move to another format (I feel YAML a zillion less predictable than XML because of whitespace handling btw). Paul PS: when I say editing XML is for sane people, I don't mean everything should be XML, which is why I proposed the XAR extension to include XInclude, textinclude and binaryInclude. PPS: another, maybe deeper, disagreement might be the need to round-trip (server, source) vs the need to source-edit: hand-tuning the format of a file (e.g. in XML or in JSON) is easily in contradiction with a server generation from a structured source. Maybe this is the hard bone. > Vincent Massol <mailto:[email protected]> > 27 August 2016 at 17:50 >> On 27 Aug 2016, at 16:46, Vincent Massol <[email protected]> wrote: >> >>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>> >>> Thanks for having this extra thread. >>> I think this thread is much more important than starting to design >>> something already. >>>> Issues with the XAR format >>>> ====================== >>>> >>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>> editor to edit content >>>> * XML also requires content to be XML-encoded and thus is really not >>>> easy to make modification (there’s a risk of breaking the XML easily) >>> I completely disagree with these two statements. >> I probably didn’t use the right words because that’s the reason why I think >> that's you started the xinclude proposal :) >> >> AFAIK you started the xinclude proposal because you wanted to be able to >> edit content with specific editor (js, css, etc)… which is exactly what the >> second part of the first point is about. >> >> Let me rephrase the first sentence: >> >> * XML is not an easy to edit format for human beings (it’s very verbose and >> easy to make mistakes: missing encoding, etc). It’s also very hard to edit >> with a plain text editor. >> >> As for the 2nd sentence, I don’t see how you can disagree since it’s a >> fact... >> >>> XML is easy to edit and is supported by very very many editors and IDEs. >>> It can also be validated. >> What you’re saying is very theoretical. The practice (that we’ve had for 10 >> years of using the XAR format) is that our XML format that is hard to edit >> and can break easily (as was proven numerous times by our committers and >> contributors). It’s actually so dangerous that we’ve had to develop a >> process which goes like this: >> * Never edit the xml directly >> * Always import it into a running XWiki instance, make the modifications >> there and export from the wiki into XAR >> * Then unpack the XAR into the source directory and run mvn xar:format to go >> back to the original format. >> >> Nobody is using a pure XML editor with validation. We are all using Java >> IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as >> plain text and that’s what we’re doing. No developer I know of is using an >> XML editor in a structured way (just too painful and complex to navigate the >> structure). > > To be more specific, the main issue we’ve had is contributors/committers who > committed unencoded content, such as && instead of && or > instead of > > > > Now, to be accurate, if you remove the problem with the encoding (which can > be removed IMO with CDATA) then we never touch much of the rest of the > metadata. > > In practice it would be nice if most of it could be generated by the maven > plugin. In practice we don’t need much specific data (for a pure doc, it’s a > bit more for xclass/xobjects): title, reference, syntax, parent, hidden, and > language/translation. Syntax and reference could be computed from the > directory structure. Parent could too. And hidden could default to visible by > default. That said it doesn’t matter that much since the process is to export > from a running xwiki instance (we need that to validate that it works at > execution time, or we’d need good unit/integration tests for pages). > > So once we take the content out, the format of the metadata doesn’t matter > that much probably since we’re not going to author it from scratch anyway > (it’ll come from exported wiki pages). > > So I guess XML, even though very verbose, could still be ok. But XML for doc > content or xproperty textareas, or attachments is a sure no go. > > Thanks > -Vincent > >> (see below) >> >>> XML can be written in a very elegant and readable fashion if you care >>> for it. >>> Generally however, XML is produced form other structured information in >>> a way that does not help readability. >>>> Can you see more issues? >>> The problem is how we put *everything* into XML. >>> (you get the same horror if you use Caleb's tools xardump and do not >>> tune anything: the resulting javascript is horrible.) >>>> Use cases for an alternative filesystem format >>>> =================================== >>>> >>>> (some UC taken from >>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>> >>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>> an IDE style view >>>> * UC2: it should be easy to add content (a new script or attachment on >>>> an existing structure). It should allow using specific editors for >>>> different content types, e.g. if a page content is in markdown, it >>>> should be editable with a MD editor, js and css should be editable >>>> with web editors, etc. >>> UC2.1: Attachments should also be present as standalone files. >>>> * UC3: It should be possible to build a packaged version of the >>>> sources with Maven >>>> * UC4: It should be possible to import the packaged version into a >>>> running XWiki instance >>>> * UC5: It should be possible to export a portion of a running XWiki >>>> instance in this format >>>> * UC6: This format should be able to fully replace the XAR format . >>>> The new format should support at least all features supported by the >>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>> so that the XAR format can be swapped out by introducing extension >>>> points/APIs. The idea would be to deprecate the XAR format and >>>> introduce this new format instead, and the 2 formats should be avle to >>>> cohabit next to each other in XWiki. >>>> * UC7: When importing in a wiki and exporting again (without making >>>> any change in the wiki), it should generate an identical structure and >>>> content, with no difference. >>> I do believe that UC7 is not doable in full generality. >> Why not, this is what we have with the XAR format actually and UC7 is >> actually already contained in UC6 (but it’s better to be explicit)? >> >>> Any more? >>> >>> UC8: the core representation should be using a syntax that is widely >>> spread and completely specified (i.e. we should not invent another >>> syntax for this) >> This is not a requirement for me. The syntax should be easy to write into, >> especially using a plain text editor. YAML for example is a perfectly valid >> syntax for me. >> >>> UC9: the system should make an archival process a widespread practice, >>> in the form of zipped files probably. >> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I >> didn’t want to limit us to that but it’s probably going to be zip anyway. >> >>> UC10: developers should have the discretion to decide when a component >>> needs to be in a separate file or not. That is, small text fragments and >>> even small attachments should be includable within bigger files >> I don’t agree with this one. I’d like convention over flexibility, i.e. a >> fixed format on which tools can easily build upon. This is similar to Maven >> vs Ant. >> >> This is why in my proposal for a format I’ve proposed fixed file names. >> >> Allowing discretion means everyone will do it differently and we’ll need to >> define best practices and best practices are hard to enforce and always >> cause problems. >> >>> UC11: there should be the possibility to share content of a file between >>> several files or components (e.g. creator elements) >> I wouldn’t phrase it this way. I’d prefer to say that it should be possible >> to apply default values to missing elements in metadata. >> >> The way I see this for example for the format I’ve proposed in the other >> thread, is by having default properties that can be put on the filesystem, >> for example in default.properties file) so that when an element is missing >> the default would be used. >> >> Now I’m not sure we want this requirement and for me it’s an optional >> requirement and not a mandatory one. It makes it much harder to develop an >> exporter. >> >>> UC12: (transformation) simple search and replace operation should be >>> supported by the build mechanism, especially when dependencies are followed. >> Could you explain more, I don’t understand? >> >>> UC13: it would be good that the format can be specified by a grammer >>> against which one can validate (e.g. RelaxNG) >> I don’t agree in the way it’s phrased since it’s too limiting. I’d change it >> to: it should be possible for the packager tool to validate the content >> (what xar:verify does right now but that could be extended to verify that >> the required metadata are defined). I don’t think we need a formal grammar. >> The important part is that we have validation. >> >> Thanks >> -Vincent >> >>> Paul >>> >>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>> composable vs contextual so as to give us greater flexibility) > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > Vincent Massol <mailto:[email protected]> > 27 August 2016 at 16:46 >> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >> >> Thanks for having this extra thread. >> I think this thread is much more important than starting to design >> something already. >>> Issues with the XAR format >>> ====================== >>> >>> * XML is not an easy to edit format and doesn’t allow use a specific >>> editor to edit content >>> * XML also requires content to be XML-encoded and thus is really not >>> easy to make modification (there’s a risk of breaking the XML easily) >> I completely disagree with these two statements. > > I probably didn’t use the right words because that’s the reason why I think > that's you started the xinclude proposal :) > > AFAIK you started the xinclude proposal because you wanted to be able to edit > content with specific editor (js, css, etc)… which is exactly what the second > part of the first point is about. > > Let me rephrase the first sentence: > > * XML is not an easy to edit format for human beings (it’s very verbose and > easy to make mistakes: missing encoding, etc). It’s also very hard to edit > with a plain text editor. > > As for the 2nd sentence, I don’t see how you can disagree since it’s a fact... > >> XML is easy to edit and is supported by very very many editors and IDEs. >> It can also be validated. > > What you’re saying is very theoretical. The practice (that we’ve had for 10 > years of using the XAR format) is that our XML format that is hard to edit > and can break easily (as was proven numerous times by our committers and > contributors). It’s actually so dangerous that we’ve had to develop a process > which goes like this: > * Never edit the xml directly > * Always import it into a running XWiki instance, make the modifications > there and export from the wiki into XAR > * Then unpack the XAR into the source directory and run mvn xar:format to go > back to the original format. > > Nobody is using a pure XML editor with validation. We are all using Java IDEA > (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain > text and that’s what we’re doing. No developer I know of is using an XML > editor in a structured way (just too painful and complex to navigate the > structure). > > (see below) > >> XML can be written in a very elegant and readable fashion if you care >> for it. >> Generally however, XML is produced form other structured information in >> a way that does not help readability. >>> Can you see more issues? >> The problem is how we put *everything* into XML. >> (you get the same horror if you use Caleb's tools xardump and do not >> tune anything: the resulting javascript is horrible.) >>> Use cases for an alternative filesystem format >>> =================================== >>> >>> (some UC taken from >>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>> >>> * UC1: the structure should be (as) easy (as possible) to navigate in >>> an IDE style view >>> * UC2: it should be easy to add content (a new script or attachment on >>> an existing structure). It should allow using specific editors for >>> different content types, e.g. if a page content is in markdown, it >>> should be editable with a MD editor, js and css should be editable >>> with web editors, etc. >> UC2.1: Attachments should also be present as standalone files. >>> * UC3: It should be possible to build a packaged version of the >>> sources with Maven >>> * UC4: It should be possible to import the packaged version into a >>> running XWiki instance >>> * UC5: It should be possible to export a portion of a running XWiki >>> instance in this format >>> * UC6: This format should be able to fully replace the XAR format . >>> The new format should support at least all features supported by the >>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>> so that the XAR format can be swapped out by introducing extension >>> points/APIs. The idea would be to deprecate the XAR format and >>> introduce this new format instead, and the 2 formats should be avle to >>> cohabit next to each other in XWiki. >>> * UC7: When importing in a wiki and exporting again (without making >>> any change in the wiki), it should generate an identical structure and >>> content, with no difference. >> I do believe that UC7 is not doable in full generality. > > Why not, this is what we have with the XAR format actually and UC7 is > actually already contained in UC6 (but it’s better to be explicit)? > >> Any more? >> >> UC8: the core representation should be using a syntax that is widely >> spread and completely specified (i.e. we should not invent another >> syntax for this) > > This is not a requirement for me. The syntax should be easy to write into, > especially using a plain text editor. YAML for example is a perfectly valid > syntax for me. > >> UC9: the system should make an archival process a widespread practice, >> in the form of zipped files probably. > > That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I > didn’t want to limit us to that but it’s probably going to be zip anyway. > >> UC10: developers should have the discretion to decide when a component >> needs to be in a separate file or not. That is, small text fragments and >> even small attachments should be includable within bigger files > > I don’t agree with this one. I’d like convention over flexibility, i.e. a > fixed format on which tools can easily build upon. This is similar to Maven > vs Ant. > > This is why in my proposal for a format I’ve proposed fixed file names. > > Allowing discretion means everyone will do it differently and we’ll need to > define best practices and best practices are hard to enforce and always cause > problems. > >> UC11: there should be the possibility to share content of a file between >> several files or components (e.g. creator elements) > > I wouldn’t phrase it this way. I’d prefer to say that it should be possible > to apply default values to missing elements in metadata. > > The way I see this for example for the format I’ve proposed in the other > thread, is by having default properties that can be put on the filesystem, > for example in default.properties file) so that when an element is missing > the default would be used. > > Now I’m not sure we want this requirement and for me it’s an optional > requirement and not a mandatory one. It makes it much harder to develop an > exporter. > >> UC12: (transformation) simple search and replace operation should be >> supported by the build mechanism, especially when dependencies are followed. > > Could you explain more, I don’t understand? > >> UC13: it would be good that the format can be specified by a grammer >> against which one can validate (e.g. RelaxNG) > > I don’t agree in the way it’s phrased since it’s too limiting. I’d change it > to: it should be possible for the packager tool to validate the content (what > xar:verify does right now but that could be extended to verify that the > required metadata are defined). I don’t think we need a formal grammar. The > important part is that we have validation. > > Thanks > -Vincent > >> Paul >> >> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >> composable vs contextual so as to give us greater flexibility) > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > Paul Libbrecht <mailto:[email protected]> > 27 August 2016 at 15:44 > Thanks for having this extra thread. > I think this thread is much more important than starting to design > something already. >> Issues with the XAR format >> ====================== >> >> * XML is not an easy to edit format and doesn’t allow use a specific >> editor to edit content >> * XML also requires content to be XML-encoded and thus is really not >> easy to make modification (there’s a risk of breaking the XML easily) > I completely disagree with these two statements. > XML is easy to edit and is supported by very very many editors and IDEs. > It can also be validated. > XML can be written in a very elegant and readable fashion if you care > for it. > Generally however, XML is produced form other structured information in > a way that does not help readability. >> Can you see more issues? > The problem is how we put *everything* into XML. > (you get the same horror if you use Caleb's tools xardump and do not > tune anything: the resulting javascript is horrible.) >> Use cases for an alternative filesystem format >> =================================== >> >> (some UC taken from >> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >> >> * UC1: the structure should be (as) easy (as possible) to navigate in >> an IDE style view >> * UC2: it should be easy to add content (a new script or attachment on >> an existing structure). It should allow using specific editors for >> different content types, e.g. if a page content is in markdown, it >> should be editable with a MD editor, js and css should be editable >> with web editors, etc. > UC2.1: Attachments should also be present as standalone files. >> * UC3: It should be possible to build a packaged version of the >> sources with Maven >> * UC4: It should be possible to import the packaged version into a >> running XWiki instance >> * UC5: It should be possible to export a portion of a running XWiki >> instance in this format >> * UC6: This format should be able to fully replace the XAR format . >> The new format should support at least all features supported by the >> XAR format (versioned, etc). Note: XE will need to be refactor a bit >> so that the XAR format can be swapped out by introducing extension >> points/APIs. The idea would be to deprecate the XAR format and >> introduce this new format instead, and the 2 formats should be avle to >> cohabit next to each other in XWiki. >> * UC7: When importing in a wiki and exporting again (without making >> any change in the wiki), it should generate an identical structure and >> content, with no difference. > I do believe that UC7 is not doable in full generality. > > Any more? > > UC8: the core representation should be using a syntax that is widely > spread and completely specified (i.e. we should not invent another > syntax for this) > > UC9: the system should make an archival process a widespread practice, > in the form of zipped files probably. > > UC10: developers should have the discretion to decide when a component > needs to be in a separate file or not. That is, small text fragments and > even small attachments should be includable within bigger files > > UC11: there should be the possibility to share content of a file between > several files or components (e.g. creator elements) > > UC12: (transformation) simple search and replace operation should be > supported by the build mechanism, especially when dependencies are followed. > > UC13: it would be good that the format can be specified by a grammer > against which one can validate (e.g. RelaxNG) > > Paul > > (FYI UC10, UC11, and UC12 follow the architecture recommendation to be > composable vs contextual so as to give us greater flexibility) > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > Vincent Massol <mailto:[email protected]> > 27 August 2016 at 15:01 > Hi, > > This is a follow-up on the threads: > * "Designing the perfect FS representation of a wiki”: > http://markmail.org/message/3yghqwetmdt5woez > * "XAR source projects should allow source files”: > http://markmail.org/message/432o36r4klh7yv24 > > It’s also a continuation of the work done here: > http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications > > Once we get convergence on those thread (or even if we don’t), I’ll > update design.xwiki.org with the results. > > The goal is to define the use case for an alternate filesystem to XAR. > > Issues with the XAR format > ====================== > > * XML is not an easy to edit format and doesn’t allow use a specific > editor to edit content > * XML also requires content to be XML-encoded and thus is really not > easy to make modification (there’s a risk of breaking the XML easily) > > Can you see more issues? > > Use cases for an alternative filesystem format > =================================== > > (some UC taken from > http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) > > * UC1: the structure should be (as) easy (as possible) to navigate in > an IDE style view > * UC2: it should be easy to add content (a new script or attachment on > an existing structure). It should allow using specific editors for > different content types, e.g. if a page content is in markdown, it > should be editable with a MD editor, js and css should be editable > with web editors, etc. > * UC3: It should be possible to build a packaged version of the > sources with Maven > * UC4: It should be possible to import the packaged version into a > running XWiki instance > * UC5: It should be possible to export a portion of a running XWiki > instance in this format > * UC6: This format should be able to fully replace the XAR format . > The new format should support at least all features supported by the > XAR format (versioned, etc). Note: XE will need to be refactor a bit > so that the XAR format can be swapped out by introducing extension > points/APIs. The idea would be to deprecate the XAR format and > introduce this new format instead, and the 2 formats should be avle to > cohabit next to each other in XWiki. > * UC7: When importing in a wiki and exporting again (without making > any change in the wiki), it should generate an identical structure and > content, with no difference. > > Any more? > > Thanks > -Vincent > > > > > > > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
