> On 29 Aug 2016, at 09:40, Vincent Massol <[email protected]> wrote: > >> >> On 27 Aug 2016, at 18:37, Paul Libbrecht <[email protected]> wrote: >> >> Vincent, >> >> I think we disagree on two major aspects which are kind of related: >> >> - writing document structure and metadata should be doable by hand >> editing as opposed to writing into the wiki alone: > > I don’t disagree on this. I just gave some arguments against doing it to > avoid problems. Can you tell me how you validate that the changes work? For > me it means deploying it in a wiki and if you deploy you’ll need to make > changes and export the changes back (that’s the safest). And if you deploy > you might as well start in the wiki to save you time. Side note: this is why > it’s important that we have good editing tools inside the wiki (hence XWebIDE > for example). We need devs to be able to write code in the wiki. > > BTW that was one reason I was suggesting YAML and not XML since XML is not a > human-writable syntax (it’s not made for humans, it’s made for machines). > >> - I believe this is a fundamental aspect of source editing and should >> be supported (hence the wish to a validatable widespread syntax) >> - you seem to indicate that the current process you've agreed upon has >> discouraged it (but, at the same time, you indicate you wish to do it >> for YAML) > > It’s just very hard to write a full wiki page from scratch in sources and > brings little value since you need to validate it anyway. > >> - writing XML can be done by hand and is easy: >> - I am a strong defender of this: all editors I've worked with >> (IntelliJ IDEA, jEdit, BBEdit, XCode, maybe not vim) support XML very >> well so that it's just careless to commit a && (not to mention that mvn >> would not validate it, and would not build it if with my patch, simply >> because it parses). > > This is very nice (and something we should have implemented a long time ago) > if your patch makes validation on the XML. > >> Clearly, editing in a textbox can get you there, >> same with MS Word! >> - you seem to say that bad things happen with such commits as not even >> valid XML documents and seem to motivate that to move to another format >> (I feel YAML a zillion less predictable than XML because of whitespace >> handling btw). > > AFAIK whitespaces are not significant in YAML and you can align the elements > which makes it even more readable. > >> Paul >> >> PS: when I say editing XML is for sane people, I don't mean everything >> should be XML, which is why I proposed the XAR extension to include >> XInclude, textinclude and binaryInclude. >> >> PPS: another, maybe deeper, disagreement might be the need to round-trip >> (server, source) vs the need to source-edit: hand-tuning the format of a >> file (e.g. in XML or in JSON) is easily in contradiction with a server >> generation from a structured source. Maybe this is the hard bone. > > IMO you’ll be able to get away from roundtrip only when we’ll have an extra > easy way of unit testing pages without having to deploy it in a full-fledged > running XWiki. But even then, you won’t be able to see the impact on other > stuff so functional tests is more what we’ll need and this requires > deployment. > > Now, as I mentioned in the other thread, having a mvn xar:deploy and mvn > xar:fetch mojos would help a lot for writing changes directly in the source > tree. > > My only point is that we need to support both use cases: > - round trip strategy (i.e. write in wiki, test, export, save) > - source tree changes strategy (i.e. write in source, push, test, fetch, save)
To be more precise: - round trip strategy (i.e. write in wiki, [test, make changes]*, export, save) - source tree changes strategy (i.e. write in sources, [push, test in wiki, make changes locally]*, save) Thanks -Vincent > > Thanks > -Vincent > >> Vincent Massol <mailto:[email protected]> >>> 27 August 2016 at 17:50 >>>> On 27 Aug 2016, at 16:46, Vincent Massol <[email protected]> wrote: >>>> >>>>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>>>> >>>>> Thanks for having this extra thread. >>>>> I think this thread is much more important than starting to design >>>>> something already. >>>>>> Issues with the XAR format >>>>>> ====================== >>>>>> >>>>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>>>> editor to edit content >>>>>> * XML also requires content to be XML-encoded and thus is really not >>>>>> easy to make modification (there’s a risk of breaking the XML easily) >>>>> I completely disagree with these two statements. >>>> I probably didn’t use the right words because that’s the reason why I >>>> think that's you started the xinclude proposal :) >>>> >>>> AFAIK you started the xinclude proposal because you wanted to be able to >>>> edit content with specific editor (js, css, etc)… which is exactly what >>>> the second part of the first point is about. >>>> >>>> Let me rephrase the first sentence: >>>> >>>> * XML is not an easy to edit format for human beings (it’s very verbose >>>> and easy to make mistakes: missing encoding, etc). It’s also very hard to >>>> edit with a plain text editor. >>>> >>>> As for the 2nd sentence, I don’t see how you can disagree since it’s a >>>> fact... >>>> >>>>> XML is easy to edit and is supported by very very many editors and IDEs. >>>>> It can also be validated. >>>> What you’re saying is very theoretical. The practice (that we’ve had for >>>> 10 years of using the XAR format) is that our XML format that is hard to >>>> edit and can break easily (as was proven numerous times by our committers >>>> and contributors). It’s actually so dangerous that we’ve had to develop a >>>> process which goes like this: >>>> * Never edit the xml directly >>>> * Always import it into a running XWiki instance, make the modifications >>>> there and export from the wiki into XAR >>>> * Then unpack the XAR into the source directory and run mvn xar:format to >>>> go back to the original format. >>>> >>>> Nobody is using a pure XML editor with validation. We are all using Java >>>> IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as >>>> plain text and that’s what we’re doing. No developer I know of is using an >>>> XML editor in a structured way (just too painful and complex to navigate >>>> the structure). >>> >>> To be more specific, the main issue we’ve had is contributors/committers >>> who committed unencoded content, such as && instead of && or > >>> instead of > >>> >>> Now, to be accurate, if you remove the problem with the encoding (which can >>> be removed IMO with CDATA) then we never touch much of the rest of the >>> metadata. >>> >>> In practice it would be nice if most of it could be generated by the maven >>> plugin. In practice we don’t need much specific data (for a pure doc, it’s >>> a bit more for xclass/xobjects): title, reference, syntax, parent, hidden, >>> and language/translation. Syntax and reference could be computed from the >>> directory structure. Parent could too. And hidden could default to visible >>> by default. That said it doesn’t matter that much since the process is to >>> export from a running xwiki instance (we need that to validate that it >>> works at execution time, or we’d need good unit/integration tests for >>> pages). >>> >>> So once we take the content out, the format of the metadata doesn’t matter >>> that much probably since we’re not going to author it from scratch anyway >>> (it’ll come from exported wiki pages). >>> >>> So I guess XML, even though very verbose, could still be ok. But XML for >>> doc content or xproperty textareas, or attachments is a sure no go. >>> >>> Thanks >>> -Vincent >>> >>>> (see below) >>>> >>>>> XML can be written in a very elegant and readable fashion if you care >>>>> for it. >>>>> Generally however, XML is produced form other structured information in >>>>> a way that does not help readability. >>>>>> Can you see more issues? >>>>> The problem is how we put *everything* into XML. >>>>> (you get the same horror if you use Caleb's tools xardump and do not >>>>> tune anything: the resulting javascript is horrible.) >>>>>> Use cases for an alternative filesystem format >>>>>> =================================== >>>>>> >>>>>> (some UC taken from >>>>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>>>> >>>>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>>>> an IDE style view >>>>>> * UC2: it should be easy to add content (a new script or attachment on >>>>>> an existing structure). It should allow using specific editors for >>>>>> different content types, e.g. if a page content is in markdown, it >>>>>> should be editable with a MD editor, js and css should be editable >>>>>> with web editors, etc. >>>>> UC2.1: Attachments should also be present as standalone files. >>>>>> * UC3: It should be possible to build a packaged version of the >>>>>> sources with Maven >>>>>> * UC4: It should be possible to import the packaged version into a >>>>>> running XWiki instance >>>>>> * UC5: It should be possible to export a portion of a running XWiki >>>>>> instance in this format >>>>>> * UC6: This format should be able to fully replace the XAR format . >>>>>> The new format should support at least all features supported by the >>>>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>>>> so that the XAR format can be swapped out by introducing extension >>>>>> points/APIs. The idea would be to deprecate the XAR format and >>>>>> introduce this new format instead, and the 2 formats should be avle to >>>>>> cohabit next to each other in XWiki. >>>>>> * UC7: When importing in a wiki and exporting again (without making >>>>>> any change in the wiki), it should generate an identical structure and >>>>>> content, with no difference. >>>>> I do believe that UC7 is not doable in full generality. >>>> Why not, this is what we have with the XAR format actually and UC7 is >>>> actually already contained in UC6 (but it’s better to be explicit)? >>>> >>>>> Any more? >>>>> >>>>> UC8: the core representation should be using a syntax that is widely >>>>> spread and completely specified (i.e. we should not invent another >>>>> syntax for this) >>>> This is not a requirement for me. The syntax should be easy to write into, >>>> especially using a plain text editor. YAML for example is a perfectly >>>> valid syntax for me. >>>> >>>>> UC9: the system should make an archival process a widespread practice, >>>>> in the form of zipped files probably. >>>> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since >>>> I didn’t want to limit us to that but it’s probably going to be zip anyway. >>>> >>>>> UC10: developers should have the discretion to decide when a component >>>>> needs to be in a separate file or not. That is, small text fragments and >>>>> even small attachments should be includable within bigger files >>>> I don’t agree with this one. I’d like convention over flexibility, i.e. a >>>> fixed format on which tools can easily build upon. This is similar to >>>> Maven vs Ant. >>>> >>>> This is why in my proposal for a format I’ve proposed fixed file names. >>>> >>>> Allowing discretion means everyone will do it differently and we’ll need >>>> to define best practices and best practices are hard to enforce and always >>>> cause problems. >>>> >>>>> UC11: there should be the possibility to share content of a file between >>>>> several files or components (e.g. creator elements) >>>> I wouldn’t phrase it this way. I’d prefer to say that it should be >>>> possible to apply default values to missing elements in metadata. >>>> >>>> The way I see this for example for the format I’ve proposed in the other >>>> thread, is by having default properties that can be put on the filesystem, >>>> for example in default.properties file) so that when an element is missing >>>> the default would be used. >>>> >>>> Now I’m not sure we want this requirement and for me it’s an optional >>>> requirement and not a mandatory one. It makes it much harder to develop an >>>> exporter. >>>> >>>>> UC12: (transformation) simple search and replace operation should be >>>>> supported by the build mechanism, especially when dependencies are >>>>> followed. >>>> Could you explain more, I don’t understand? >>>> >>>>> UC13: it would be good that the format can be specified by a grammer >>>>> against which one can validate (e.g. RelaxNG) >>>> I don’t agree in the way it’s phrased since it’s too limiting. I’d change >>>> it to: it should be possible for the packager tool to validate the content >>>> (what xar:verify does right now but that could be extended to verify that >>>> the required metadata are defined). I don’t think we need a formal >>>> grammar. The important part is that we have validation. >>>> >>>> Thanks >>>> -Vincent >>>> >>>>> Paul >>>>> >>>>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>>>> composable vs contextual so as to give us greater flexibility) >>> >>> _______________________________________________ >>> devs mailing list >>> [email protected] >>> http://lists.xwiki.org/mailman/listinfo/devs >>> Vincent Massol <mailto:[email protected]> >>> 27 August 2016 at 16:46 >>>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>>> >>>> Thanks for having this extra thread. >>>> I think this thread is much more important than starting to design >>>> something already. >>>>> Issues with the XAR format >>>>> ====================== >>>>> >>>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>>> editor to edit content >>>>> * XML also requires content to be XML-encoded and thus is really not >>>>> easy to make modification (there’s a risk of breaking the XML easily) >>>> I completely disagree with these two statements. >>> >>> I probably didn’t use the right words because that’s the reason why I think >>> that's you started the xinclude proposal :) >>> >>> AFAIK you started the xinclude proposal because you wanted to be able to >>> edit content with specific editor (js, css, etc)… which is exactly what the >>> second part of the first point is about. >>> >>> Let me rephrase the first sentence: >>> >>> * XML is not an easy to edit format for human beings (it’s very verbose and >>> easy to make mistakes: missing encoding, etc). It’s also very hard to edit >>> with a plain text editor. >>> >>> As for the 2nd sentence, I don’t see how you can disagree since it’s a >>> fact... >>> >>>> XML is easy to edit and is supported by very very many editors and IDEs. >>>> It can also be validated. >>> >>> What you’re saying is very theoretical. The practice (that we’ve had for 10 >>> years of using the XAR format) is that our XML format that is hard to edit >>> and can break easily (as was proven numerous times by our committers and >>> contributors). It’s actually so dangerous that we’ve had to develop a >>> process which goes like this: >>> * Never edit the xml directly >>> * Always import it into a running XWiki instance, make the modifications >>> there and export from the wiki into XAR >>> * Then unpack the XAR into the source directory and run mvn xar:format to >>> go back to the original format. >>> >>> Nobody is using a pure XML editor with validation. We are all using Java >>> IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as >>> plain text and that’s what we’re doing. No developer I know of is using an >>> XML editor in a structured way (just too painful and complex to navigate >>> the structure). >>> >>> (see below) >>> >>>> XML can be written in a very elegant and readable fashion if you care >>>> for it. >>>> Generally however, XML is produced form other structured information in >>>> a way that does not help readability. >>>>> Can you see more issues? >>>> The problem is how we put *everything* into XML. >>>> (you get the same horror if you use Caleb's tools xardump and do not >>>> tune anything: the resulting javascript is horrible.) >>>>> Use cases for an alternative filesystem format >>>>> =================================== >>>>> >>>>> (some UC taken from >>>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>>> >>>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>>> an IDE style view >>>>> * UC2: it should be easy to add content (a new script or attachment on >>>>> an existing structure). It should allow using specific editors for >>>>> different content types, e.g. if a page content is in markdown, it >>>>> should be editable with a MD editor, js and css should be editable >>>>> with web editors, etc. >>>> UC2.1: Attachments should also be present as standalone files. >>>>> * UC3: It should be possible to build a packaged version of the >>>>> sources with Maven >>>>> * UC4: It should be possible to import the packaged version into a >>>>> running XWiki instance >>>>> * UC5: It should be possible to export a portion of a running XWiki >>>>> instance in this format >>>>> * UC6: This format should be able to fully replace the XAR format . >>>>> The new format should support at least all features supported by the >>>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>>> so that the XAR format can be swapped out by introducing extension >>>>> points/APIs. The idea would be to deprecate the XAR format and >>>>> introduce this new format instead, and the 2 formats should be avle to >>>>> cohabit next to each other in XWiki. >>>>> * UC7: When importing in a wiki and exporting again (without making >>>>> any change in the wiki), it should generate an identical structure and >>>>> content, with no difference. >>>> I do believe that UC7 is not doable in full generality. >>> >>> Why not, this is what we have with the XAR format actually and UC7 is >>> actually already contained in UC6 (but it’s better to be explicit)? >>> >>>> Any more? >>>> >>>> UC8: the core representation should be using a syntax that is widely >>>> spread and completely specified (i.e. we should not invent another >>>> syntax for this) >>> >>> This is not a requirement for me. The syntax should be easy to write into, >>> especially using a plain text editor. YAML for example is a perfectly valid >>> syntax for me. >>> >>>> UC9: the system should make an archival process a widespread practice, >>>> in the form of zipped files probably. >>> >>> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since >>> I didn’t want to limit us to that but it’s probably going to be zip anyway. >>> >>>> UC10: developers should have the discretion to decide when a component >>>> needs to be in a separate file or not. That is, small text fragments and >>>> even small attachments should be includable within bigger files >>> >>> I don’t agree with this one. I’d like convention over flexibility, i.e. a >>> fixed format on which tools can easily build upon. This is similar to Maven >>> vs Ant. >>> >>> This is why in my proposal for a format I’ve proposed fixed file names. >>> >>> Allowing discretion means everyone will do it differently and we’ll need to >>> define best practices and best practices are hard to enforce and always >>> cause problems. >>> >>>> UC11: there should be the possibility to share content of a file between >>>> several files or components (e.g. creator elements) >>> >>> I wouldn’t phrase it this way. I’d prefer to say that it should be possible >>> to apply default values to missing elements in metadata. >>> >>> The way I see this for example for the format I’ve proposed in the other >>> thread, is by having default properties that can be put on the filesystem, >>> for example in default.properties file) so that when an element is missing >>> the default would be used. >>> >>> Now I’m not sure we want this requirement and for me it’s an optional >>> requirement and not a mandatory one. It makes it much harder to develop an >>> exporter. >>> >>>> UC12: (transformation) simple search and replace operation should be >>>> supported by the build mechanism, especially when dependencies are >>>> followed. >>> >>> Could you explain more, I don’t understand? >>> >>>> UC13: it would be good that the format can be specified by a grammer >>>> against which one can validate (e.g. RelaxNG) >>> >>> I don’t agree in the way it’s phrased since it’s too limiting. I’d change >>> it to: it should be possible for the packager tool to validate the content >>> (what xar:verify does right now but that could be extended to verify that >>> the required metadata are defined). I don’t think we need a formal grammar. >>> The important part is that we have validation. >>> >>> Thanks >>> -Vincent >>> >>>> Paul >>>> >>>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>>> composable vs contextual so as to give us greater flexibility) >>> >>> _______________________________________________ >>> devs mailing list >>> [email protected] >>> http://lists.xwiki.org/mailman/listinfo/devs >>> Paul Libbrecht <mailto:[email protected]> >>> 27 August 2016 at 15:44 >>> Thanks for having this extra thread. >>> I think this thread is much more important than starting to design >>> something already. >>>> Issues with the XAR format >>>> ====================== >>>> >>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>> editor to edit content >>>> * XML also requires content to be XML-encoded and thus is really not >>>> easy to make modification (there’s a risk of breaking the XML easily) >>> I completely disagree with these two statements. >>> XML is easy to edit and is supported by very very many editors and IDEs. >>> It can also be validated. >>> XML can be written in a very elegant and readable fashion if you care >>> for it. >>> Generally however, XML is produced form other structured information in >>> a way that does not help readability. >>>> Can you see more issues? >>> The problem is how we put *everything* into XML. >>> (you get the same horror if you use Caleb's tools xardump and do not >>> tune anything: the resulting javascript is horrible.) >>>> Use cases for an alternative filesystem format >>>> =================================== >>>> >>>> (some UC taken from >>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>> >>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>> an IDE style view >>>> * UC2: it should be easy to add content (a new script or attachment on >>>> an existing structure). It should allow using specific editors for >>>> different content types, e.g. if a page content is in markdown, it >>>> should be editable with a MD editor, js and css should be editable >>>> with web editors, etc. >>> UC2.1: Attachments should also be present as standalone files. >>>> * UC3: It should be possible to build a packaged version of the >>>> sources with Maven >>>> * UC4: It should be possible to import the packaged version into a >>>> running XWiki instance >>>> * UC5: It should be possible to export a portion of a running XWiki >>>> instance in this format >>>> * UC6: This format should be able to fully replace the XAR format . >>>> The new format should support at least all features supported by the >>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>> so that the XAR format can be swapped out by introducing extension >>>> points/APIs. The idea would be to deprecate the XAR format and >>>> introduce this new format instead, and the 2 formats should be avle to >>>> cohabit next to each other in XWiki. >>>> * UC7: When importing in a wiki and exporting again (without making >>>> any change in the wiki), it should generate an identical structure and >>>> content, with no difference. >>> I do believe that UC7 is not doable in full generality. >>> >>> Any more? >>> >>> UC8: the core representation should be using a syntax that is widely >>> spread and completely specified (i.e. we should not invent another >>> syntax for this) >>> >>> UC9: the system should make an archival process a widespread practice, >>> in the form of zipped files probably. >>> >>> UC10: developers should have the discretion to decide when a component >>> needs to be in a separate file or not. That is, small text fragments and >>> even small attachments should be includable within bigger files >>> >>> UC11: there should be the possibility to share content of a file between >>> several files or components (e.g. creator elements) >>> >>> UC12: (transformation) simple search and replace operation should be >>> supported by the build mechanism, especially when dependencies are followed. >>> >>> UC13: it would be good that the format can be specified by a grammer >>> against which one can validate (e.g. RelaxNG) >>> >>> Paul >>> >>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>> composable vs contextual so as to give us greater flexibility) >>> _______________________________________________ >>> devs mailing list >>> [email protected] >>> http://lists.xwiki.org/mailman/listinfo/devs >>> Vincent Massol <mailto:[email protected]> >>> 27 August 2016 at 15:01 >>> Hi, >>> >>> This is a follow-up on the threads: >>> * "Designing the perfect FS representation of a wiki”: >>> http://markmail.org/message/3yghqwetmdt5woez >>> * "XAR source projects should allow source files”: >>> http://markmail.org/message/432o36r4klh7yv24 >>> >>> It’s also a continuation of the work done here: >>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications >>> >>> Once we get convergence on those thread (or even if we don’t), I’ll >>> update design.xwiki.org with the results. >>> >>> The goal is to define the use case for an alternate filesystem to XAR. >>> >>> Issues with the XAR format >>> ====================== >>> >>> * XML is not an easy to edit format and doesn’t allow use a specific >>> editor to edit content >>> * XML also requires content to be XML-encoded and thus is really not >>> easy to make modification (there’s a risk of breaking the XML easily) >>> >>> Can you see more issues? >>> >>> Use cases for an alternative filesystem format >>> =================================== >>> >>> (some UC taken from >>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>> >>> * UC1: the structure should be (as) easy (as possible) to navigate in >>> an IDE style view >>> * UC2: it should be easy to add content (a new script or attachment on >>> an existing structure). It should allow using specific editors for >>> different content types, e.g. if a page content is in markdown, it >>> should be editable with a MD editor, js and css should be editable >>> with web editors, etc. >>> * UC3: It should be possible to build a packaged version of the >>> sources with Maven >>> * UC4: It should be possible to import the packaged version into a >>> running XWiki instance >>> * UC5: It should be possible to export a portion of a running XWiki >>> instance in this format >>> * UC6: This format should be able to fully replace the XAR format . >>> The new format should support at least all features supported by the >>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>> so that the XAR format can be swapped out by introducing extension >>> points/APIs. The idea would be to deprecate the XAR format and >>> introduce this new format instead, and the 2 formats should be avle to >>> cohabit next to each other in XWiki. >>> * UC7: When importing in a wiki and exporting again (without making >>> any change in the wiki), it should generate an identical structure and >>> content, with no difference. >>> >>> Any more? >>> >>> Thanks >>> -Vincent >>> > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
