Hey everybody, in the latest release <https://groups.google.com/forum/#!topic/tiddlywikiclassic/GDBYezPtdlg> of TiddlyWiki Classic I've fixed the upgrading engine and now it's time to deal with saving <https://groups.google.com/forum/#!topic/tiddlywikiclassic/SiD6QnG9wAM> more consistently. That's where I'd like not only to ask about some dev aspects but also discuss some conventions and create docs for consistent support of saving both TWC and TW5.
I'll start with TiddlyFox since it is still used by many users (with either FireFox 56- or Pale Moon, WaterFox and other FF forks). One thing about it is building the xpi: I've read about building it a little and the crucial quesion is: is it still possible to create signed xpi, or updating TiddlyFox is impossible in principle? If it's not, I'd only like to ask Jeremy to add 2.x tags to the repo and attach xpi files to the github releases; or add me to the repo and I'll do that myself. Otherwise, I'd like to contribute a bit more and have some more questions. Next and the main point of this thread is: both TWC and TW5 have many supporting solutions (native apps, servers, browser plugins etc) for saving but I'm not aware of docs that describe how actually one supports saving (and may be more functionality) of both TWC and TW5. For instance, MainTiddlyServer <https://yakovl.github.io/MainTiddlyServer/> could be extended a bit so that it fits the php server gap <https://github.com/TiddlyWiki/php-server>, but I (as a developer of TWC extensions, MTS and maintainer of the core) am not really aware of how to support TW5 correctly; likewise, the aTW <https://github.com/TiddlyWiki/aTW> saver for Android supports only TW5 (as far as I know), but I'd like to participate and support TWC as well, yet I still didn't even get an answer <https://github.com/TiddlyWiki/aTW/issues/1#issuecomment-459076550> about the app status (and I suspect Simon may be reluctant to support TWC without nice docs describing that). Finally, it seems every time one tries to support saving in their software, they invent some new implementation which (this diversity) is difficult to follow and the ecosystem is quite difficult to maintain. Thus, it seems to me that we really need 2 things: - nice docs describing how to support saving of TWC, TW5 (and, in perpective, more features like image uploading and more) - core libraries in different languages (js for browser plugins? js for node, PHP, Java/Android, Swift/iOS, may be more) which one can use for developing new solutions Now this thread is mostly about the first part, the docs (and by the way saving is not the only thing which would be nice to document; think, for example, of the .tid format which is not described anywhere, at least Jeremy wrote so some months ago). My proposal consists of 2 parts: 1. create a repo in the TiddlyWiki <https://github.com/TiddlyWiki/> org @github (I'd like to be a member to contribute) 2. start building docs For the second part, I now have a draft based on what I've learned from the TiddlyFox sources, see it below the main post. I invite you to comment it, correct where I'm wrong, fill the gaps, especially about TW5. Best regards, Yakov. == first draft of "how to support TW saving" docs == To support TW saving, it is to be desided: 1. how to recognize a TW? 2. when to inject new JS? 3. what JS to inject (and where)? (and also the "protocol" and the "back-end logic" stuff) *How to recognize a TW?* First, let's note that the simplest implementation can omit TW recognition: just ask whether to inject JS or inject it into every html "served" with the saver app. Looking at TiddlyFox code, it seems the following procedure is ok: - to recognize TWC, check whether there's #storeArea element [TF searches DOM, not source], #versionArea element and the latter contains the word "TiddlyWiki" - to recognize TW5, check whether among <meta> elements there's one with name "application-name" and content "TiddlyWiki" This surely can give false positives which seems to be ok; this also doesn't support "pure store" format. *When (and where) to inject new JS?* this section is to be written yet. Notes: - MainTiddlyServer injects JS bits into the TWC code before serving it and strips those back when saves "the whole thing" (it can also send save increments); some modifications (like support of incremental saving) are made as hijacks which are done in loadPlugins <https://github.com/YakovL/MainTiddlyServer/blob/master/index.php#L467> so that the store is ready - server-side implementations (that inject JS into TW so that user doesn't have to install a plugin into TW) also need to decide where to inject JS, see next section - not sure whether TiddlyFox adds JS on load or eariler (probably the latter) - neither I am sure of the exact moment tiddly-node-saver uses <https://github.com/fallwest/tiddly-node-saver/blob/master/server.js#L27> although the approach looks interesting - same for TiddlyDesktop overwriting <https://github.com/Jermolene/TiddlyDesktop/blob/ffd0c7be4d4220a67211e3b18ece208d11d305e4/source/js/utils/saving.js#L19> *What JS to inject (and where)?* For now, I'm describing some existing approaches instead of a "canonical" one to be proposed. TiddlyFox overwrites <https://github.com/TiddlyWiki/TiddlyFox/blob/master/data/inject.js#L52> window.mozillaSaveFile and window.mozillaLoadFile with new implementations and also sets window.convertUriToUTF8 and window.convertUnicodeToFileFormat to plain s => s functions (needed for TWc before 2.9.2). This works *for TWC* and seems to be enough for bare loading/saving functionality. One thing to add here is overwriting the config.options.chkHttpReadOnly option if TW is served via http(s): instead of file: scheme. A similar set of changes is implemented <https://github.com/fallwest/tiddly-node-saver/blob/00444358fb4d77b8ca0b8115403e61143f065f66/saveScripts.js> in tiddly-node-saver, it's simpler (window.saveFile is -overwritten-, loading is not implemented at all, chkHttpReadOnly is set to false). Actually window.saveFile is not overwritten but rather defined and I suspect it relies on the following definition in TWC core: window.saveFile = window.saveFile || function(fileUrl,content) which substituted this function saveFile(fileUrl,content) in 2.7.0 which is a quite sane limitation. I haven't quite figured the approach <https://github.com/Jermolene/TiddlyDesktop/blob/master/source/js/utils/saving.js> of TiddlyDesktop yet. Unlike most solutions for local usages only, MTS is supposed to be used on servers as well as locally, that's why it is more strict and currently doesn't support saving arbitrary files. Instead, it injects <https://github.com/YakovL/MainTiddlyServer/blob/master/index.php#L510> "return saveOnlineChanges();" into saveChanges (and also inject <https://github.com/YakovL/MainTiddlyServer/blob/master/index.php#L505> the definitions before "function saveMain(" and changes <https://github.com/YakovL/MainTiddlyServer/blob/master/index.php#L513> the default value of chkHttpReadOnly, but now I see this should be refactored since there are better approaches). As *for TW5*, I'm not really sure about the exact stack, but it seems for saving, a handler of the "tiddlywiki-save-file" custom event should be added. The handler should expect the "event" (first argument) to have .path and .content properties which should hold (local?) path to the file to save and (UTF-8?) content string of the file. Not sure if some sort of "file-save-success" event is supported to notify the core that the file was actually saved (and TiddlyFox actually has a bug to always report that TW was saved, even if it was on a USB storage which was removed before saving, because the injected saver always returns true <https://github.com/TiddlyWiki/TiddlyFox/blob/eb4c0d9822cf8beffe27a7cacb3b00c2d18e3771/data/inject.js#L26> ). *Further considerations* There's a number of topics that should be covered as well, mostly after those above are, but some of them will affect architecture and hence should be considered anyway (can affect the core, especially in case of TWC; I'm ready to adapt it to better architecture): - sync vs async saving - full saving vs sending changes, shortcuts for backuping (sending full file to backup vs sending "copy file A into B" message) - protocol layer (shouldn't necessarily be uniform, but recommendations can facilitate development of new tools) - streaming changes via websockets and sync editing - security consideratinos - server logic (authorization & roles, ..) - federation (including content from other TWs, saving it back) - advanced features (image uploading, integration with git etc) ==== Looks like that's all I've gathered for now. Again, contributions are very much welcome. -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/e4526996-ad42-45fd-af38-caac660cc07c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
