Hi Joseph, Thanks for your thoughtful and detailed response. It gives me a good view of the required effort. My intent is to extend Fossil with Lua, not to embed it into Fossil (that could be, in the future). Extension could be worked to decrease the natural friction and inertia you mention over other embedding other languages and give an easy probe of concept.
I will try an organic approach to this problem. First starting with some external functionality in the similar vein of how syntax highlighting is done using JS [1], then trying some bindings between JS and installed and then I will try to enable something like Asciidoctor.js to put it in the rendering hook, to see if, in the same way that files ended with particular extensions can be rendered with color, I can render files ended in 'adoc' or 'ad' using Asciidoctor.js. After that I would try some approach that enables something similar using Lunamark[2]. [1] https://www.fossil-scm.org/index.html/wiki?name=Cookbook#source-hilight [2] http://jgm.github.io/lunamark/ This will be a slow project. But I think that is important to increase Fossil extensibility and scriptability. Javascript seems the easiest path now, but at some point, some easy to embed and (kind of) more widely used languages as Lua should be part of this ecosystem. I will keep you updated about my advances and making documentation as I go. I'm subscribed to the mailing list, so any answer to it will be fine also. Cheers, Offray Ps: My used family name is Luna (yes, it means the same in Spanish that Lua in Portuguese or Moon in English, but that's just a coincidence). Here in Latin America we have two family names: one taken from the family name of your father and other taken from the family name of your mother, following the Spanish tradition... which seems more fair :-). On 24/03/18 17:52, Joseph R. Justice wrote: > [Response intentionally CC'd to the original poster, on the off chance > he's not already subscribed to the mailing list] > > > On Mon, Mar 19, 2018 at 7:42 PM, Offray Vladimir Luna Cárdenas > <off...@riseup.net <mailto:off...@riseup.net>> wrote: > > This is just kind of a next semester project, but I would like to > start > exploring the options. > > I'm using Fossil mainly to store documentation (using the superb > embedded documentation feature) for my Grafoscopio[1] project. I think > that SQLite and Fossil are good examples of what I call pocket > infrastructures (simple, self-contained and working well on-line and > off-line) that can help to democratize fields like data visualization > and storytelling[2]. But, despite its advantages, Fossil is > starting to > feel limited, particularly with the Yaml metadata blocks that the > Pandoc > Markdown variant uses[3]. So I would like to explore the > possibility to > use Lua (which is already integrated in Pandoc and easily > integrable in > other programs) to read such metadata y show it in a proper way. I > also > think that the support for Lua should open a lot of scriptability > options and ecosystem, far beyond TH1 and Tcl, by using a more wide > spread scripting language with a bigger ecosystem. > > [1] http://mutabit.com/grafoscopio/index.en.html > <http://mutabit.com/grafoscopio/index.en.html> > [2] http://mutabit.com/offray/blog/en/entry/panama-papers-1 > <http://mutabit.com/offray/blog/en/entry/panama-papers-1> > [3] https://pandoc.org/MANUAL.html#extension-yaml_metadata_block > <https://pandoc.org/MANUAL.html#extension-yaml_metadata_block> > > What should be done to explore such Lua extensions to Fossil? > Where can > I start? > > > Dear Mr. Cardenas (my apologies for misspelling your name, but I'm > just Yet Another Damned Monolingual Yankee), > > I saw your question above when you first sent it to the fossil-users > list, and meant to respond to it, at least to give "one bystander's > opinion". My apologies for taking so long to respond. I *do* > personally think it's an interesting idea, but for reasons I'll give > below, it might not be as interesting to the core Fossil developers, > at least at this point. > > Before I continue, I need to emphasize to the utmost extent that I am > *not* a core Fossil developer or committer to this project; I am *not* > an active external developer for it submitting patches or modules to > be integrated with it; nor am I even just someone actively using it > and perhaps submitting bug reports, documentation, user feedback, > etc. (Perhaps I will be some of these things at some point if I ever > get around to getting myself a real system suitable for development, > but at this time I am not.) I have, however, been following the > project and this list for a fair while as an outside bystander and > observer. The following should be taken with that in mind; obviously, > anything I say can and should be superseded by an opinion by someone > with more activity and status within this project. > > For anyone reading this message who's not already aware of it, Lua > (the site https://www.lua.org/ is the canonical home for it) is a > small, relatively new (compared to others anyway) programming language > originally coming from academia in Brazil but now in use at least > somewhat in commercial and production environments internationally, > especially in certain niches. It was intentionally designed to be > embedded within larger applications to serve as a scripting or > programming language for those applications, as I understand TCL was > as well. It is under active development, and has significant existing > documentation (including in dead trees form) and a user community. As > such, it doesn't appear to me at least at first glance (again, solely > based from the viewpoint of an external bystander / observer of the > Fossil project) to be a completely unreasonable notion to integrate > Lua as an alternative scripting / programming language for Fossil. > > > > That being said, Mr. Cardenas, I have to warn you that in my opinion > there's going to be ... I won't say an active objection by the core > Fossil development team to doing this, but at the least a significant > amount of inertia and friction (*not* necessarily maliciously or > consciously intended!) to the idea. I'd say much of this will have to > do with the history surrounding the creation and development of Fossil > and of its sister and Siamese twin project, the embedded database > SQLite. My apologies if the following brief synopsis and analysis of > said history is incorrect or misleading; I eagerly welcome corrections > or augmentations of what I've written. > > As I understand it, SQLite was originally created by Dr. D. Richard > Hipp as an add-on for the TCL programming language, and only later > subsequently became an independent entity in its own right able to be > used without reference to TCL. Fossil then was developed by Dr. Hipp > both as a tool for use in the development of SQLite (presumably the > version control systems commonly available at the time Fossil was > created were deemed by him to be unsuitable) and also as an example of > the use of SQLite by a third-party active project, insofar as Fossil > project repositories are created as SQLite databases, etc. > > Due to the historical origins of SQLite springing out of TCL, and > Fossil springing out of SQLite (and thus at second-hand out of TCL), > both of these projects have naturally considered TCL to be a, or even > the only, first-class embedded scripting and programming language for > their project. (As I recall there's also TC1, but as I understand it > that's derived in its design from TCL, and it's not nearly as capable > as TCL.) The architects and developers of SQLite and Fossil were > originally extremely experienced with, and had comfort using, TCL and > already were (and are) presumably expert in its use, and can use it to > achieve anything they feel necessary to achieve. As such, presumably > they have seen no great need to support alternative embedded scripting > or programming languages for SQLite or Fossil to date. > > > > Given this, I'd say you have (at least) two major problems to solve in > your goal of being able to use Lua with or within Fossil: > > (1) Assumptions within the Fossil code base that there is only the TCL > (and perhaps TC1) embedded programming language available and needing > to be accommodated; objects and concepts which are "natural" for TCL > but might not be natural for Lua; etc. > > (2) Unwillingness by the core Fossil development team to include > whatever integration for Lua you might achieve into the primary > sources and distribution for Fossil which *they* will have to > subsequently maintain and support, especially if they do not see any > need for Fossil to support Lua but rather feel that Fossil's support > of TCL provides everything needed by any likely user of Fossil, and > also especially if they do not feel capable of providing adequate > maintenance and support for this capability, the use of Lua within > Fossil by people, etc. > > The first point is, obviously, primarily if not solely a technical > issue; the second has both technical and social aspects. > > > > For the first point, obviously I have no idea what if any problems you > might encounter, as I am entirely unfamiliar at this time with the > Fossil code base. I'd say that the first obvious step is to examine > the Fossil sources, to see what is enabled or disabled when TCL > support is activated or disabled at compile time. (You'll want to see > what Makefile switches are used to enable or disable this support; > they should be well documented in the documentation for Fossil.) > > See what if anything should be abstracted out of the enabling of > support for embedded TCL to allow for enabling of support for embedded > Lua. (This would presumably also allow later for others to more > easily embed other suitable scripting / programming languages if they > find the need to do so.) > > I'd say that you should watch for problems caused both by the enabling > of Lua support and the disabling of TCL support, and also problems > caused by the enabling of support for both embedded languages > simultaneously; I see no obvious reason why there might not be people > wanting to have both Lua *and* TCL capability at the same time, > especially given the historical background of Fossil, so you should > make sure that can be done. And, this will aid later with the > problems of point (2) mentioned above -- people familiar with TCL do > not have to give it up if support for Lua is also enabled. > > Make sure to create a suitable test suite for Fossil with embedded Lua > (and simultaneous embedded TCL and Lua), to demonstrate that the basic > functionality is working properly. And you'll want to make sure that > any changes you make do not break any existing Fossil test suite > functionality. There are people on this list specifically interested > in that point, and again it will aid later with the problems of point > (2) above. > > And of course you'll need to create adequate documentation for your > changes, both how to enable embedded Lua support and also on how to > actually *use* this support once it is enabled! I wouldn't say you > need to write much if any documentation on how to use Lua itself (tho > providing pointers to such documentation maintained elsewhere, such as > by the Lua project itself, is probably desirable), or at least no more > than might already exist within the Fossil documentation concerning > use of TCL. But, you *will* want to document how Lua concepts, data > objects, etc are exposed and available within the embedded Lua support > in Fossil, and how Fossil concepts, data objects, etc are made > available to scripts using the embedded Lua support. > > Oh, and as an aside, you should make sure that any support for > embedded Lua that you develop is capable of using *both* Lua support > compiled statically within the Fossil binary (which will be important > especially for the Microsoft Windows platform, and possibly for Apple > MacOS/X) and provided by Lua sources incorporated within the source > code for Fossil, *and* *also* provided by a separate system library > and compilation header files for Lua provided by the operating system > and dynamically linked in with the Fossil binary, and not making use > of any sources included with the source code for Fossil itself. This > latter point will be important to most if not all Linux and Unix > distributions which provide Fossil; they typically as a matter of > policy strip out any embedded libraries in the source code for any > application they provide and require said distribution to use the > distribution-provided version of the library, its compilation headers, > etc. (This isn't necessarily the approach that the developers of > Fossil favor, mind you, but it's not something they can control for > any Fossil binary provided by a O/S distribution as opposed to by they > themselves.) > > In particular, you should not assume that the version of Lua *you* > provide, especially if it's a "bleeding edge" version, will be the > version of Lua available to a Fossil with Lua support enabled which is > provided by a O/S distribution; you'll need to incorporate checks that > whatever version of Lua is provided contains the minimum capabilities > you need for your embedded support. To this end, you'll want to look > at whatever the primary sources for Fossil do with respect to using > SQLite, making sure it is available and has the minimum functionality > needed, etc, and also with whatever the Fossil sources for > representative members of the Unix / Linux ecosystem do. I'd suggest > looking at the sources provided by Debian (and or Ubuntu), Fedora, > perhaps also OpenSUSE or Arch or Gentoo, and also those provided by > one or two BSD derivatives, perhaps FreeBSD and or OpenBSD. (IIRC, > the person who maintains Fossil and I guess also SQLite for OpenBSD is > active on this list. I think some of the maintainers of Fossil for > various Linux distributions are also active here.) > > Note that if you *don't* do this yourself, you'll just make the > maintainers of Fossil for the various Unix / Linux distributions have > to work harder to do this themselves (and perhaps they'll do it in a > way you disagree with), because they *will* do it and their having to > do it themselves may impede their ability to provide versions of > Fossil which include embedded Lua support, which will adversely affect > you with respect to point (2) above. (Since the core Fossil > developers want their new versions of Fossil adopted by O/S > distributions in a timely fashion, and anything which impedes that > will be viewed in a negative light by them, I believe.) So, you might > as well do it this way to begin with, to make sure it gets done right. > > This isn't meant to be an exhaustive list of things to look out for, > of course. It's just my first quick pass over the notion. > > > > For the second point, you have to remember that while *you* see the > desirability if not need for the capability to embed and use Lua > within Fossil (and, this is NOT in any way insignificant!), it's not > yet obvious to others, particularly to others who are not experienced > users of Lua or familiar with its capabilities, that this is a > necessary or even desirable capability. > > In particular, if you desire for your work to be integrated into the > primary sources of Fossil where they will be potentially available to > anyone who uses such a version of Fossil, as opposed to merely being a > fork of Fossil which you yourself will have to maintain and keep up to > date with respect to the primary version of Fossil and which will only > be available to people who are aware of and seek out your fork, then > you'll need to convince the core developers of Fossil that not only is > it at the least desirable (if not necessary) for the primary version > of Fossil they develop, maintain, and support to have the capability > of using Lua as an embedded programming language with status equal to > that of TCL (which they are already intimately familiar with), but > that this can be done at a cost in terms of increased cost of > maintenance and support which they can afford and are willing to pay, > and that the benefits that end users gain by their doing this are > worth the cost to them of doing it. (After all, if they don't see why > it's worth the cost to do it, why should they?) > > Note that I *assume* you will ultimately desire and have as a goal for > your work to be integrated into the primary sources of Fossil if you > can achieve it. If that is unimportant to you, then obviously you can > ignore all the following. > > To this end, you'll need to keep in mind that the core developers of > Fossil are *not* necessarily expert in, or even familiar with, Lua. > As such, I think it likely they will initially see any request to > support embedded Lua in Fossil as something with a potential high or > even unbounded cost upon them in terms of requirements for future > maintenance and support of this capability, and with minimal or even > no benefit to users of Fossil. (After all, they *themselves* use > Fossil and don't use embedded Lua with it!) > > Therefore, you'll need to demonstrate the benefit of having embedded > Lua available, not just in theory but with one or more concrete > demonstrations of it. You'll need to also demonstrate that any > additional cost upon them to maintain and support this capability will > be no greater than they are willing to pay, and preferably as minimal > as possible. Ideally, you'll need to show that there are others, > preferably a significant number of others, besides yourself who would > benefit from this capability, again ideally by demonstrating that they > are already actively *using* the capability. (E.g. that it's worth > the Fossil developers paying the cost of maintenance for this > capability because it won't be going to waste.) > > You may also need to revise your approach to embedding Lua within > Fossil in response to feedback from the Fossil developers as to how > they see your work. Remember that while you're paying the upfront > cost of development of this capability, you're asking them to pay the > cost of maintaining and supporting it into the indefinite future. As > such, if there's two or more approaches to doing something where they > think the cost to them of the approach you choose will be greater long > term than that of a different approach, even if the different approach > costs more up front in development (e.g. to you), they may well ask > you to take the different approach. You may need to negotiate with > them concerning this. > > In this aspect, especially if you follow Linux Kernel News, it's much > as if you were a Linux kernel developer who's come up with some > wonderful new feature which you want accepted into the main kernel > sources (e.g. the version controlled by Linus Torvalds). But, when > you submit your proposed feature on the appropriate development > mailing list (be it the main Linux Kernel Mailing List or a subsystem- > or topic-specific list), you get responses back that you should have > considered approach B instead of A, or why didn't you consider > approach C, or did you consider doing something more generally, or or > or or. Often there will be several or *many* rounds of feedback, > where changes to the feature will be made in response to prior > feedback, and then there'll be feedback to those changes which result > in still more changes. The goal here isn't to subject a developer > with a new feature to having to run a gauntlet or gratuitously jump > through hoops just for the fun of it, but to make sure the new feature > is the best that it can be and to make as sure as possible that once > the feature is incorporated into a released kernel that nothing will > come back to bite people in terms of too expensive maintenance cost, > technical debt which has to be paid later, etc. > > On that same vein, if and when you are ready to propose having this > capability included into the primary sources for Fossil, set things up > much as a new capability or feature for the Linux kernel would ideally > be presented. Rather than have one giant patch that does everything > in one fell swoop, have several patches that each individually > accomplish one sub-goal or way point in terms of enabling the > capability, and which build on one another incrementally. It'll be a > lot easier for the Fossil developers to follow what you're doing step > by step and be convinced that what's being done is reasonable and > makes sense that way, I think. > > Further and in particular, I'd say you need to consider how to handle > having Fossil having an external optional dependency (e.g. on Lua) on > a project which the core Fossil developers do not themselves control > or are already intimately involved with. (SQLite doesn't count for > this as it's maintained by the same people who maintain Fossil and the > two are kept well in sync. TCL is an external dependency, but > presumably the core Fossil developers are already intimately involved > with the development of TCL due to the history of the projects > involved. Lua will be in neither category.) > > If Lua has a "stable" version, or better yet a "long term support" > version, available to choose from in terms of what you choose to > select to embed within Fossil, I would advise choosing that version if > at all possible. It'll likely reduce the cost of maintenance both for > the primary Fossil developers and maintainers, and also for the third > party maintainers of Fossil as provided by the various Unix / Linux > distributions. > > Oh, and remember the obvious things like following whatever coding > style guidelines exist for Fossil, etc, to make it as easy to merge if > and when that comes to pass as possible. Also, there's some legal > hoops you'll have to jump through before the core Fossil developers > will agree to accept your changes into the primary Fossil sources. > You can find out more about all of this on the primary web site for > Fossil (it should also be already included in your own personal copy > of Fossil) in the information about how to contribute to Fossil. > > > > In short, even if and when you get something *you* find to be suitable > in terms of providing a capability for embedded Lua within Fossil, it > may be a good while longer before others agree with you and until then > you may have to maintain and support your changes to Fossil by > yourself. And you may have to change how you do things one or more > times before the core Fossil developers agree to incorporate your > feature into the primary sources for Fossil. Keep in mind that "slow > and steady, and persistence, wins the race". > > > > I realize this is a very long response to your question, and that it > may come across as at least a little negative towards what you want to > do. Please don't think I am trying to dissuade you from attempting to > do this thing; I am not, in the least. But, I *am* trying to give my > opinion, as best as is possible, as to what I think you will need to > do to achieve what I (perhaps incorrectly) believe your ultimate goals > to be, so that you will have as accurate a picture as you can get of > the effort you'll need to make, and so you can decide if you really > want to make the effort or not. I'd hate for you to put a lot of work > into this only to become discouraged because it wasn't going as easily > as you believed it would go and end up giving up and wasting all the > time you'd spent to that point. > > Other than that, I hope this is of some use and interest to you. And > I thank you for your time in reading this message. > > Also, my apologies in advance if I've come across as, I don't know, > condescending or dismissive in any way. That was not my intent or > desire in any way whatsoever. Unfortunately, I have Male Answer > Syndrome, and sometimes it expresses itself in an undesirable fashion. > > Anyway. I wish you the very best of luck and success with this > effort, if you do decide to go ahead with it. I'm sure you'll find it > interesting if you do decide to pursue it. Be well. > > > > Joseph >
_______________________________________________ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
