See some quick reaction below. On Tue, Jul 7, 2020 at 3:25 PM Siegfried Goeschl < siegfried.goes...@gmail.com> wrote:
> Hi folks, > > I assembled the open discussions & TODOs - see below > > Thanks in advance, > > Siegfried Goeschl > > > 1. Open Discussions > > ============================================================================= > > > 1.1 Naming of CLI file > > ----------------------------------------------------------------------------- > > It is currently named "freemarker-cli" but it was suggested to call it > "freemarker-generator" - I'm happy with "freemarker-cli" since it > reflects the Maven project layout > To clarify, mainly for others, the idea behind that is that we have a single tool, called "freemarker-generator". That tool can be called through CLI for now, but in the future maybe can be also called Maven, Gradle, or directly through Java API. This unification didn't happen though; we only have a CLI tool. freemarker-generator-maven is currently totally separate, both technically and conceptually. My opinion about the naming: That you are calling the CLI interface is obvious when you are doing it, so it's not very helpful to restate. Seeing "freemarker-generator" tells more. Also people who use the "freemarker-cli" script will call the whole product "freemarker CLI", which wasn't our intent when we named this subroject freemarker-generator. Also note that the Maven module name is "freemarker-generator-cli" (not just "freemarker-cli"). 1.2 Complex Transformations using the CLI > > ----------------------------------------------------------------------------- > > That a large topic (seed templates, seed datasources, shared > datasources, aggregation versus generation) - currently low on my > priority list because > > * Transforming multiple templates and template directories is supported > * Personally I would not use a command-line tool for source code > generation but use a plugin (to avoid extra dependencies to be > installed) > Sure, for source code generation most will use a Maven/Gradle/whatever-build-tool plugin (regardless of the number of dependencies). But data-seeded-generation is not tied to source code generation (that was just what we happened to run into here). Also people may prefer doing non-source-code generation from Maven/Gradle/etc too (for example, to generate a site as part of the build), so template-seeded-generation is not tied to the CLI either. It's less guess-work regarding user needs if we let these mix, instead of separating features into specialized bubbles (like maven-plugin does data-seeded only, CLI does template-seeded only). This fragmentation also makes it more likely that some of these "bubbles" will become unmaintained, which is my primary fear when it comes to any FreeMarker satellite projects. To keep the number of dependencies under control, probably the data loaders should go into separate modules. For the CLI binary distribution it's fine to pack all of them (if licenses allow that), but for anything else, of course that would be awkward. > > Question being discussed are > > * Is this an important feature? > * Can it be implemented later (if required) without breaking stuff? > > > 1.3 Site Generation > > ----------------------------------------------------------------------------- > > Currently Maven & Markdown is used but it is suggested to migrate to > XDocBook to match the existing FreeMarker documentation. > > * Daniel is having a look at it > Yeah. I try to apply the same solution everywhere as much as possible here, given our constraints. > 1.4 Data Loaders > > ----------------------------------------------------------------------------- > > Currently it is possible to materialize certain files types directly in > the model using "-m" or "--data-model" parameter which parses the given > content based on the content type > > ``` > freemarker-cli --data-model > post=https://jsonplaceholder.typicode.com/posts/2 -i 'post title is: > ${post.title}'; echo > ``` > > It was suggested to change to something like > > ``` > -s 'post=json("https://jsonplaceholder.typicode.com/posts/2";)' > ``` > > which adds more flexibility > An additional major point there was that then pretty much the same could work from templates as well, where we currently use a different (and IMO quite verbose) approach. Two places (-m, and inside the template), same(ish) solution. Easier to grasp for users, and hopefully easier to maintain for us in the long term. > 1.5 Named URIs > > ----------------------------------------------------------------------------- > > Currently Named URIs are supported to provide a name and additional > metadats (mimetype, encoding) - the current implementation is not > complete but it should look like > > ``` > -s 'myTable=foo/bar.csv#format=EXCEL&delimiter=TAB' > ``` > > Current suggestion is to switch to something like > > ``` > -s 'myTable=csv("foo/bar.csv", {"format": "EXCEL", "delimiter": ";"})' > ``` > > 1.6 Exposing DataSources > > ----------------------------------------------------------------------------- > > Currently a class `DataSources` is used which keeps a list of data > sources and allows lookup by name (mixture of list and map) - we have > ongoing discussions > > * Does a DataSource has a name when not directly specified by the user? > * Shall we enforce unique data source names? > > > 1.7 Making Usage More FreeMarker Friendly > > ----------------------------------------------------------------------------- > > In FREEMARKER-148 I wrapped the `DataSources` into a `BeanModel` > allowing better FTL support, e.g. > > * `DataSources[0]` instead of `DataSources.get(0)` > * `DataSources["user"]` instead of `DataSource.get("user")` > Or rather: DataSources.user That's the same as DataSources["user"], but nicer. > > Care must be taken regarding name collisions of data sources and exposed > methods of `DataSources` > > > 1.8 Overlap Of CLI And Maven Plugin > > ----------------------------------------------------------------------------- > > Both artifacts reside in the same Maven build but currently the `Maven > Plugin` is completely independent > > * This might change in the future but I spend little time on the Maven > plugin > I don't think the maven plugin should be in freemaker-generator in this form. There's useful functionality there, but then it's practically like if we started another project next to what you have worked on, and, see my fragmentation argument earlier. We got enough to maintain. Anyway, rest of my stand point () it already known. > 1.9 Support Of Backward Compatibility > > ----------------------------------------------------------------------------- > > I lean towards 0.X.Y release and not guaranteeing backward compatibility > - little field testing and no real users. > > > 1.10 Support Of Groups > > ----------------------------------------------------------------------------- > > The Named URI allow to assign an optional group which can be used to > lookup up data sources > > ``` > -data-source 'myTable:data=foo/bar.csv' > ``` > > * I implemented it since I though it would be a nice feature but not so > sure about it any longer :-) > * It was suggest to swap name and group, e.g. `-data-source > 'data:myTable=foo/bar.csv'` > Can you show the intended use cases for groups? 2. TODOs > > ============================================================================= > > > 2.1 Discuss And Prioritise The Open Topics > > ----------------------------------------------------------------------------- > > Discuss and define what needs to be done for the first public release. > > > 2.2 Get The Release Process Going > > ----------------------------------------------------------------------------- > > FreeMarker project is using `Ant` for the release process (see > https://freemarker.apache.org/committer-howto.html) but that does not > work out-of-the-box with the `freemarker-generator` Maven project. > > > 2.3 Tool Access in FreeMarker Model > > ----------------------------------------------------------------------------- > > All tools are directly exposed which adds a lot of noise - might be a > map a better idea? > > ``` > tools["csv"] > tools["gson"] > ``` > The nicer syntax for the same: tools.csv tools.gson Seems to be a good idea to me. > > instead of > > ``` > ${CSVTool} > ${GsonTool} > ``` > > > 2.3 DataSource Access in FreeMarker Model > > ----------------------------------------------------------------------------- > > As discussed above a `DataSources` instance is exposed - maybe it should > be called `dataSources` since it might be only a dumb list/map? > I think so, but because it was never like a static utility, unlike CSVTool, etc. > On 5 Jul 2020, at 18:38, Daniel Dekany wrote: > > > On Sun, Jul 5, 2020 at 4:13 PM Siegfried Goeschl < > > siegfried.goes...@gmail.com> wrote: > > > >> Hi Daniel, > >> > >> No problem with the XDocBook if you feel strongly about it :-) Will > >> the > >> Maven site also be published somehow? > >> > >> Regarding releasing "freemarker-generator" > >> > >> * At the end of the day it is a command line tool people might use > >> sporadically - having said that the Maven plugin is a slightly > >> different > >> story > >> * As far as I'm concerned my use-cases (data-centric processing & > >> cloud > >> configuration stuff) are reasonably working > >> * I guess we can spend another 12 months discussing assumptions, > >> architectural decisions and implementation details (while > >> appreciating your > >> knowledge and insights) > >> > > > > Let's not do that, but it's not what is happening either, as far as I > > see. > > There were like 3(ish) issues of the more fundamental nature (not > > details) > > that I brought up. Actually, re-raised them now after a few months, > > because > > I see no closure on your side. I mean, you agree but can't invest into > > it, > > or you don't think these serve real demands, or... what is it? > > > > I also said that as far as I'm concerned, we can do a release, if > > users are > > properly informed about what we do not promise. > > > > > >> * As long as there is no release & real users out there those > >> discussion > >> are mostly an intellectual exercise > >> > > > > In general, yes, but it depends on the concrete cases. For example, > > data-file-seeded generation. The Maven plugin donators did that at > > work. > > FMPP users did both directions too. Those are real users. That we > > don't > > address that can have some valid reasons, but not having real/proven > > value > > is not one of them. Or at least I guess you will agree with that. > > > > > >> * There is a danger that we never get a release out as it happened > >> with > >> the Maven plugin > >> > > > > The maven plugin was abandoned by the donators right after the > > donation. > > Nobody cared, or had the time they may wanted to have, so nobody did > > anything with it. (That's really the #1 issue with OS development, > > when > > there's no company with paid developers behind it.) > > > > > >> I think the code base went a long way already so we should clearly > >> define > >> what's ABSOLUTELY missing to get it out the door . > >> > > > > Sure, let's see what others find (I have already told my > > observations). > > > > > >> Please note that there are a couple of additional tasks actually to > >> create > >> a user base > >> > >> * No idea how to get the Apache release procedures working (signing, > >> staging, etc ..) > >> > > > > It's documented here, but we (or at least I) will help: > > https://freemarker.apache.org/committer-howto.html > > Whatever you will find missing from there, we should add. > > > > > >> * Need to look int Brew and Linux distributions > >> > > > > And Windows, if you have access... tests fail on that for example. > > (Some > > charset or line ending mess... didn't investigate yet.) > > > > > >> * Getting some public awareness - blogs, presentations, ApacheCon > >> anyone? > >> > > > > Tweet is where we have some people... and of course menu item on > > freemarker.apache.org. > > > > Thanks in advance, > >> > >> Siegfried Goeschl > >> > >> > >>> On 05.07.2020, at 15:18, Daniel Dekany <daniel.dek...@gmail.com> > >>> wrote: > >>> > >>> Find answers inline. > >>> > >>> On Sun, Jul 5, 2020 at 9:31 AM Siegfried Goeschl < > >>> siegfried.goes...@gmail.com <mailto:siegfried.goes...@gmail.com>> > >>> wrote: > >>> > >>>> Hi Daniel, > >>>> > >>>> thanks for your feedback ... > >>>> > >>>> # 1. Site Generation > >>>> > >>>> Maven is not soo bad - I uploaded the generated site to > >>>> http://www.senilesoftwareengineer.org/freemarker-generator/ - I > >>>> personally have no problem using FreeMarker documentation approach > >>>> but > >>>> decided no to follow it up for the time being due to time > >>>> constraints. > >>>> > >>> > >>> So if I convert the markdown to XDocBook, and set up site generation > >>> for > >>> it, and thereafter you will edit XDocBook (I recommend XXE for > >>> that), > >> will > >>> that be OK with you? > >>> > >>> # 2. Design/Architectural Doubts > >>>> > >>>> ## 2.1 Parsed Data > >>>> > >>>> Currently we can load files/URLs as "DataSource" (which need to be > >>>> processed in FTL) or we can use "-m / --data-model" to parse and > >>>> expose > >>>> the content directly (see > >>>> > >>>> > >> > http://www.senilesoftwareengineer.org/freemarker-generator/freemarker-generator-cli/cli/concepts/data-models.html > >>>> ). > >>>> > >>> > >>> (Yeah, that's where I said that it looks like a quick ad-hoc > >>> solution, > >> and > >>> as such is quite limited.) > >>> > >>> Regarding a less programmatic way - Named URIs are used (see > >>>> > >>>> > >> > http://www.senilesoftwareengineer.org/freemarker-generator/freemarker-generator-cli/cli/concepts/named-uris.html > >> < > >> > http://www.senilesoftwareengineer.org/freemarker-generator/freemarker-generator-cli/cli/concepts/named-uris.html > >>> ) > >>>> > >>>> to pass additional metadata but the implementation requires some > >>>> re-work > >>>> to pass arbitrary metadata in the fragment. > >>>> > >>>> E.g. what should be possible is > >>>> > >>>> ``` > >>>> freeMarker-cli -t some.ftl -m > >>>> > >>>> > >> > user=somepath/user.csv#delimiter=TAB&format=DEFAULT&charset=UTF-16&header=false > >>>> ${CSVtool.parse(DataSources["user"])} > >>>> ``` > >>>> > >>> The idea is to be able to pass all metadata being used to parse the > >>> data > >>>> source - that would be also helpful for defining a SQL data source. > >>>> > >>> > >>> Yes. But I guess we should break out from the preconception that a > >>> data > >>> source must be an URL, and also that therefore we have to encode all > >>> the > >>> parsing and selection parameters into the anchor as name-value pairs > >> (which > >>> also implies that you might have % escapes). Because we sometimes > >>> List > >> and > >>> Map<String, Object>-like data structures as well, etc. (Also, using > >>> the > >>> anchor is kind of an abuse of the URL syntax, and &name=value is > >>> query > >>> parameter syntax, not anchor syntax. So this is maybe confusing for > >> users, > >>> like they will keep writing ? instead of #.) > >>> > >>> My idea is that a Data Source should be the name of a Data Loader > >>> (like, > >>> "csv", "excel", "jdbc", etc.), plus its parameters. To specify that, > >>> we > >>> should use FTL method call syntax, because that syntax the users > >>> have to > >>> learn anyway, and it's still relatively expressive (and can be > >>> re-used in > >>> actual templates - see that later): > >>> -s 'myTable=csv("foo/bar.csv", {"format": "EXCEL", "delimiter": > >>> ";"})' > >>> > >>> If we have this (which has basically the same meaning as it has > >> currently): > >>> -s myRawTable=foo/bar.csv > >>> that could be just a shorthand for: > >>> -s myRawTable='raw("foo/bar.csv")' > >>> > >>> So "raw" data is what you pass to "tools" for parsing in the current > >>> version. > >>> > >>> If you prefer do parsing in the templates, then you can reuse what > >>> you > >>> learned about -s, as it's pretty much the same: > >>> <#assign myTable = DataLoaders.csv(DataSources.myRawTable, { > >>> "format": > >>> "EXCEL", "delimiter": ";" })> > >>> > >>> Also, if you decide to parse data inside the template, doing it will > >>> be > >>> very similar as in the -s switch. > >>> > >>> > >>>> ## Data versus Template Driven Generation > >>>> > >>>> Currently a variety of transformations are supported on the command > >>>> line > >>>> (see > >>>> > >>>> > >> > http://www.senilesoftwareengineer.org/freemarker-generator/freemarker-generator-cli/cli/concepts/transformation.html > >>>> ) > >>>> > >>>> * single template > >>>> * multiple templates > >>>> * directory of templates > >>>> > >>>> Regarding "template versus data driven" generation - CLI does not > >>>> support data-driven generation where one output file is generated > >>>> per > >>>> data source. > >>>> > >>> > >>> Yes. That's exactly what I feel is a major mistake on the long run. > >>> At > >>> least we should be able to add it later, but I guess, to ensure > >>> that, we > >>> pretty much have to implement it. So what will happen with this, > >>> what's > >>> your standpoint? > >>> > >>> Furthermore, I don't think that we should have a "mode" for this, > >>> which > >>> applies for a whole run. I think I'm repeating myself from some > >>> months > >> ago, > >>> but, to recap, there are simply "seeds" that cause generating an > >>> output > >>> file. That "seed" can be a template, or a data file, or part of > >>> loaded > >> data > >>> (like each row in a table is a "seed"). And I believe it's worth it > >>> to > >>> allow mixing these in the same run, because then > >>> freemarker-generator > >> will > >>> be good enough fit for more use-cases. So for example, you have some > >>> JSON > >>> files in your project, each will generate a java file (typical > >>> source > >> code > >>> generation), but you also have templates for pom.xml, etc., that > >>> will > >>> generate its own output. > >>> > >>> ## 2.2 Convergence of Maven Plugin & CLI > >>>> > >>>> It is on my list but does not have high priority since it should > >>>> not > >>>> affect Maven plugin usage (regarding backward compatibility). > >>>> > >>> > >>> But that would be a completely new pugin, I assume. > >>> > >>> # 3. Maturity / Backward Compatibility > >>>> > >>>> The project is not mature - we have no real users and little field > >>>> testing - in short I don't want to guarantee backward-compatibility > >>>> for > >>>> now because some of my decisions might turn out to be plain wrong > >>>> (or > >>>> even stupid) :-) > >>>> > >>>> According to https://semver.org we can release 0.x.y and follow > >>>> https://semver.org/#spec-item-4 - not so nice but pragmatic. > >>>> > >>> > >>> > >>>> # 4. General Considerations > >>>> > >>>> There is a danger we never get "freemarker-generator" out of the > >>>> door > >>>> ("perfect is the enemy of good") therefore the whole PMC should > >>>> look > >>>> into the questions/answers > >>> > >>> * "freemarker-generator" needs community usage to become really > >>> useful - > >>>> the unreleased stuff does not add any value > >>>> > >>> * Is it acceptable to release a 0.X.Z where we not > >>> backward-compatible > >>>> until 1.0.0? > >>>> > >>> > >>> With the clear warning that we won't maintain this "branch" (a > >>> series of > >>> backward compatible releases), etc. we can release. That, however, > >>> discourages usage. And that's really the problem, not that we can't > >>> release, because we can publish stuff in whatever development stage > >>> (assuming users will be well informed). But if we are to hope for > >>> any > >>> significant user base, I believe we should be able to promise some > >>> stability, and maintenance of the BC branch. That's why I'm saying > >>> that > >> we > >>> should get the core architecture about right, and the project scope. > >>> (I'm > >>> not talking about missing features that can be added later, nor > >>> about > >> bugs > >>> and other rough edges.) > >>> > >>> > >>>> * Are there any deal breakers which needs to fixed before we start > >>>> with > >>>> a first public release? > >>>> > >>> > >>> As far as I'm concerned (but I'm just one of the PMC members), if > >>> you > >> have > >>> strong feelings for doing a release, with the proper warnings, sure, > >>> I > >> will > >>> collaborate. The deal breaker is more for the users, who get no good > >>> BC > >>> promise or maintenance. > >>> > >>> Thanks in advance, > >>>> > >>>> Siegfried Goeschl > >>>> > >>>> On 4 Jul 2020, at 1:10, Daniel Dekany wrote: > >>>> > >>>>> It should be in https://freemarker.apache.org/generator/. You will > >>>>> need to > >>>>> commit/push into > >>>>> https://github.com/apache/freemarker-site/tree/asf-site > >>>>> for that. (See > >>>>> https://freemarker.apache.org/committer-howto.html#updating-homepage > ) > >>>>> > >>>>> We need to generate the web pages somehow though. I saw you try to > >>>>> do > >>>>> that > >>>>> with Maven "site", but personally, I can't imagine that it can be > >>>>> tweaked > >>>>> to generate reasonable output. Not sure how others see this. Maven > >>>>> "site" > >>>>> is basically the Maven model dumped into HTML pages, and "reports" > >>>>> like > >>>>> even the log of the Rat run etc. Dozens of menus and sub-pages for > >>>>> details > >>>>> users mostly don't care about. The actual content users do care > >>>>> about > >>>>> is > >>>>> stuffed under "About", and from there it's not properly navigable > >>>>> or > >>>>> searchable, as apparently it's assumed to be a single page. I know > >>>>> back > >>>>> then you didn't want to use the tool that's used both for the main > >>>>> web > >>>>> site > >>>>> and for the FreeMarker documentation, but I think that's the most > >>>>> economical solution currently (and then we also have common > >>>>> styling, > >>>>> and > >>>>> common place to fix whatever technical issues), so please > >>>>> reconsider > >>>>> that. > >>>>> (I can help setting that up, of course.) > >>>>> > >>>>> Now, some more difficult-to-address problematic things. My problem > >>>>> is > >>>>> that > >>>>> these just weren't concluded. Discussion just died. To be > >>>>> concluded > >>>>> can > >>>>> even mean that other PMC members say we should step over these, > >>>>> even > >>>>> if I > >>>>> disagree. But these should be understood, and considered. So, > >>>>> these > >>>>> are: > >>>>> > >>>>> - Design/architectural doubts that are probably not realistic to > >>>>> address > >>>>> much later: > >>>>> - Currently, data sources are just URL-s (locations), and > >>>>> templates > >>>>> are meant to call tool API-s to parse them. As I said back > >>>>> then, > >>>>> one > >>>>> consequence is that then, you can't put parsed data into the > >>>>> data-model > >>>>> shared by all templates (i.e., via -m/--data-model). Because, > >>>>> you have no > >>>>> convenient/concise way to load the actual (parsed) data, > >>>>> instead > >>>>> you have > >>>>> to "program it" in FTL, because the assumption was that you > >>>>> only > >>>>> want to do > >>>>> that in templates anyway. Furthermore, as I saw it just now, > >>>>> --data-model > >>>>> actually supports some ad-hoc way of parsing the data pointed > >>>>> by > >>>>> a > >>>>> data-source-like URL, as JSON, YAML, maybe some more. Here's > >>>>> an > >>>>> example of > >>>>> that: `-m config=env:///DB_CONFIG#mimetype=application/json`. > >>>>> So > >>>>> there is a > >>>>> need actually. But compared to what you can do in templeats, > >>>>> it's totally > >>>>> different, and of course very restrictive. I was also looking > >>>>> for a less > >>>>> "programmatic" way of loading data because even doing it in > >>>>> templates is > >>>>> not very friendly as it is. (For ultimate flexibility you > >>>>> might > >>>>> need > >>>>> program logic for sure, but certainly not just to grab a > >>>>> worksheet from an > >>>>> Excel file.) We also should have a POC for loading from a > >>>>> less > >>>>> file centric > >>>>> data source, like from a DB with a SQL query, to see if the > >>>>> concept works. > >>>>> - Not sure what's with template-driven VS data driven output > >>>>> generation. Data driven is when you generate one output file > >>>>> per > >>>>> some data > >>>>> unit (like per JSON file, or per DB record), and all is > >>>>> transformed by a > >>>>> common template. How would that look currently in > >>>>> freemarker-cli? > >>>>> - The Maven plugin provides very different functionality than > >>>>> the > >>>>> CLI. > >>>>> The original concept was that they are just two interfaces to > >>>>> the > >>>>> same > >>>>> product. That was also brought up, and I don't think it was > >>>>> addressed much, > >>>>> like, why you disagree, if you do, or what's going on. Anyway, > >>>>> regarding > >>>>> what we have now. The naming implies that, as both are > >>>>> "FreeMarker > >>>>> Generator", just one is "CLI", and the other is "Maven". But as > >>>>> far > >>>>> as I > >>>>> see, they are two different "products" really, focusing on > >>>>> different use > >>>>> cases. So, if it stays like this, then some of them have to be > >>>>> renamed at > >>>>> least. Or I don't know what others think we can do. (Then it's > >>>>> still > >>>>> somewhat weird to release them together.) > >>>>> > >>>>> (I have also run through the CLI documentation, and found a lot of > >>>>> less > >>>>> fundamental things to address, but I don't want to overload the > >>>>> thread.) > >>>>> > >>>>> Also, if the project is released mostly as it is now, what will we > >>>>> promise > >>>>> regarding backward compatibility? What do we communicate about the > >>>>> matureness of the project? > >>>>> > >>>>> On Wed, Jul 1, 2020 at 8:29 AM Siegfried Goeschl < > >>>>> siegfried.goes...@gmail.com> wrote: > >>>>> > >>>>>> Hi folks, > >>>>>> > >>>>>> I'm pretty much finished with the stuff I would like to have for > >>>>>> the > >>>>>> first > >>>>>> public release but there is a still a lot of work ahead > >>>>>> > >>>>>> * FREEMARKER-148 [freemarker-cli] Make usage of "DataSources" > >>>>>> more > >>>>>> "Freemarker" like > >>>>>> * FREEMARKER-150 [freemarker-cli] Setup "freemarker-generator" > >>>>>> web > >>>>>> site > >>>>>> * Setup the release process (Daniel, I guess I need some help > >>>>>> here) > >>>>>> * An iteration of reviews and discussions > >>>>>> > >>>>>> Some thought along the line > >>>>>> > >>>>>> * I would like to finish FREEMARKER-148 this week > >>>>>> * Daniel, any suggestion where to host the public website? > >>>>>> > >>>>>> Thanks in advance, > >>>>>> > >>>>>> Siegfried Goeschl > >>>>>> > >>>>>> > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> Daniel Dekany > >>>> > >>> > >>> > >>> -- > >>> Best regards, > >>> Daniel Dekany > >> > >> > > > > -- > > Best regards, > > Daniel Dekany > -- Best regards, Daniel Dekany
