That's a good suggestion. One thing I want to point out that Builtins.java [1] contains all the dml-bodied and java-bodied built-ins. For the dml built-ins, it is easier to parse the corresponding dml script (if documentation is available there), but for other built-ins, I am not aware of any particular source today where they are documented -- a few of them are documented as a part of commit message but not all. Though it is certainly possible to slice the dml builtins from [1] out, but it will be awesome to generate docs for all of those.
[1] https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java Regards, Arnab.. On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <baunsga...@tugraz.at> wrote: > Hi dev team! > > > Update / suggestion. > > > First of all, i think it is great that we have people working on our > documentation. It is really important. But I can't help to note that the > improvements, are going in an unmaintainable direction where we have to > maintain the documentation at multiple locations. > > > For instance our new toOneHot function already had some basic > documentation inside its build-in script already [1], it would be much > nicer if we modified that and then had some automation to generate the docs > made in commit [2]. Now we have to maintain two locations, resulting in > high risk of either being outdated. > > > We need to ask ourselves where do we want to make the edit of > documentation. I would argue that it should be located as close to the > internal definition of a function, because then when making or modifying > that function the documentation is also reviewed. > > > To achieve this i suggest that we instead relocate the documentation > source and generation to an appropriately named file inside [3], that > simply loops through all function definitions from [4], and generate > markdown files for our /docs/ folder. > > > Shifting to such an implementation enables us to systematically cover all > functions defined in the system. It also is a full list of the systems > current build-in functions thereby giving us the exact file paths to > extract the source documentation from the comments and generate the docs > for those files. > > In the future it could be extended to verify and detecting which input and > output types each function definition has, such that we have even less > manual documentation needs. > > > Best regards > > Sebastian > > > [1] > https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml > > [2] > https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8 > > [3] > https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common > > [4] > https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java > > > ________________________________ > From: arnab phani <phaniar...@gmail.com> > Sent: Sunday, May 31, 2020 3:15:44 PM > To: dev@systemml.apache.org > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > Hi, > > Another advantage of markdown syntax is that it gives more freedom to add a > description as needed. Java methods and builtins might not fall in the same > bucket and different builtins might need different ways to describe it. > For example, glm needs more details than other built-ins. Too much > standardization can be a bad idea. > > Regards, > Arnab. > > On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian < > baunsga...@tugraz.at> > wrote: > > > Hi Janardhan, > > > > > > That would be one option, lets call it option 1, I have another > suggestion: > > > > > > 1. Markdown syntax with out commented individual lines. > > 2. Jdoc like syntax with annotations > > > > Pros and cons > > > > * 1 Pros > > * Easy Syntax most ppl know it. > > * Easy to implement. > > * 1 Cons > > * Enables custom docs (everyone make their own distinct format > > again) > > * No standard way of verifying correctness of the docs (correct > > parameter names etc.) > > * No standard way of marking Input, return parameters. > > * Requires modified Syntax with # at each line. where # normally > > means header in markdown > > * 2 Pros > > * Easy Syntax Java ppl know it > > * Standard way of making input > > * Do not have to change any syntax at all from Jdoc > > * Already supported syntax in DML > > * 2 Cons > > * Harder to implement since one would have to look into java doc > > extraction and what is needed to support that, maybe we have to make > such a > > thing ourselves? > > > > I personally like option 2 with Jdoc more and then extending into > > automatically parsing and making the either markdown files or HTML files > > for the docs that you either way have to do in option 1. > > > > Best regards > > Sebastian > > ________________________________ > > From: Janardhan <janardhan.pulivar...@gmail.com> > > Sent: Sunday, May 31, 2020 9:25:43 AM > > To: dev@systemml.apache.org > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > Hi Sebastian, :) > > > > Since the builtin DML files do not have full documentation yet, Can we > > write > > markdown syntax with the following heading in each dml file itself! > > > > 1. Function description > > 2. Usage > > 3. Arguments > > 4. Returns > > 5. Usage > > > > After that, as you've mentioned can be parsed with only removing # at the > > start of each line, and write > > all this data to one `builtins-reference.md` file. > > > > Thank you, > > Janardhan > > > > On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian < > > baunsga...@tugraz.at> > > wrote: > > > > > > > > If this is meant as a question then, yes from me (this probably calls > for > > > a vote?), if: > > > > > > > > > 1. someone is able to change the settings for the repository, to use > > > the docs folder as the GitHub docs folder. > > > 2. someone wants to make a PR containing the content of the gh-pages > > > branch. > > > > > > Best regards > > > Sebastian > > > ________________________________ > > > From: Janardhan <janardhan.pulivar...@gmail.com> > > > Sent: Tuesday, May 26, 2020 1:22:11 PM > > > To: dev@systemml.apache.org > > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > > > Hi Sebastian, > > > > > > Can we move the `docs` folder in the existing `master` branch > > > and merge the `gh-pages` branch to `docs` folder in the top-level > > > directory. :) > > > > > > - Janardhan > > > > > > On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian < > > > baunsga...@tugraz.at> > > > wrote: > > > > > > > Hi Janardhan, > > > > > > > > > > > > Reworking the documentation seems like a really good idea! > > > > > > > > > > > > A first step in my opinion is to start in the main repository. > > > > > > > > > > > > I would suggest the following: > > > > > > > > > > > > - Merge the current gh-pages branch into master docs folder. > > > > > > > > - Change the documentations page : http://apache.github.io/systemml/ > SystemDS Documentation - SystemDS<http://apache.github.io/systemml/> > apache.github.io > SystemDS Documentation > > > > to > > > > use the docs folder for documentation. Should be doable inside the > > > settings > > > > on the repository. > > > > > > > > - Delete the gh-pages branch > > > > > > > > > > > > I want this change because then new PRs can change the documentation > in > > > > one PR rather than two (which to be honest is not going to happen > most > > of > > > > the time). > > > > > > > > The downside (we have to mention this) is that when you clone the > > > > repository, you have slightly larger downloads, but i think the > > trade-off > > > > is fair. If you see any other issues please mention these in this > > thread. > > > > > > > > > > > > Next task I would like to suggest to include: > > > > > > > > > > > > - Java docs generated and > > > > > > > > - Python docs generated to also be included as a sub-folder in docs, > so > > > > that the current webpage contains the docs for that to. > > > > > > > > > > > > How to include these in a sensible manner is currently a good > question, > > > > and has to be seriously well considered. Including these will also > > enable > > > > us to move documentation closer to the individual code parts, making > it > > > > again more likely to get the documentation done. > > > > > > > > > > > > Another more programmatic task is to make a parser for our DML files, > > > such > > > > that we can generate webpage documentation based on the documentation > > > > syntax used (much like the python and java docs generator). This > would > > be > > > > great since it will remove the duplication of documentation in the > > > > individual DML files, and the documentation we have already in files > > such > > > > as: docs/dml-language-reference.md. > > > > > > > > This also gives us an excuse to clean up that documentation/scripts > > which > > > > has very different implementations in our scripts folder: > > > > > > > > > > > > examples: > > > > > > > > > > > > - <https://github.com/apache/systemml/tree/master/scripts> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > https://github.com/apache/systemml/tree/master/scripts> > > systemml/scripts at master · apache/systemml · GitHub< > https://github.com/apache/systemml/tree/master/scripts> > github.com > Mirror of Apache SystemML. Contribute to apache/systemml development by > creating an account on GitHub. > > > > > > > > > > > > https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml > > > > systemml/batch_norm1d.dml at master · apache/systemml · GitHub< > https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml > > > github.com > Mirror of Apache SystemML. Contribute to apache/systemml development by > creating an account on GitHub. > > > > > > < > > > > > > > > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml > [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< > https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml > > > > systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub< > https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml > > > github.com > Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development > by creating an account on GitHub. > > > > > > > > > > > > > > > - < > > > > > > > > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml > [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< > https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml > > > > systemml/glm.dml at master · Baunsgaard/systemml · GitHub< > https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml > > > github.com > Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development > by creating an account on GitHub. > > > > > > > > > > <https://github.com/apache/systemml/tree/master/scripts> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > https://github.com/apache/systemml/tree/master/scripts> > > systemml/scripts at master · apache/systemml · GitHub< > https://github.com/apache/systemml/tree/master/scripts> > github.com > Mirror of Apache SystemML. Contribute to apache/systemml development by > creating an account on GitHub. > > > > > > > > > > > > https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> > > systemml/Cox.dml at master · apache/systemml · GitHub< > https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> > github.com > Mirror of Apache SystemML. Contribute to apache/systemml development by > creating an account on GitHub. > > > > > > > > > > > > > > > > > > Furthermore, I would suggest to postpone changing the main website: > > > > > > > > https://github.com/apache/systemml-website > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > https://github.com/apache/systemml-website> > > GitHub - apache/systemml-website: Mirror of Apache SystemML site< > https://github.com/apache/systemml-website> > github.com > Mirror of Apache SystemML site. Contribute to apache/systemml-website > development by creating an account on GitHub. > > > > > > > > > > But that task has to be started when the next release is scheduled, > > > > because then we need to find out how to copy the current > documentation > > > to a > > > > archive list of static webpages located: > > > > > > > > https://systemml.apache.org/documentation > Documentation<https://systemml.apache.org/documentation> > systemml.apache.org > Project Documentation Page > > > > > > > > > > > > > > best regards > > > > > > > > Sebastian > > > > > > > > ________________________________ > > > > From: Janardhan <janard...@apache.org> > > > > Sent: Sunday, May 24, 2020 11:02:53 AM > > > > To: dev@systemml.apache.org > > > > Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > > > > > Hi Sebastian, > > > > > > > > In our discussion[1][2] a while ago, there was a topic about changing > > > > website > > > > stack. > > > > > > > > Let us start a discussion about this in this thread. > > > > > > > > We are planning to work on this, after a feasibility study and tech > > stack > > > > vetting. > > > > > > > > Anybody would like to give suggestions would be great. > > > > > > > > [1] https://github.com/apache/systemml/pull/877 > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > https://github.com/apache/systemml/pull/877> > > [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by > Baunsgaard · Pull Request #877 · apache/systemml · GitHub< > https://github.com/apache/systemml/pull/877> > github.com > Brute change of documentation page to reflect name change. To fully > enforce this two more steps are needed: > https://github.com/apache/systemml-website Access to SystemML website > server on https://systemml.apache.org/ Furthermore, we might want to > consider starting fresh in the documentation. > > > > > > [2] https://github.com/apache/systemml-website/pull/66 > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > https://github.com/apache/systemml-website/pull/66> > > [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request > #66 · apache/systemml-website · GitHub< > https://github.com/apache/systemml-website/pull/66> > github.com > Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used, > and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, since > mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art. > Commit package-lock.json file to version control. > > > > > > > > > > Thank you, > > > > Janardhan > > > > > > > > > >
