Hi Dev,
I merged in the documentation PR, this was an slight oversight, while trying to merge something else. That said nothing bad happened except you did not get a chance to look through the PR, and it is progress in the direction discussed in this thread. Best regards Sebastian ________________________________ From: Matthias Boehm <mboe...@gmail.com> Sent: Saturday, June 6, 2020 7:03:19 PM To: dev@systemml.apache.org Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. from my perspective it would be very important to have all builtin functions in a single markdown file to allow users to search for things and users don't care how a builtin function is internally implemented. So we might want to use this opportunity to consolidate the already documented builtin functions into the new list and add everything that is not yet documented as I'm sure many functions are missing. Btw, in the INFRA ticket for renaming our repositories [1], I also asked to disable sending all these github comments and PR interactions to our dev list as I'm sure people get overwhelmed by these auto-generated emails which leads to important messages being missed. Related to documentation, we might want to group categories of builtin functions into larger PRs instead of having a PR per builtin function as its primarily a copy of existing documentation so they can be reviewed more efficiently together to avoid unnecessary overhead. [1] https://issues.apache.org/jira/browse/INFRA-20362 Regards, Matthias On 6/6/2020 5:23 PM, Janardhan wrote: > Hi Arnab, > > We have all the Java-based builtin functions documented at [1]. > Most of these functions are already documented! > > [1] > http://apache.github.io/systemml/dml-language-reference.html#built-in-functions > > > Thank you, > Janardhan > > On Fri, Jun 5, 2020 at 4:37 PM arnab phani <phaniar...@gmail.com> wrote: > >> That's a good suggestion. One thing I want to point out that Builtins.java >> [1] contains all the dml-bodied and java-bodied built-ins. For the dml >> built-ins, it is easier to parse the corresponding dml script (if >> documentation is available there), but for other built-ins, I am not aware >> of any particular source today where they are documented -- a few of them >> are documented as a part of commit message but not all. >> Though it is certainly possible to slice the dml builtins from [1] out, but >> it will be awesome to generate docs for all of those. >> >> [1] >> >> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java >> >> Regards, >> Arnab.. >> >> On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian < >> baunsga...@tugraz.at> >> wrote: >> >>> Hi dev team! >>> >>> >>> Update / suggestion. >>> >>> >>> First of all, i think it is great that we have people working on our >>> documentation. It is really important. But I can't help to note that the >>> improvements, are going in an unmaintainable direction where we have to >>> maintain the documentation at multiple locations. >>> >>> >>> For instance our new toOneHot function already had some basic >>> documentation inside its build-in script already [1], it would be much >>> nicer if we modified that and then had some automation to generate the >> docs >>> made in commit [2]. Now we have to maintain two locations, resulting in >>> high risk of either being outdated. >>> >>> >>> We need to ask ourselves where do we want to make the edit of >>> documentation. I would argue that it should be located as close to the >>> internal definition of a function, because then when making or modifying >>> that function the documentation is also reviewed. >>> >>> >>> To achieve this i suggest that we instead relocate the documentation >>> source and generation to an appropriately named file inside [3], that >>> simply loops through all function definitions from [4], and generate >>> markdown files for our /docs/ folder. >>> >>> >>> Shifting to such an implementation enables us to systematically cover all >>> functions defined in the system. It also is a full list of the systems >>> current build-in functions thereby giving us the exact file paths to >>> extract the source documentation from the comments and generate the docs >>> for those files. >>> >>> In the future it could be extended to verify and detecting which input >> and >>> output types each function definition has, such that we have even less >>> manual documentation needs. >>> >>> >>> Best regards >>> >>> Sebastian >>> >>> >>> [1] >>> >> https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml >>> >>> [2] >>> >> https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8 >>> >>> [3] >>> >> https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common >>> >>> [4] >>> >> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java >>> >>> >>> ________________________________ >>> From: arnab phani <phaniar...@gmail.com> >>> Sent: Sunday, May 31, 2020 3:15:44 PM >>> To: dev@systemml.apache.org >>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. >>> >>> Hi, >>> >>> Another advantage of markdown syntax is that it gives more freedom to >> add a >>> description as needed. Java methods and builtins might not fall in the >> same >>> bucket and different builtins might need different ways to describe it. >>> For example, glm needs more details than other built-ins. Too much >>> standardization can be a bad idea. >>> >>> Regards, >>> Arnab. >>> >>> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian < >>> baunsga...@tugraz.at> >>> wrote: >>> >>>> Hi Janardhan, >>>> >>>> >>>> That would be one option, lets call it option 1, I have another >>> suggestion: >>>> >>>> >>>> 1. Markdown syntax with out commented individual lines. >>>> 2. Jdoc like syntax with annotations >>>> >>>> Pros and cons >>>> >>>> * 1 Pros >>>> * Easy Syntax most ppl know it. >>>> * Easy to implement. >>>> * 1 Cons >>>> * Enables custom docs (everyone make their own distinct format >>>> again) >>>> * No standard way of verifying correctness of the docs (correct >>>> parameter names etc.) >>>> * No standard way of marking Input, return parameters. >>>> * Requires modified Syntax with # at each line. where # normally >>>> means header in markdown >>>> * 2 Pros >>>> * Easy Syntax Java ppl know it >>>> * Standard way of making input >>>> * Do not have to change any syntax at all from Jdoc >>>> * Already supported syntax in DML >>>> * 2 Cons >>>> * Harder to implement since one would have to look into java doc >>>> extraction and what is needed to support that, maybe we have to make >>> such a >>>> thing ourselves? >>>> >>>> I personally like option 2 with Jdoc more and then extending into >>>> automatically parsing and making the either markdown files or HTML >> files >>>> for the docs that you either way have to do in option 1. >>>> >>>> Best regards >>>> Sebastian >>>> ________________________________ >>>> From: Janardhan <janardhan.pulivar...@gmail.com> >>>> Sent: Sunday, May 31, 2020 9:25:43 AM >>>> To: dev@systemml.apache.org >>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. >>>> >>>> Hi Sebastian, :) >>>> >>>> Since the builtin DML files do not have full documentation yet, Can we >>>> write >>>> markdown syntax with the following heading in each dml file itself! >>>> >>>> 1. Function description >>>> 2. Usage >>>> 3. Arguments >>>> 4. Returns >>>> 5. Usage >>>> >>>> After that, as you've mentioned can be parsed with only removing # at >> the >>>> start of each line, and write >>>> all this data to one `builtins-reference.md` file. >>>> >>>> Thank you, >>>> Janardhan >>>> >>>> On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian < >>>> baunsga...@tugraz.at> >>>> wrote: >>>> >>>>> >>>>> If this is meant as a question then, yes from me (this probably calls >>> for >>>>> a vote?), if: >>>>> >>>>> >>>>> 1. someone is able to change the settings for the repository, to >> use >>>>> the docs folder as the GitHub docs folder. >>>>> 2. someone wants to make a PR containing the content of the >> gh-pages >>>>> branch. >>>>> >>>>> Best regards >>>>> Sebastian >>>>> ________________________________ >>>>> From: Janardhan <janardhan.pulivar...@gmail.com> >>>>> Sent: Tuesday, May 26, 2020 1:22:11 PM >>>>> To: dev@systemml.apache.org >>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. >> Thanks. >>>>> >>>>> Hi Sebastian, >>>>> >>>>> Can we move the `docs` folder in the existing `master` branch >>>>> and merge the `gh-pages` branch to `docs` folder in the top-level >>>>> directory. :) >>>>> >>>>> - Janardhan >>>>> >>>>> On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian < >>>>> baunsga...@tugraz.at> >>>>> wrote: >>>>> >>>>>> Hi Janardhan, >>>>>> >>>>>> >>>>>> Reworking the documentation seems like a really good idea! >>>>>> >>>>>> >>>>>> A first step in my opinion is to start in the main repository. >>>>>> >>>>>> >>>>>> I would suggest the following: >>>>>> >>>>>> >>>>>> - Merge the current gh-pages branch into master docs folder. >>>>>> >>>>>> - Change the documentations page : >> http://apache.github.io/systemml/ >>> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/> >>> apache.github.io >>> SystemDS Documentation >>> >>> >>>> to >>>>>> use the docs folder for documentation. Should be doable inside the >>>>> settings >>>>>> on the repository. >>>>>> >>>>>> - Delete the gh-pages branch >>>>>> >>>>>> >>>>>> I want this change because then new PRs can change the >> documentation >>> in >>>>>> one PR rather than two (which to be honest is not going to happen >>> most >>>> of >>>>>> the time). >>>>>> >>>>>> The downside (we have to mention this) is that when you clone the >>>>>> repository, you have slightly larger downloads, but i think the >>>> trade-off >>>>>> is fair. If you see any other issues please mention these in this >>>> thread. >>>>>> >>>>>> >>>>>> Next task I would like to suggest to include: >>>>>> >>>>>> >>>>>> - Java docs generated and >>>>>> >>>>>> - Python docs generated to also be included as a sub-folder in >> docs, >>> so >>>>>> that the current webpage contains the docs for that to. >>>>>> >>>>>> >>>>>> How to include these in a sensible manner is currently a good >>> question, >>>>>> and has to be seriously well considered. Including these will also >>>> enable >>>>>> us to move documentation closer to the individual code parts, >> making >>> it >>>>>> again more likely to get the documentation done. >>>>>> >>>>>> >>>>>> Another more programmatic task is to make a parser for our DML >> files, >>>>> such >>>>>> that we can generate webpage documentation based on the >> documentation >>>>>> syntax used (much like the python and java docs generator). This >>> would >>>> be >>>>>> great since it will remove the duplication of documentation in the >>>>>> individual DML files, and the documentation we have already in >> files >>>> such >>>>>> as: docs/dml-language-reference.md. >>>>>> >>>>>> This also gives us an excuse to clean up that documentation/scripts >>>> which >>>>>> has very different implementations in our scripts folder: >>>>>> >>>>>> >>>>>> examples: >>>>>> >>>>>> >>>>>> - <https://github.com/apache/systemml/tree/master/scripts> >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>> https://github.com/apache/systemml/tree/master/scripts> >>> >>> systemml/scripts at master · apache/systemml · GitHub< >>> https://github.com/apache/systemml/tree/master/scripts> >>> github.com >>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>> creating an account on GitHub. >>> >>> >>>>>> >>>>> >>>> >>> >> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>> >> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml >>>> >>> >>> systemml/batch_norm1d.dml at master · apache/systemml · GitHub< >>> >> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml >>>> >>> github.com >>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>> creating an account on GitHub. >>> >>> >>>>>> < >>>>>> >>>>> >>>> >>> >> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml >>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< >>> >> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml >>>> >>> >>> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub< >>> >> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml >>>> >>> github.com >>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development >>> by creating an account on GitHub. >>> >>> >>>>>>> >>>>>> >>>>>> - < >>>>>> >>>>> >>>> >>> >> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml >>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< >>> >> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml >>>> >>> >>> systemml/glm.dml at master · Baunsgaard/systemml · GitHub< >>> >> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml >>>> >>> github.com >>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development >>> by creating an account on GitHub. >>> >>> >>>>>> >>>>>> <https://github.com/apache/systemml/tree/master/scripts> >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>> https://github.com/apache/systemml/tree/master/scripts> >>> >>> systemml/scripts at master · apache/systemml · GitHub< >>> https://github.com/apache/systemml/tree/master/scripts> >>> github.com >>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>> creating an account on GitHub. >>> >>> >>>>>> >>>>> >>>> >>> >> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>> >> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> >>> >>> systemml/Cox.dml at master · apache/systemml · GitHub< >>> >> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> >>> github.com >>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>> creating an account on GitHub. >>> >>> >>>>>> >>>>>> >>>>>> >>>>>> Furthermore, I would suggest to postpone changing the main website: >>>>>> >>>>>> https://github.com/apache/systemml-website >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>> https://github.com/apache/systemml-website> >>> >>> GitHub - apache/systemml-website: Mirror of Apache SystemML site< >>> https://github.com/apache/systemml-website> >>> github.com >>> Mirror of Apache SystemML site. Contribute to apache/systemml-website >>> development by creating an account on GitHub. >>> >>> >>>>>> >>>>>> But that task has to be started when the next release is scheduled, >>>>>> because then we need to find out how to copy the current >>> documentation >>>>> to a >>>>>> archive list of static webpages located: >>>>>> >>>>>> https://systemml.apache.org/documentation >>> Documentation<https://systemml.apache.org/documentation> >>> systemml.apache.org >>> Project Documentation Page >>> >>> >>>>>> >>>>>> >>>>>> best regards >>>>>> >>>>>> Sebastian >>>>>> >>>>>> ________________________________ >>>>>> From: Janardhan <janard...@apache.org> >>>>>> Sent: Sunday, May 24, 2020 11:02:53 AM >>>>>> To: dev@systemml.apache.org >>>>>> Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. >>>>>> >>>>>> Hi Sebastian, >>>>>> >>>>>> In our discussion[1][2] a while ago, there was a topic about >> changing >>>>>> website >>>>>> stack. >>>>>> >>>>>> Let us start a discussion about this in this thread. >>>>>> >>>>>> We are planning to work on this, after a feasibility study and tech >>>> stack >>>>>> vetting. >>>>>> >>>>>> Anybody would like to give suggestions would be great. >>>>>> >>>>>> [1] https://github.com/apache/systemml/pull/877 >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>> https://github.com/apache/systemml/pull/877> >>> >>> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by >>> Baunsgaard · Pull Request #877 · apache/systemml · GitHub< >>> https://github.com/apache/systemml/pull/877> >>> github.com >>> Brute change of documentation page to reflect name change. To fully >>> enforce this two more steps are needed: >>> https://github.com/apache/systemml-website Access to SystemML website >>> server on https://systemml.apache.org/ Furthermore, we might want to >>> consider starting fresh in the documentation. >>> >>> >>>>>> [2] https://github.com/apache/systemml-website/pull/66 >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>> https://github.com/apache/systemml-website/pull/66> >>> >>> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request >>> #66 · apache/systemml-website · GitHub< >>> https://github.com/apache/systemml-website/pull/66> >>> github.com >>> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used, >>> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, >> since >>> mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art. >>> Commit package-lock.json file to version control. >>> >>> >>>>>> >>>>>> Thank you, >>>>>> Janardhan >>>>>> >>>>> >>>> >>> >> >
