Anyways the changeset looks good to me. Including the pyhton api, and java.
- Janardhan On Friday, June 19, 2020, Matthias Boehm <mboe...@gmail.com> wrote: > don't worry about - it's all fine. Leaving it like that (for a single > commit) is MUCH better than trying to forcefully rewrite the history. > > Regards, > Matthias > > On 6/18/2020 5:35 PM, Baunsgaard, Sebastian wrote: > >> Hi Dev, >> >> >> I merged in the documentation PR, this was an slight oversight, while >> trying to merge something else. That said nothing bad happened except you >> did not get a chance to look through the PR, and it is progress in the >> direction discussed in this thread. >> >> >> Best regards >> >> Sebastian >> >> >> ________________________________ >> From: Matthias Boehm <mboe...@gmail.com> >> Sent: Saturday, June 6, 2020 7:03:19 PM >> To: dev@systemml.apache.org >> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. >> >> from my perspective it would be very important to have all builtin >> functions in a single markdown file to allow users to search for things >> and users don't care how a builtin function is internally implemented. >> So we might want to use this opportunity to consolidate the already >> documented builtin functions into the new list and add everything that >> is not yet documented as I'm sure many functions are missing. >> >> Btw, in the INFRA ticket for renaming our repositories [1], I also asked >> to disable sending all these github comments and PR interactions to our >> dev list as I'm sure people get overwhelmed by these auto-generated >> emails which leads to important messages being missed. Related to >> documentation, we might want to group categories of builtin functions >> into larger PRs instead of having a PR per builtin function as its >> primarily a copy of existing documentation so they can be reviewed more >> efficiently together to avoid unnecessary overhead. >> >> [1] https://issues.apache.org/jira/browse/INFRA-20362 >> >> Regards, >> Matthias >> >> On 6/6/2020 5:23 PM, Janardhan wrote: >> >>> Hi Arnab, >>> >>> We have all the Java-based builtin functions documented at [1]. >>> Most of these functions are already documented! >>> >>> [1] >>> http://apache.github.io/systemml/dml-language-reference. >>> html#built-in-functions >>> >>> >>> Thank you, >>> Janardhan >>> >>> On Fri, Jun 5, 2020 at 4:37 PM arnab phani <phaniar...@gmail.com> wrote: >>> >>> That's a good suggestion. One thing I want to point out that >>>> Builtins.java >>>> [1] contains all the dml-bodied and java-bodied built-ins. For the dml >>>> built-ins, it is easier to parse the corresponding dml script (if >>>> documentation is available there), but for other built-ins, I am not >>>> aware >>>> of any particular source today where they are documented -- a few of >>>> them >>>> are documented as a part of commit message but not all. >>>> Though it is certainly possible to slice the dml builtins from [1] out, >>>> but >>>> it will be awesome to generate docs for all of those. >>>> >>>> [1] >>>> >>>> https://github.com/apache/systemml/blob/master/src/main/java >>>> /org/apache/sysds/common/Builtins.java >>>> >>>> Regards, >>>> Arnab.. >>>> >>>> On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian < >>>> baunsga...@tugraz.at> >>>> wrote: >>>> >>>> Hi dev team! >>>>> >>>>> >>>>> Update / suggestion. >>>>> >>>>> >>>>> First of all, i think it is great that we have people working on our >>>>> documentation. It is really important. But I can't help to note that >>>>> the >>>>> improvements, are going in an unmaintainable direction where we have to >>>>> maintain the documentation at multiple locations. >>>>> >>>>> >>>>> For instance our new toOneHot function already had some basic >>>>> documentation inside its build-in script already [1], it would be much >>>>> nicer if we modified that and then had some automation to generate the >>>>> >>>> docs >>>> >>>>> made in commit [2]. Now we have to maintain two locations, resulting in >>>>> high risk of either being outdated. >>>>> >>>>> >>>>> We need to ask ourselves where do we want to make the edit of >>>>> documentation. I would argue that it should be located as close to the >>>>> internal definition of a function, because then when making or >>>>> modifying >>>>> that function the documentation is also reviewed. >>>>> >>>>> >>>>> To achieve this i suggest that we instead relocate the documentation >>>>> source and generation to an appropriately named file inside [3], that >>>>> simply loops through all function definitions from [4], and generate >>>>> markdown files for our /docs/ folder. >>>>> >>>>> >>>>> Shifting to such an implementation enables us to systematically cover >>>>> all >>>>> functions defined in the system. It also is a full list of the systems >>>>> current build-in functions thereby giving us the exact file paths to >>>>> extract the source documentation from the comments and generate the >>>>> docs >>>>> for those files. >>>>> >>>>> In the future it could be extended to verify and detecting which input >>>>> >>>> and >>>> >>>>> output types each function definition has, such that we have even less >>>>> manual documentation needs. >>>>> >>>>> >>>>> Best regards >>>>> >>>>> Sebastian >>>>> >>>>> >>>>> [1] >>>>> >>>>> https://github.com/apache/systemml/blob/master/scripts/built >>>> in/toOneHot.dml >>>> >>>>> [2] >>>>> >>>>> https://github.com/apache/systemml/commit/859ad3a72906c67b73 >>>> 00c9980251da4cde9ed8f8 >>>> >>>>> [3] >>>>> >>>>> https://github.com/apache/systemml/tree/master/src/main/java >>>> /org/apache/sysds/common >>>> >>>>> [4] >>>>> >>>>> https://github.com/apache/systemml/blob/master/src/main/java >>>> /org/apache/sysds/common/Builtins.java >>>> >>>>> >>>>> ________________________________ >>>>> From: arnab phani <phaniar...@gmail.com> >>>>> Sent: Sunday, May 31, 2020 3:15:44 PM >>>>> To: dev@systemml.apache.org >>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. >>>>> >>>>> Hi, >>>>> >>>>> Another advantage of markdown syntax is that it gives more freedom to >>>>> >>>> add a >>>> >>>>> description as needed. Java methods and builtins might not fall in the >>>>> >>>> same >>>> >>>>> bucket and different builtins might need different ways to describe it. >>>>> For example, glm needs more details than other built-ins. Too much >>>>> standardization can be a bad idea. >>>>> >>>>> Regards, >>>>> Arnab. >>>>> >>>>> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian < >>>>> baunsga...@tugraz.at> >>>>> wrote: >>>>> >>>>> Hi Janardhan, >>>>>> >>>>>> >>>>>> That would be one option, lets call it option 1, I have another >>>>>> >>>>> suggestion: >>>>> >>>>>> >>>>>> 1. Markdown syntax with out commented individual lines. >>>>>> 2. Jdoc like syntax with annotations >>>>>> >>>>>> Pros and cons >>>>>> >>>>>> * 1 Pros >>>>>> * Easy Syntax most ppl know it. >>>>>> * Easy to implement. >>>>>> * 1 Cons >>>>>> * Enables custom docs (everyone make their own distinct >>>>>> format >>>>>> again) >>>>>> * No standard way of verifying correctness of the docs >>>>>> (correct >>>>>> parameter names etc.) >>>>>> * No standard way of marking Input, return parameters. >>>>>> * Requires modified Syntax with # at each line. where # >>>>>> normally >>>>>> means header in markdown >>>>>> * 2 Pros >>>>>> * Easy Syntax Java ppl know it >>>>>> * Standard way of making input >>>>>> * Do not have to change any syntax at all from Jdoc >>>>>> * Already supported syntax in DML >>>>>> * 2 Cons >>>>>> * Harder to implement since one would have to look into java >>>>>> doc >>>>>> extraction and what is needed to support that, maybe we have to make >>>>>> >>>>> such a >>>>> >>>>>> thing ourselves? >>>>>> >>>>>> I personally like option 2 with Jdoc more and then extending into >>>>>> automatically parsing and making the either markdown files or HTML >>>>>> >>>>> files >>>> >>>>> for the docs that you either way have to do in option 1. >>>>>> >>>>>> Best regards >>>>>> Sebastian >>>>>> ________________________________ >>>>>> From: Janardhan <janardhan.pulivar...@gmail.com> >>>>>> Sent: Sunday, May 31, 2020 9:25:43 AM >>>>>> To: dev@systemml.apache.org >>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. >>>>>> Thanks. >>>>>> >>>>>> Hi Sebastian, :) >>>>>> >>>>>> Since the builtin DML files do not have full documentation yet, Can we >>>>>> write >>>>>> markdown syntax with the following heading in each dml file itself! >>>>>> >>>>>> 1. Function description >>>>>> 2. Usage >>>>>> 3. Arguments >>>>>> 4. Returns >>>>>> 5. Usage >>>>>> >>>>>> After that, as you've mentioned can be parsed with only removing # at >>>>>> >>>>> the >>>> >>>>> start of each line, and write >>>>>> all this data to one `builtins-reference.md` file. >>>>>> >>>>>> Thank you, >>>>>> Janardhan >>>>>> >>>>>> On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian < >>>>>> baunsga...@tugraz.at> >>>>>> wrote: >>>>>> >>>>>> If this is meant as a question then, yes from me (this probably calls >>>>>>> >>>>>> for >>>>> >>>>>> a vote?), if: >>>>>>> >>>>>>> >>>>>>> 1. someone is able to change the settings for the repository, to >>>>>>> >>>>>> use >>>> >>>>> the docs folder as the GitHub docs folder. >>>>>>> 2. someone wants to make a PR containing the content of the >>>>>>> >>>>>> gh-pages >>>> >>>>> branch. >>>>>>> >>>>>>> Best regards >>>>>>> Sebastian >>>>>>> ________________________________ >>>>>>> From: Janardhan <janardhan.pulivar...@gmail.com> >>>>>>> Sent: Tuesday, May 26, 2020 1:22:11 PM >>>>>>> To: dev@systemml.apache.org >>>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. >>>>>>> >>>>>> Thanks. >>>> >>>>> Hi Sebastian, >>>>>>> >>>>>>> Can we move the `docs` folder in the existing `master` branch >>>>>>> and merge the `gh-pages` branch to `docs` folder in the top-level >>>>>>> directory. :) >>>>>>> >>>>>>> - Janardhan >>>>>>> >>>>>>> On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian < >>>>>>> baunsga...@tugraz.at> >>>>>>> wrote: >>>>>>> >>>>>>> Hi Janardhan, >>>>>>>> >>>>>>>> >>>>>>>> Reworking the documentation seems like a really good idea! >>>>>>>> >>>>>>>> >>>>>>>> A first step in my opinion is to start in the main repository. >>>>>>>> >>>>>>>> >>>>>>>> I would suggest the following: >>>>>>>> >>>>>>>> >>>>>>>> - Merge the current gh-pages branch into master docs folder. >>>>>>>> >>>>>>>> - Change the documentations page : >>>>>>>> >>>>>>> http://apache.github.io/systemml/ >>>> >>>>> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/> >>>>> apache.github.io >>>>> SystemDS Documentation >>>>> >>>>> >>>>> to >>>>>> >>>>>>> use the docs folder for documentation. Should be doable inside the >>>>>>>> >>>>>>> settings >>>>>>> >>>>>>>> on the repository. >>>>>>>> >>>>>>>> - Delete the gh-pages branch >>>>>>>> >>>>>>>> >>>>>>>> I want this change because then new PRs can change the >>>>>>>> >>>>>>> documentation >>>> >>>>> in >>>>> >>>>>> one PR rather than two (which to be honest is not going to happen >>>>>>>> >>>>>>> most >>>>> >>>>>> of >>>>>> >>>>>>> the time). >>>>>>>> >>>>>>>> The downside (we have to mention this) is that when you clone the >>>>>>>> repository, you have slightly larger downloads, but i think the >>>>>>>> >>>>>>> trade-off >>>>>> >>>>>>> is fair. If you see any other issues please mention these in this >>>>>>>> >>>>>>> thread. >>>>>> >>>>>>> >>>>>>>> Next task I would like to suggest to include: >>>>>>>> >>>>>>>> >>>>>>>> - Java docs generated and >>>>>>>> >>>>>>>> - Python docs generated to also be included as a sub-folder in >>>>>>>> >>>>>>> docs, >>>> >>>>> so >>>>> >>>>>> that the current webpage contains the docs for that to. >>>>>>>> >>>>>>>> >>>>>>>> How to include these in a sensible manner is currently a good >>>>>>>> >>>>>>> question, >>>>> >>>>>> and has to be seriously well considered. Including these will also >>>>>>>> >>>>>>> enable >>>>>> >>>>>>> us to move documentation closer to the individual code parts, >>>>>>>> >>>>>>> making >>>> >>>>> it >>>>> >>>>>> again more likely to get the documentation done. >>>>>>>> >>>>>>>> >>>>>>>> Another more programmatic task is to make a parser for our DML >>>>>>>> >>>>>>> files, >>>> >>>>> such >>>>>>> >>>>>>>> that we can generate webpage documentation based on the >>>>>>>> >>>>>>> documentation >>>> >>>>> syntax used (much like the python and java docs generator). This >>>>>>>> >>>>>>> would >>>>> >>>>>> be >>>>>> >>>>>>> great since it will remove the duplication of documentation in the >>>>>>>> individual DML files, and the documentation we have already in >>>>>>>> >>>>>>> files >>>> >>>>> such >>>>>> >>>>>>> as: docs/dml-language-reference.md. >>>>>>>> >>>>>>>> This also gives us an excuse to clean up that documentation/scripts >>>>>>>> >>>>>>> which >>>>>> >>>>>>> has very different implementations in our scripts folder: >>>>>>>> >>>>>>>> >>>>>>>> examples: >>>>>>>> >>>>>>>> >>>>>>>> - <https://github.com/apache/systemml/tree/master/scripts> >>>>>>>> >>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>>>> https://github.com/apache/systemml/tree/master/scripts> >>>>> >>>>> systemml/scripts at master · apache/systemml · GitHub< >>>>> https://github.com/apache/systemml/tree/master/scripts> >>>>> github.com >>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>>>> creating an account on GitHub. >>>>> >>>>> >>>>> https://github.com/apache/systemml/blob/master/scripts/nn/ >>>> layers/batch_norm1d.dml >>>> >>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>>>> >>>>> https://github.com/apache/systemml/blob/master/scripts/nn/ >>>> layers/batch_norm1d.dml >>>> >>>>> systemml/batch_norm1d.dml at master · apache/systemml · GitHub< >>>>> >>>>> https://github.com/apache/systemml/blob/master/scripts/nn/ >>>> layers/batch_norm1d.dml >>>> >>>>> github.com >>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>>>> creating an account on GitHub. >>>>> >>>>> >>>>> < >>>>>>>> >>>>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/n >>>> n/layers/batch_norm2d.dml >>>> >>>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< >>>>> >>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/n >>>> n/layers/batch_norm2d.dml >>>> >>>>> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub< >>>>> >>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/n >>>> n/layers/batch_norm2d.dml >>>> >>>>> github.com >>>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml >>>>> development >>>>> by creating an account on GitHub. >>>>> >>>>> >>>>> - < >>>>>>>> >>>>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/b >>>> uiltin/glm.dml >>>> >>>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< >>>>> >>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/b >>>> uiltin/glm.dml >>>> >>>>> systemml/glm.dml at master · Baunsgaard/systemml · GitHub< >>>>> >>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/b >>>> uiltin/glm.dml >>>> >>>>> github.com >>>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml >>>>> development >>>>> by creating an account on GitHub. >>>>> >>>>> >>>>> <https://github.com/apache/systemml/tree/master/scripts> >>>>>>>> >>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>>>> https://github.com/apache/systemml/tree/master/scripts> >>>>> >>>>> systemml/scripts at master · apache/systemml · GitHub< >>>>> https://github.com/apache/systemml/tree/master/scripts> >>>>> github.com >>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>>>> creating an account on GitHub. >>>>> >>>>> >>>>> https://github.com/apache/systemml/blob/master/scripts/algor >>>> ithms/Cox.dml >>>> >>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>>>> >>>>> https://github.com/apache/systemml/blob/master/scripts/algor >>>> ithms/Cox.dml> >>>> >>>>> systemml/Cox.dml at master · apache/systemml · GitHub< >>>>> >>>>> https://github.com/apache/systemml/blob/master/scripts/algor >>>> ithms/Cox.dml> >>>> >>>>> github.com >>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by >>>>> creating an account on GitHub. >>>>> >>>>> >>>>> >>>>>>>> >>>>>>>> Furthermore, I would suggest to postpone changing the main website: >>>>>>>> >>>>>>>> https://github.com/apache/systemml-website >>>>>>>> >>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>>>> https://github.com/apache/systemml-website> >>>>> >>>>> GitHub - apache/systemml-website: Mirror of Apache SystemML site< >>>>> https://github.com/apache/systemml-website> >>>>> github.com >>>>> Mirror of Apache SystemML site. Contribute to apache/systemml-website >>>>> development by creating an account on GitHub. >>>>> >>>>> >>>>> But that task has to be started when the next release is scheduled, >>>>>>>> because then we need to find out how to copy the current >>>>>>>> >>>>>>> documentation >>>>> >>>>>> to a >>>>>>> >>>>>>>> archive list of static webpages located: >>>>>>>> >>>>>>>> https://systemml.apache.org/documentation >>>>>>>> >>>>>>> Documentation<https://systemml.apache.org/documentation> >>>>> systemml.apache.org >>>>> Project Documentation Page >>>>> >>>>> >>>>> >>>>>>>> best regards >>>>>>>> >>>>>>>> Sebastian >>>>>>>> >>>>>>>> ________________________________ >>>>>>>> From: Janardhan <janard...@apache.org> >>>>>>>> Sent: Sunday, May 24, 2020 11:02:53 AM >>>>>>>> To: dev@systemml.apache.org >>>>>>>> Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. >>>>>>>> >>>>>>>> Hi Sebastian, >>>>>>>> >>>>>>>> In our discussion[1][2] a while ago, there was a topic about >>>>>>>> >>>>>>> changing >>>> >>>>> website >>>>>>>> stack. >>>>>>>> >>>>>>>> Let us start a discussion about this in this thread. >>>>>>>> >>>>>>>> We are planning to work on this, after a feasibility study and tech >>>>>>>> >>>>>>> stack >>>>>> >>>>>>> vetting. >>>>>>>> >>>>>>>> Anybody would like to give suggestions would be great. >>>>>>>> >>>>>>>> [1] https://github.com/apache/systemml/pull/877 >>>>>>>> >>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>>>> https://github.com/apache/systemml/pull/877> >>>>> >>>>> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by >>>>> Baunsgaard · Pull Request #877 · apache/systemml · GitHub< >>>>> https://github.com/apache/systemml/pull/877> >>>>> github.com >>>>> Brute change of documentation page to reflect name change. To fully >>>>> enforce this two more steps are needed: >>>>> https://github.com/apache/systemml-website Access to SystemML website >>>>> server on https://systemml.apache.org/ Furthermore, we might want to >>>>> consider starting fresh in the documentation. >>>>> >>>>> >>>>> [2] https://github.com/apache/systemml-website/pull/66 >>>>>>>> >>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< >>>>> https://github.com/apache/systemml-website/pull/66> >>>>> >>>>> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull >>>>> Request >>>>> #66 · apache/systemml-website · GitHub< >>>>> https://github.com/apache/systemml-website/pull/66> >>>>> github.com >>>>> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were >>>>> used, >>>>> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, >>>>> >>>> since >>>> >>>>> mixins are not supported in 3.0 Upgrade npm dependencies to >>>>> state-of-art. >>>>> Commit package-lock.json file to version control. >>>>> >>>>> >>>>> Thank you, >>>>>>>> Janardhan >>>>>>>> >>>>>>>> >
